Data Mover Job Support

Note: Data Mover jobs are only supported on Unix/Linux agents.

By default, when the 3.2 agent is installed, it can run with Java 1.8 as previous agents, but it will not support Amazon S3 (AS3) or Hadoop DFS (HFS) Data Mover operations. Data Mover jobs sent to the default agent will fail with a 'wrong agent' indication.

In order to utilize AS3 or HFS Data Mover functionality, you must be running the appropriate level of Java and you must copy the associated support files into the Agent/ lib directory. AS3 and HFS functionalities require Java 1.8.

There are new subdirectories in the Agent/Unix directory. There is a new DataMover directory with three subdirectories - AS3, HFS-A (Apache Hadoop) and HFS-C (Cloudera Hadoop) that contain the associated files to support the DataMover functionality, if the associated support is needed.

AS3 Functionality

The TAgent.AS35 file in the installed Agent/lib directory is the TAgent.jar file that is compiled with Java 1.8 and contains the AS3 interface support.It will replace the existing TAgent.jar file in the installed Agent/lib directory. Rename the existing TAgent.jar file (not using the .jar extension), if desired, and then copy or rename the TAgent.AS35 file to TAgent.jar. The tagent.ini file for this installation of the agent must point to Java 1.8 version.

Copy all files from the above referenced AS3 subdirectory into the Agent/lib directory.

AS3 Usage Notes

The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from 1 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

When using Multipart upload, each part must be at least 5 MB in size, except the last part. So, in the list of files provided on the dialog, each must be at least 5MB other than the last file in the list.

HFS Functionality

The TAgent.HFS6 file in the installed Agent/lib directory is the TAgent.jar file compiled with Java 1.8 and contains the Hadoop Distributed File System interface support. It will replace the existing TAgent.jar file in the installed Agent/lib directory. Rename the existing TAgent.jar file (not using the .jar extension), if desired, and then copy or rename the TAgent.HFS6 file to TAgent.jar. The tagent.ini file entry for this installation of the agent must point to a Java 1.8 or the default Java must be 1.8. You can also run AS3 DataMover jobs with this TAgent.jar, but you must copy all the files from the AS3 subdirectory into the Agent/lib directory also in order to run AS3 jobs.

Hortonworks Hadoop

Download the Hortonworks Hadoop client libraries and copy into the Agent/lib directory.

Cloudera Hadoop

Download the Cloudera Hadoop client libraries and copy into the Agent/lib directory.

MapR Hadoop

In order to use DataMover for MapR Hadoop, the MapR Client must be installed on the machine running the TA agent. The TA agent supports MapR Client versions 1.2.9 and 2.0.0. It is the user's responsibility to ensure that the MapR Client is installed properly and is communicating with the MapR Cluster.

There are no files to be copied for MapR Hadoop. However, updates to the tagent.ini file are required. See HFS Usage Notes for details.

HFS Usage Notes

Agent Ini File

Kerberos Configuration

If the Agent is going to access any Hadoop file system that is secured by Kerberos, then the Kerberos Realm and Kerberos KDC Name must be specified in the Agent's tagent.ini file. The new parameters are KerberosRealm and KerberosKDC. Like other tagent.ini parameters, these values can be specified at a global (all) agent level and on a per agent basis. Unless both of these parameters are defined, the agent will not attempt Kerberos authentication even if the Hadoop Data Mover Job has checked the Use Kerberos Authentication check box.

MapR Configuration

When using MapR Hadoop on a 64-bit machine, add this line to your tagent.ini file (assuming the MapR Client is installed in the default location):

jvmpara=-Djava.library.path=/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64

When using MapR Hadoop on a 32-bit machine, add this line to your tagent.ini file (assuming the MapR Client is installed in the default location):

jvmpara=-Djava.library.path=/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-i386-32

To use MapR Hadoop, you must also specify the location of the MapR Hadoop jar files. Use the MapRClasspath parameter to specify the full path to the required MapR Hadoop jar file directory.

Add this line to your tagent.ini file (assuming the MapR Client is installed in the default location): maprclasspath=/opt/mapr/hadoop/hadoop-0.20.2/lib/*

User Configuration File

With this release of the Agent, there is a new user configuration file, TdlUser.cfg, that specifies parameters for the runtime user associated with a job. It is located in the agent's root directory.

Example: /opt/TIDAL/Agent/<name of agent>.

The user configuration file has this layout:

parameter=value parameter=value

[user-1] parameter=value parameter=value

[user-2] parameter=value parameter=value

A parameter value is specified in a parameter/value line which has the form parameter=value. Default configuration parameters to be applied to all users are specified before the first user-specific parameter values. This is referred to as the “default section”. To specify parameter values and to override a default parameter value for a particular user, add a section for that user. A user section starts with a "user section" line that contains the user name enclosed in brackets (“[", “]”) followed by a number of parameter/value lines. All parameter/value lines following a user section line up until the next user section line (or end of the file) are applied to that specific user. Parameter values specified in a user section override parameter values that are specified in the default section. Lines that start with the "#" character are ignored.

The new user configuration parameters are KerberosPrincipal and KeyTabFilePath. These parameters specify the Principal and KeyTab file for the Agent to use when performing Kerberos authentication.