FTP Job Definition Dialog

For a remote FTP (File Transfer Protocol) to work, these limitations apply:

  • FTP server exists

  • FTP server on the local and remote side must be configured

    Some FTP jobs may require that the domain name be part of the user name. If a FTP job requires the domain be part of the user name, before you can run the job you must modify the user name by adding the domain to the User field in the format, domain\user.

When you select the Add FTP Job option, you will find that the FTP Job Definition dialog differs from the standard Job Definition dialog in two ways:

  • Instead of a Program tab, there is an FTP tab

  • Some of the fields on the Run tab are different to accommodate FTP requirements

These two tabs with their differences are described below. The other tab and fields in the FTP Job Definition dialog are the same as the normal Job Definition dialog. .

FTP Regular Expressions

The Filename field on the FTP tab of the FTP Job Definition dialog accepts file name and query. When $ is entered as the first symbol in the Filename field, the field recognizes it as a query. When any other symbol is entered as the first symbol in the Filename field, the field recognizes it as a file name.

Query Syntax

A query syntax is as follows:

command ('wildcard mask' or /reg exp/)d;

where

  • command – Any action with files from the below Commands table.

  • parenthesis ( ) – Used for grouping conditions. Within the group, you can use ‘|’ for alternation.

  • single quotes (' ') – Recognized as a wildcard. A wildcard is just an asterisk * that can match 0 to many characters or a question mark.

  • ? – Matches any single character or none.

  • two backslashes (//) – Recognized as a regular expression.

  • reg exp – A regular expression that specifies a sequence of characters that matches it. For example, if the file name contains a digit, it should be an integer with a value from 1 to 3.

  • d – deletes the files. For more information, see Commit and Rollback.

  • semicolon (;) – Separates one command from the other command.

These examples show how single quotes are used in a query syntax.

  • $'?xyz.txt'; – Finds all the TXT files with names starting with any symbol (or none) and ending with “xyz”.

  • $'*.txt'; – Finds all the TXT files. An asterisk (*) is used to find files with names having 0 to many characters.

Commands

The Upload query is performed after all the commands have been executed successfully.

The Download query is performed right after the dollar sign ($) command, where the file is downloaded to the local directory before other commands.

Command

Description

Example

Input Data

Output Data

Regex Support

Commit with ‘d’

Rollback

$

For uploading, the files are searched locally.

For downloading, the files are searched remotely

$'*.txt' or $/../

empty file list

list of files

+

Delete found files from local storage

Upload:

-Download:

Deletes files from local storage

Z

zip file

Z’relative path towards working dir’

If such folders are not found, they are created automatically

list of files

one zip file

–

deletes zip file and folders created from local storage

deletes zip file and folders created from local storage

U

unzip file

U

Extracts to working directory

list of all zip files

all the files from all zip files

–

deletes unzipped files from local storage

deletes unzipped files from local storage

M

generate MD5

(checksum) for each file

M'<>.md5'

The same directory as for original files

list of files

list of files + their checksums

–

deletes checksums from local storage

deletes checksums from local storage

R

rename

R'pre-fix<original filename>post-fix’

list of files

renamed files

–

deletes renamed files from local storage

renames to the previous ones

C

copy

C'relative path towards working dir'

list of files

copies of files

–

deletes copied files from local storage

deletes copied files from local storage

V

validates files withmask to search for these files

V'x<>.md5'

list of files and their checksums

list of files and their checksums

–

 

 

+

includes to the list

+'...' or +/.../

list of files

list of files + added files

+

deletes added files from local storage

 

–

excludes from the list

-'...' or -/.../

list of files

the list of files - excluded file

+

 

 

All the queries are executed locally. Regular expressions and wildcards are applied only to file names.

Commit and Rollback

A commit is a successful execution of a query.

If commands contain d (for example, Z'relative path towards working dir'd;), the files (folders) that are obtained from the execution of the commands are deleted from the working directory. After the last command of the query is executed with a success, the deletion process begins from the last executed command to the first one (that contains d).

A rollback is an unsuccessful execution of a query.

When any command fails (connection refused, no disk space, and so on), there is a rollback: every command tries to recover the state before the command was executed. The process goes from the last executed command to the first command and recovers the initial state.

Example: $'*.txt'; Z'relative path towards working dir'd; C'relative path towards working dir'

Commit: When all the commands are executed with a success, deletes a zip file.

Rollback: If C command fails, deletes a zip file (recovers the initial state).

Command Builder

For the FTP operations MPUT Files and MGET Files, a Command Builder button is shown using which query can be built.

To build the query:

  1. Click the FTP Operation drop-down list on the FTP Job Definition > FTP tab, select either MPUT Files or MGET Files.

  2. Click Command Builder shown next to the File Name/Command field.

    The Command Builder dialog appears.

  3. Click Add in the Command Builder dialog. The Command Definition dialog appears.

  1. Choose one of these commands from the Command drop-down list and input the details required for the selected command.

    • Search Files – Searches the files using the File Mask. For uploading, the files are searched locally. For downloading, the files are searched remotely.

    • File Mask – Enter the file mask or select a variable representing the file mask from the Variables menu. If the Wild Card option is selected, the file mask is enclosed within single quotes (Example: ‘*log’). If the Regular Expression option is selected, the file mask is enclosed within slash (Example: /*log/). This is a required field.

      Enable the Delete found files from local storage checkbox, if you want the file(s) to be deleted from local storage.

    • Zip – Zip file.

    • File Name – Enter the zip file name or select a variable from the Variables menu. This is a required field.

    If the folders mentioned in the relative path are not found, they are created automatically.

    Enable the Delete zip files and folders from local storage checkbox, if you want the zip files and folders to be deleted from local storage.

  • Unzip – Unzip file. Enable the Delete unzipped files from local storage checkbox, if you want the unzipped files to be deleted from local storage.

  • Checksum – Generates MD5 (checksum) for each file.

    Checksum file name prefix – Enter the file name prefix or select a variable from the Variables menu.

    Checksum file name suffix – Enter the file name suffix or select a variable from the Variables menu. This is a required field.

    Enable the Delete checksums from local storage checkbox, if you want the checksums to be deleted from local storage.

  • Rename Files – Renames the files using the Prefix Original file name and Suffix Original file name. Value to one of these fields is mandatory.

    Prefix Original file name – Enter the file name prefix or select a variable from the Variables menu.

    Suffix Original file name – Enter the file name suffix or select a variable from the Variables menu.

    Enable the Delete renamed files from local storage checkbox, if you want the renamed files to be deleted from local storage.

  • Copy Files – Copies the files to the specified Destination Directory.

    Destination Directory – Enter the directory path or select a variable representing a directory path from the Variables menu to which the files will be copied. This is a required field.

    Enable the Delete copied files from local storage checkbox, if you want the copied files to be deleted from local storage.

  • Validate – Validates the MD5 checksum of files using the checksum files found using the file name pattern specified by Checksum file name prefix and Checksum file name suffix.

    Checksum file name prefix – Enter the file name prefix or select a variable from the Variables menu.

    Checksum file name suffix – Enter the file name suffix or select a variable from the Variables menu. This is a required field.

  • Include to list – Includes a list of files provided in the Files to include field.

    Files to include – Enter the files to be included in the list or select a variable from the Variables menu. If the Wild Card option is selected, the file is enclosed within single quotes (Example: ‘*log’). If the Regular Expression option is selected, the file is enclosed within slash (Example: /*log/). This is a required field.

    Enable the Delete added files from local storage checkbox, if you want the included files to be deleted from local storage.

  • Exclude from list – Excludes a list of files provided in the Files to exclude field.

  • Files to exclude – Enter the files to be excluded from the list or select a variable from the Variables menu. If the Wild Card option is selected, the file is enclosed within single quotes (Example: ‘*log’). If the Regular Expression option is selected, the file is enclosed within slash (Example: /*log/). This is a required field.

  1. Click OK.

  1. Repeat step 4 until the required query is built.

    The query built from step 4 is shown in the Command text box on the Command Builder dialog.

  1. Click OK. The query is populated in the File Name/Command field on the FTP Job Definition > FTP tab.

FTP Tab

The FTP tab contains these fields:

  • Protocol – Select one of these FTP protocols to use in this job:

    • FTP – The job will use the normal FTP protocol. No file encryption is used with this option.

    • FTPS – The job will use FTPS or FTP Secure and FTP-SSL.

    • SFTP – The job will use SFTP or Secure File Transfer Protocol. The file is encrypted when transferred between hosts using SSH2. SFTP can only use the binary format so the ASCII format is unavailable if this option is selected. The FTP Quote feature that assigns file formatting to a file being transferred between a mainframe and non-mainframe platform is unavailable with SFTP.

  • FTP Operation – Define what the FTP job will do by selecting an option from the FTP Operation drop-down list. Choose from these options:

    • PUT File – Copies the designated file from the agent to the FTP server.

    • GET File – Copies the designated file from the FTP server to the agent.

    • MPUT Files – Copies multiple files using wildcards from the agent to the FTP server. For more information on wildcards, refer to Wildcards in File Dependencies.

    • MGET Files – Copies multiple files using wildcards from the FTP server to the agent. For more information on wildcards, refer to Wildcards in File Dependencies.

    • RENAME File – Changes the name of the file on the FTP server.

    • DELETE File – Removes the designated file from the FTP server.

    • MDELETE File – Deletes files on remote computers.

    • LIST Directory – Lists the files within a designated directory on the FTP server.

    • MAKE Directory – Creates a designated directory on the FTP server.

    • DELETE Directory – Deletes a designated directory on the FTP server.

The remaining fields on the FTP tab vary according to which FTP Operation option was selected.

  • Format – There are two file formats available to use with FTP

    • ASCII – This format maintains the line returns within a file but loses most other formatting. This format option is not available if the SFTP protocol option is selected.

    • Binary – This format saves formatting within the file.

  • Filename – Type the name of the file for the FTP operation. The Filename field accepts the file name and query. When $ is entered as the first symbol, the Filename field recognizes it as a query. When any other symbol is entered as the first symbol, the Filename field recognizes it as a file name. For more information on the file name query syntax and the associated regular expressions, see FTP Regular Expressions.

  • Replace Files – Choose this option if you wish to overwrite existing files when using the MPUT or MGET options; otherwise, existing files are skipped.

  • Local Path – This is the pathname to a directory on the local or agent machine.

  • Remote Path – This is the pathname to a directory on the FTP server.

  • Quote – Type the formatting attributes for a file being transferred between a main frame platform and another platform, e.g., Windows to z/OS. The field is limited to 255 characters. This option is not available if the SFTP protocol option is selected.

  • New Filename (Only used with the Rename File option.) – Designate the new name to be used when renaming the remote file.

When SFTP is selected from the Protocol drop-down list, the SFTP-only fields are displayed.

  • SFTP Options – The only secure shell option currently supported is SSH2 only.

  • Advanced – Clicking this displays the SFTP Advanced Settings dialog with the standard encryption options. Use this feature to configure the order to be used when cycling through the encryption protocols as the agent searches for the protocol used by the remote machine. The Encryption Cypher Selection Policy field displays the AES encryption method.

The encryption method may be used during the file transferal between two machines with the secure FTP procedure.

Note: If you want to use one of the outdated encryption protocols, like, Blowfish, 3DES, or DES, update the AllowDeprecatedEncryption value in the sysval table. When AllowDeprecatedEncryption (id 241) is set to true, these deprecated encryption methods will become visible and available for use.

A user can configure the order in which TA attempts to find the encryption protocol recognized by the other machine. If the user knows which encryption protocol is used by the other system, they can choose that protocol and use the up and down arrows to move it to the top of the list. This will make it the first protocol attempted during file transfer. Moving the correct protocol to the top of the list eliminates guesswork and saves time during the FTP process.

  • User/Password – The user account and password used to logon to the FTP server. This could be a Windows or Unix account. If this option is selected then the Pass Phrase and Private Keyfile Location fields are not displayed.

  • Private Key File – Indicates that a private key file is being used with this job instead of system login by the user.

    Note: The FTP user must match the user under which the key was created.

    The key file must be generated by an SSH open system. To use a private key file, you must supply the pass phrase and directory location of the private key file.

  • Pass Phrase – The password needed to access the private key file.

  • Private Keyfile Location – The directory pathname to the private key file for the agent. This should be a local directory rather than a network directory.

  • Variables – When you choose a SFTP protocol, the Variable drop-down list is enabled on clicking the Private key File Location field. You can choose these variables from the Variable drop-down list:

    • System Variables – Using these variables, you can provide the details such as System Time, System Date, and so on, in the resource path.

      Example: <SysTime.H:mm>

      This example provides the system time in resource path, where H is Hour and mm is minutes.

    • Job Variables – Using these variables, you can provide the details related to jobs such as Job name, Job Owner, Job Command, Agent Name, Earliest Start Time, Latest Finish Time, and so on, in the resource path.

      Example: <JobName>

      This example provides the name of the job in the resource path.

    • Job Run Variables – Includes the variables such as Job Output, Job Exit Code, Job Status, Start Time, Finish Time, and so on.

      Example: <JobOutput>

      In this example, the output of the job is taken as the input of the resource path.

    • User defined Variables – Includes the variables that are created for the particular user.

    • Public Variables – Includes the variables which are defined as Public.

    You can save a single variable or multiple variables along with the string. The FTP job output obtained when you choose a variable or when you manually enter the location of the private key file will be the same.

    If you create an action that is tied to an FTP job, you can click the Variables to use a variable listed under Job Variables in the FTP Statistics submenu. There are four FTP job variable options available:

    • # of Files – Details the number of files transferred.

    • # of Bytes – Details the size of the file (in bytes) that was transferred.

    • # Skipped – Details the number of files within a designated directory that were not FTP’ed.

    • Duration (Seconds) – Details how long (in seconds) that it took to perform the FTP operation.

Run Tab (FTP)

Agent Information

In the Agent Information section you define the agent, the user account to be used and the FTP server information.

  • Agent (Local Host) – The agent machine that will act as an FTP TA client. You must select an agent from the drop-down list.

  • Agent List – An agent list is a set of agents that can run the FTP job. Agent lists allow for workload balancing, broadcasting, and dynamic rerouting. If you select the Agent List option, the Agent Name drop-down list displays agent lists instead of agents. Be sure to select an agent list for the platform on which the FTP job can run.

  • Local User – A runtime user account that the job runs under. Provides an additional layer of security since the user must be added to the runtime user’s list. If the FTP Local User Mandatory option on the Master tab of the System Configuration dialog is selected, an FTP job cannot run without an authorized local user.

  • FTP Host – The name of the machine that hosts an FTP server. If the FTP server uses a port that is not the default port, 21, you may also specify the port number immediately after the host machine name, separated by a colon, such as sputnik:5621. In FTP Host line, you may also place the FTP in to Passive mode by sputnik:5621:P.

    Note: Tidal Automation supports explicit FTPS mode, where a secured session is established on port 21 (explicit FTPS mode) using SSL or TLS, upon explicit request from the Client. For implicit FTPS mode support, you must specify port 990 in the FTP Host as described above.

  • FTP User – Specify a user from those defined to TA that has an associated password to log on to the FTP server. A user account may not be necessary if the FTP server offers Anonymous login.

  • Anonymous – Choose this option if you know that the remote FTP server has been configured to accept this option. Using anonymous as a FTP login does not normally require a user account and password. You must supply a TA user with an email account before using this option.

Tracking Section

In the Tracking section, select a method to determine that a job has completed.

These four options are available:

  • External – This option requires an external application or user to signal that the FTP task completed.

  • Exit code – This option bases the status of the job on the success or failure of the FTP operation.

  • Scan Output: Normal String – If you select this option, type a text string in the field below that signals a job completed normally if the text string appears in the job’s output.

  • Scan Output: Abnormal String – If you select this option, type a text string in the field below that signals a job completed abnormally if the text string appears in the job’s output.

Duration Section

In the Duration section, specify the time parameters for job completion.

These parameters are used to send an alert to the console if a job is not completing as expected.

  • Estimated – This value configures the forecast next to the Job Activity pane.

  • Minimum – If the job completes before the specified number of minutes, an alert can be created to notify the operator.

  • Maximum – If the job completes after the specified number of minutes, an alert can be created to notify the operator.