Troubleshooting the Master

Verifying the Master’s Version

The Tidal Support Center requires the version of your TA Master to troubleshoot any issues related to the Master.

To verify the TA Master version:

Click Operations > Master Status on the Navigation pane of the client to display the Master Status pane.

The version of the Master being used is noted at the top of the Master Status pane.

Diagnostic Logging for the Master

If you are enabling diagnostics before contacting the Tidal Support Center, start at the Info level and the Tidal Support Center will provide additional instructions after receiving the diagnostic files you have recorded.

These components are monitored:

Scheduler Log – Records system level messages regarding the Master.
Client Manager Log – Records messages about client activity.
Agent Manager Log – Records messages from agents running jobs.
Compiler Log – Records messages about the status of production schedules being compiled.
Job Manager Log – Records messages about the status of jobs.
Event Manager Log – Records messages about events defined in TA.
Queue Manager Log – Records messages about queue activity.
Database Log – Records messages relating to the state of the database.
Communications Log – Records messages concerning all defined connections and sockets. Be aware that setting this component to a high level of logging results in a large amount of information that consumes large amounts of disk space.

Each component has a drop-down list with seven levels of progressively more detailed logging. Each level includes the messages of the previous levels of logging.

The levels of logging are:

None – No logging for the component.
Severe – Logs only serious problems for that component (default).
Warning – Logs potential problems for the component as well as messages from the Severe logging level.
Info – Logs status messages about the normal operation as well as messages from lower logging levels.
Low Debug – Logs important debugging messages as well as messages from lower logging levels.
Medium Debug – Logs an increasing amount of debugging information as well as messages from lower logging levels.
High Debug – Logs the largest amount of debugging information as well as messages from lower logging levels.

The TA Master writes the diagnostic log entries to the current log file until the file reaches the maximum file size (in MB) specified in the master.props’s LogFileSize parameter. For more information about the LogFileSize parameter, see the master.props properties table in the Basic TA Configuration chapter in the TA Installation Guide. If the file does not exist at the time the TA Master relays the diagnostic information, then the relayed information is buffered and written to the log file after the file is created.

Once the current log file reaches the specified maximum file size, the TA Master saves the file with a name in the

Master-YYYYMMDDHHMMSS.log format (Example: Master-20020814042231.log), deletes the current log file, and starts writing the diagnostic information in a new log file. The diagnostic log files can be located in the TA Master’s log directory.

Note: If any diagnostic information is relayed between the time the current log file is deleted and the new log file is created, then that information will be lost.

In some cases, the actual size of a diagnostic log file may be less than the specified maximum file size because:

The TA Master shuts down (whether by design or not) before the log file could reach the specified maximum file size.
The TA Master is still creating the current log file.

Note: Do not delete the log file with the latest timestamp as it indicates the current log file.

Diagnostic Logging Tasks

Adjusting the Log File Retention

Each time the log file for the Master reaches one megabyte in size, it is saved and another log file started. By default, the Master is configured to store the 25 most recent log files. When the log files exceed the maximum number, the oldest log file is automatically deleted to make room for a newer log file.

You may need to adjust the log file retention number whenever you are running diagnostic logging. A higher level of diagnostic detail will create log files that add up quickly. Increase the log file retention number whenever you run diagnostic logging to compensate or you will start deleting the earliest log files before you have a chance to examine them.

To adjust the log file retention:

Stop the Master: tesm stop from the command line of the Master machine.
Locate the master.props file in the config directory of the Master machine and open it.

Add the line to the master.props file:
```
MaxLogFiles=<number of log files>
```
where <number of log files> is the maximum number of log files to retain before deleting the oldest file. The minimum number of log files to be retained is three. There is no maximum number.
Restart the Master: tesm start from the command line.

Purging Old Job Data

The TA database grows in size with additional job history records unless the older data is purged on a recurring basis.

To purge old job data:

Open the Activities > System Configuration > Master tab to check the data retention value.

Choose the Automatic Daily History Cleanup option, if not selected already. If you are using a database alias other than the default “Admiral”, this feature may not work correctly.

Running Diagnostic Logging as a Job

If a problem is occurring at a time that is difficult for an operator to monitor, say when a certain job deep in the production schedule runs, you can run a job to enable diagnostic logging just as the suspected job runs. You can create a logging job from the command line using a Java utility that TA provides. Configure the job to run just before the problem occurs and then create another similar job to return the diagnostic logging levels to their original settings after recording the system information. This job will limit how long the diagnostic logging runs to prevent massive amounts of information from clogging your disk space.

To run diagnostic logging as a job:

Click Add Job on the TA toolbar or right-click in the Jobs pane and click Add Job to display the Job Definition dialog.

Type a name for the job.

Type java in the Command field on the Program tab.

Modify the command parameters as needed in the Command Parameters field:

java -DTidal_HOME=/opt/Tidal/Master -cp/opt/Tidal/lib/scheduler.jar com.tidalsoft.scheduler.SetDebugLevel

-m <Master Machine Name>

-p ClientPort(defaults to correct number)

-S (Scheduler Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-CL (Client Manager Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-A (Agent Manager Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-CO (Compiler Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-J (Job Manager Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-E (Event Manager Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-Q (Queue Manager Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-D (Database Log) SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

-COM(Comununcations Log)SEVERE|WARNING|INFO|LOW|MEDIUM|HIGH

Do not include any of the explanatory text that is in parentheses or brackets. Replace the text in brackets with the name of the machine that the Master resides upon. The section of the first line “opt/Tidal” should be the directory location where the Master is installed. Select just one of the logging levels to use with only the components you wish to enable for diagnostic logging. Any component log that is not included in the parameters is not affected by the logging job and retains its original setting. Don’t forget to run another job to return the diagnostics to the original setting.

Running the Windows Master in Debug Mode

Even if you have configured the Master to log diagnostic information, there is still a period of time between when the Master starts and when it begins logging information. If the issue you are trying to troubleshoot occurs in this time window before diagnostic information is recorded than you should configure the Master to run in debug mode. This will ensure that information is collected the entire time the Master runs.

To run the Windows Master in Debug mode:

Open a command prompt screen from the Start menu by selecting All Programs>Accessories>Command Prompt.

Type the directory path to the Master executable file and then place a debug parameter after.

Example: “C:\Program Files\TIDAL\Scheduler\Master\saMaster64.exe” -debug

Turn off the debugging mode by killing the saMaster64.exe process that you started.

Running the Unix Master in Debug Mode

Type from the command line program on the Unix Master machine:

tesm -debug start

Turn off the debugging mode by killing the debug process that you started.

Resolving Errors with OCSEXIT Jobs

If you find that jobs created using the OCSEXIT variable, and that run on Windows agents, consistently end in Completed Abnormally you may need to update your system path.

To update your system path:

Click Windows Start menu and open Settings>Control Panel.

Start the System Properties control panel.

Click the appropriate tab and append %systemroot%\system32 to your system’s Path variable.

Resolving a Master Error

The Windows TA Master runs as a Windows service. Services need to be started and controlled through a service manager, such as the TA Service Control Manager utility. Double-clicking directly on the saMaster.exe file or an icon shortcut associated with the saMaster executable will result in application errors and an access violation.

To resolve this issue, always start and stop the TA Master through the Service Control Manager or the Windows Services program.

Email Problems

If you are having trouble sending email through TA , examine the Windows Event Viewer for application errors attributed to TidalSAMASTER. The information provided in these logs can help diagnose the problem.

Internet Email

For TA to use internet mail effectively, the Master machine must have a continuous internet connection. If the mail system goes offline, you will miss email notifications. Verify that you have followed the email prerequisites and configured the email functions of TA .

Making the Master Service Run as a User

To make the Master service run as a user:

Click Windows Start menu , then open Programs>Tidal Automation > Scheduler > Master > Service Control Manager.

Choose the Scheduler Master service from the Service field drop-down menu.

Click Configure to display the Service Configuration dialog.

Click the Other Account option in the Logon Information section.

Type the complete user account (include domain) and password information.

Click OK, then exit the Service Control Manager.

Database Connectivity Loss

If database connectivity is lost, the Master pauses itself and continues to poll for connectivity. When the Master finds connectivity working, it resumes (on its own) and continues job processing as usual. During normal operation, database connectivity is checked every 15 seconds. If connectivity is lost, the Master checks every 5 seconds for connectivity.