Jobs and Job Groups

A Tidal Automation (TA) job is a set of instructions about how, when and where to perform an automated task. In the job definition, you can specify a short name for the job (job alias), a command or script to run, where to run the job (agent), the days and times to run the job, the dependencies that need to be satisfied before it can run, and other runtime criteria.

You can organize jobs into TA job groups, where each job in the group can inherit properties from its parent group. Job groups themselves can belong to other job groups. You can view the entire hierarchical structure of all defined jobs and job groups from the Jobs pane.

A job’s or job group’s definition can be added to the production schedule either manually on demand or automatically through a calendar. Each entry of the job into the production schedule is called a job instance; it is an instance of the job definition at a specific time. Job instance history can be reviewed for auditing purposes.

Job and Job Group Concepts

Job Hierarchy

Jobs are built on a hierarchy of job and job group ownership. A job group is a container for a set of jobs, usually part of a common application or department. The job group has its own name and set of runtime instructions.

You can use job groups to submit jobs that either depend on each other, or should run together. For example, all the jobs in payroll can belong to a group called Payroll. The job group can provide default settings to all the child jobs that belong to it. Jobs and job groups are displayed in the Jobs pane. Job groups can save you the time it takes to set up job definitions because each job in the job group can inherit the characteristics of that job group. When you want to create several jobs with similar scheduling characteristics, you can define those jobs within a job group and set the scheduling characteristics in the job group definition. It is also possible to change scheduling characteristics at the job level even though the job belongs to a group.

For example, if a job group is defined to run every Friday, then every job in that job group is automatically defined to run on Friday. If one job in the job group must run on Saturday, then that one job can be changed to the proper run day without affecting the other jobs – as long as you disinherit the job group calendar and change the calendar from within that job.

The ultimate ownership of a job or job group belongs to either the user or a workgroup. A workgroup is a collection of users who can share access to the same jobs. Workgroups are displayed in the Workgroups pane.

Job Groups and Inheritance

You can organize related jobs into groups. Using job group inheritance saves you time and reduces the possibility for error by having jobs automatically inherit properties from the job group to which they belong. You need only change the job properties that are unique to each job. For example, if a job group is defined to run every Friday, then each job that belongs to the job group can inherit the job group’s calendar to run on Fridays as well. Note that when you schedule a job group, every job in the group is scheduled as well, unless you choose not to inherit the calendar property.

Note: If a job within a job group does not inherit the parent group’s calendar (the Inherited option on the job’s Schedule tab is not selected), then the calendar assigned to the job must be a subset of the dates within the job group’s calendar. In other words, the job’s calendar cannot contain any dates that are not part of the job group’s calendar.

Job Commands

Each job is assigned one command or script file. The command can be an executable file, a shell script, a batch file, a command file or any other executable file available to the agent that will run the job. You can also specify parameters to be passed to the command. This allows you to create different variations of a definition, based on the parameters that you pass to it. You can even use system or user variables in the command name or parameters.

Agents

You can choose which agent will run your job. You can run your job on a specific agent in your network or you can have TA pick an agent based on a list of agents called an agent list. By running jobs using an agent list, you can take advantage of workload balancing, broadcasting and dynamic rerouting.

Scheduled vs. Unscheduled Jobs

Jobs and job groups can run either automatically on a regular schedule when you assign a calendar or on an unscheduled basis whether or not it has an associated calendar. For example, a job could run automatically on the 1st and 15th of every month. The job could also be inserted into the schedule as needed throughout the month. When a calendar is not assigned, the job will only run when you manually insert the job into the schedule. Calendars are created from the Calendars pane and can be applied to a job or job group from the Job Definition or Job Group Definition dialog.

Job, File, Variable, and Time Dependencies

You can create job dependencies, file dependencies, variable dependencies and time dependencies for a job or job group.

A job dependency is a dependency on another job reaching a certain status. For example, you can set a job to run only after a previous job has completed normally.

A file dependency is a dependency on the existence of a file, on whether or not a file has been modified, or on the size of a file. This feature ensures that your jobs run when the appropriate data is available.

A variable dependency is a dependency on the value of a variable. You can use a preset TA variable or define your own. Variables can be used to track resource usage and can be updated automatically using variable update actions.

A time dependency sets a specific time window for the job to run.

InterMaster Dependencies

You can implement interMaster dependencies by combining different TA tools and areas of functionality, such as job events. The methodology can be extended to apply to dependencies between any number of jobs on any number of Masters. For more information on interMaster dependencies, see InterMaster Dependencies.

Job Events

Job and job group definitions can include exception-based actions triggered by job events. These actions can provide different types of notification as well as perform some additional task(s) in response to certain conditions. For example, a job event could be defined for a job that completes abnormally or when a job stops. You can associate one or more job events with a job. You can also assign multiple actions to each event.

System Events

TA also watches for system events, such as when the Master stops or when the production schedule finishes compiling. Job and system events can then trigger actions in response to the event condition. Job events can also be defined at a system level so that any job will trigger a set of actions when a particular job event occurs. For example, TA can email you if any job runs longer than a predefined maximum allowable time.

Actions

TA can respond to events using multiples and combinations of these action types:

  • Email – Notifies you and others of your job event via email.

  • SNMP – Sends SNMP messages to SNMP managers that monitor network activity.

  • Alert – Sends an alert message to the operator’s console regarding the event.

  • Job Control – Performs job control, such as stopping an active job, or rerunning a job that completed abnormally.

  • Job – Inserts a new job into the production schedule, such as a recovery task.

  • Log – Posts a user-defined message to the TA log and, if desired, to the Windows application log. You can then view the logs through the TA Logs pane and the Event Viewer.

  • Variable – Updates a user-defined variable to help you control when and how jobs run, or what email messages convey.

Job Scheduling Options

As part of scheduling a job, you can specify the criteria, which further determine how your job will run:

  • Enable or disable jobs and job groups, allowing or preventing them from being added to the production schedule.

  • Assign job priority so that higher priority jobs run first, before lower priority jobs, in a busy production schedule.

  • Determine how to handle a job that is ready to run when a previous instance of the job is already running.

  • Require the operator to release a job before it can run. You can include operator instructions for the operator to follow prior to releasing the job.

  • Automatically save and view the job’s output.

  • Specify the length of time to keep job history.

  • Keeps internal notes and descriptions about the job.

Jobs and Job Groups Panes

TA shows all of your jobs in the Job Definition panes. A job consists of instructions that are associated with carrying out an automatic task such as running a script or program. Jobs and job groups can be owned by a user or workgroup. When you own a job, only you have access to its properties. No other user can edit or view your job. If a job is owned by a workgroup, anyone belonging to that workgroup has access to it, within the limits of their security policy. For more information about security policies, see Security Policy Configuration Procedures.

Job History

As each job enters the production schedule, an instance of that job is created. TA allows you to view the history of each job instance. Job history lists the time and date when each instance ran and its resulting completion status. Job history can be retained for a specified length of time. For example, you could define your job to save the history of all its instances within the past 90 days. All instances that were run prior to this period are purged automatically.

Automatic Job Compilation

While jobs can be added manually to a schedule that is running at the time, in an ad-hoc fashion, the more common method is to use automatic compilation. Automatic compilation works by assigning a calendar to a job to designate on which dates a job should run. This method creates a production schedule for each day by adding jobs and job groups according to the calendars associated with each job's definition.

The production schedule contains scheduling data for a limited number of future and past days. At midnight every night, TA automatically compiles the next leading day (day out) of the production schedule and purges data for the trailing day (the oldest date in the schedule). The purging is done on a different thread than the production work so that production can continue throughout the purging process. The production schedule date range is determined based on various settings found in the System Configuration and Job/Job Group Definition dialogs. These settings include:

  • Job Retention History – Defines how long to keep job instances for each job. There is a system setting on the Defaults tab of the System Configuration dialog to configure that default setting for retaining job history. However, the period of time to retain a job's history can also be configured on an individual basis for each job. This setting (History Retention) is located on the Options tab of the Job Definition dialog of each job in question. You can have different settings for different jobs. For example, Job A could have instances kept for 30 days, while Job B could have instances kept for 60 days.

  • Future days to include in Schedule – Defines how far in the future to schedule jobs. For example, if this setting is set to three days and today is June 1st, you can view jobs in the production schedule out to June 4th. Set the number of future days to include in the production schedule on the Master tab of the System Configuration dialog.

    When you create a job or job group with a calendar that specifies a date within the current production schedule range and click OK, the Effective Date dialog displays to choose the date to begin running the job or job group. Each day, the automatic compilation process compiles incrementally for the next leading day, checking the calendar information of each job or job group to see if it should run on that day.

    Note: Because only the leading date of the production schedule (or forecasted schedule) is compiled automatically, if you choose not to schedule a job or job group using the Effective Date dialog, the job or job group will only enter the schedule on the next compile. For example, if Job D is scheduled to run daily, you can choose Cancel from the Effective Date dialog, and if the production schedule presently spans11/1/04 to 11/5/04, the job or job group will be scheduled to run starting on 11/6/04.

    If you want to remove the job from the production schedule, you can disable the job from the Jobs pane by right-clicking it and selecting the Enable option from the context menu. You can also right-click it from the Job Activity pane and click Remove Job(s) from schedule option from the context menu.

  • Automatic Daily History Cleanup – If you are automatically creating production schedules, you should also automatically clean up the production history. This option ensures that the TA automatically purges outdated job history data from the system database. When it is time to purge data (by default at the beginning of the production day), TA looks for all job information that is older than the retention history settings. If this option is not selected, you must manually purge the database of outdated job information at regular intervals by scripting.