Monitoring Airflow Jobs

Note: In the TA Job Activity pane, the Job No. column refers to the TA job number, not the Airflow Process Instance.

Double-clicking on a job listed in the Job Activity pane displays the Job Details dialog where the operator can examine various aspects of the job after it has completed and while it is still running.

Note: The external ID on this page refers to the process instance ID of the first request in the job for jobs with multiple processes.

Viewing the Status of an Airflow Job

The Status tab of the Job Details dialog contains the request with the variables used when this job was submitted. This tab allows you to provide last-minute overrides to parameter values or other process details prior to the initial run or rerun.

  • Status – Current status of the job.

  • Est. Start Time – Estimated job launch time based on its time dependencies and the launch times and estimated durations of preceding jobs. You can view all the dependent jobs on the Dependencies tab.

  • Act. Start Time – Time the job became active. Until the job is active, this field is empty.

  • Est. Duration – Estimated job run time. If the job has never run before, it contains the value in the Estimated Duration field in the job’s definition. Otherwise, it contains a historical average of the Actual Duration.

  • Act. Duration – Actual time the job took to run.

  • Job Owner – User that owns the job.

  • Scheduled By – How the job entered the production schedule. Possible values include:

Calendar: Compiled into the schedule based on an assigned calendar.

Event: Inserted by a job action based on an event trigger.

On Demand: Inserted by an operator using a TA Client, command line, API, or Web Service.

  • Exit code – Exit code indicating how the job completed.

    Note: This field is not displayed for a job group.

    Possible values include:

0: Job completed normally

1: Job completed abnormally

<empty>: Job still running

  • External ID – ID number assigned to the job process by the operating system.

  • Track Critical Path For Job – Whether the job is critical.

  • Reruns – Number of times the job has rerun for the day.

  • Disable Carryover – Whether the job can be carried forward to the next day’s schedule if the job does not run on its scheduled day.

Displaying Information About a Job Run

The Run Info tab of the Job Details dialog contains read-only information about this instance when the job was last run. The Run Info tab displays different information based on the Select Run Type option in the associated job definition:

  • For the Run DAG option, the Run Info tab displays information about the DAG Run and the Tree View of the DAG.

  • For the Run Separate Tasks option, the Run Info tab displays information about the Tasks that the job definition selected to run.

Run Information for a DAG Run

The DAG Run tab shows the IDs, execution date, and status for a job that runs a full Airflow DAG, as shown in the this example of a successful job run.

The Tree View lists the statuses of all tasks within a DAG job run. The list of tasks matches those specified in the corresponding job definition. See How to define an Airflow job.

When a job run fails, you can use this view to identify which tasks succeeded and which tasks failed. The example shows a DAG Run whose Status is FAILED. The Tree View for this job run shows the success or failure status for each DAG task, as shown below. In this example, the search_catalog task and its downstream tasks failed. By selecting a failed task, you can display additional information about the task.

Run Information for a Tasks Run

For jobs that run a subset of DAG tasks instead of a full DAG, the Tasks tab in the Run Info tab displays information for each of the tasks.

Displaying Airflow Job Output

As with any other TA job, you can view the output from an Airflow job in the Output tab of the Job Details dialog.

To view job output:

  1. Double-click on a job listed in the Job Activity pane to display the Job Details dialog.

  2. Click the Output tab. The Output tab displays the information that the job outputs through its processing. The illustration shows sample output from an Airflow job that ran a complete DAG. The job definition for the job instance chose to display the output as JSON.

Displaying Airflow Job Information Associated with Job Details

You can choose the Airflow Job tab to show the information from the job definition that is associated with a job’s activity. The tab shows the contents for a sample full DAG run job and a job run for selected tasks within the DAG.