Apache Airflow Archives - TantusData https://tantusdata.com/insights-categories/apache-airflow/ That uncovers wisdom. Thu, 30 Jan 2025 13:43:39 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.1 https://tantusdata.com/app/uploads/2023/01/cropped-Favicon-32x32.png Apache Airflow Archives - TantusData https://tantusdata.com/insights-categories/apache-airflow/ 32 32 Monitoring Airflow jobs with TIG 1: system metrics https://tantusdata.com/insights/monitoring-airflow-jobs-with-tig-1-system-metrics/ Tue, 16 Apr 2024 10:25:34 +0000 https://tantusdata.com/?post_type=insights&p=1907 Like many server applications, Airflow can – and should – be monitored for metrics and logs. In this article, we will look into the former and the integration with the TIG stack. The goal is to pull example metrics into a time series database and visualize it in a web application. This article will focus […]

The post Monitoring Airflow jobs with TIG 1: system metrics appeared first on TantusData.

]]>

Like many server applications, Airflow can – and should – be monitored for metrics and logs. In this article, we will look into the former and the integration with the TIG stack. The goal is to pull example metrics into a time series database and visualize it in a web application. This article will focus on AF system metrics, like its performance, load or health. We will cover the topic of reporting the data metrics into the same database in an upcoming article, “TITLE”. So, stay tuned. 

Why?

The Airflow itself finally got a Cluster status dashboard built-in. Therefore, it is valid to question the need for a separate dashboard that requires additional effort and maintenance. Why bother, then? There are several possible reasons to introduce a separate monitoring stack:

  • Aggregated view: looking into one app is easier than browsing through 10 of them. TIG (or any other monitoring stack) can be used for monitoring multiple instances and apps and is easily integrated with custom ones.
  • Security: there is no need to give access to the AF itself (or any other application) to be able to see the metrics and pinpoint errors. It is aligned with the data mesh and any other data democratization approach.
  • Support friendly: your 1st line support has a starting point to check the status of the data processing and be able to call the right people in case of a problem.

The toolkit

Out of the box (but with the right pip packages) Airflow supports sending its internal metrics to a statsd server. We can leverage that and set up such a server via Telegraf to proxy the metrics further into a time series database: InfluxDB. As a convenient UI, Grafana can be used.

TIG stack configuration

Firstly, let’s configure the TIG stack to accept the metrics:

  1. Install an InfluxDB instance. Nothing fancy here.
  2. Install the Telegraf. In the configuration, look for the statsd input (which has to be enabled) and InfluxDB output. The latter will require at least authentication. Note that the Telegraf supports many more inputs and allows monitoring of, e.g. system resources (CPU, memory, disk space, etc.). This is a good idea to configure in a production environment.
  3. Install Grafana and configure the data source to point it to the InfluxDB.

We have prepared a docker compose-based demo in the GitHub repository with a pre-configured environment. You can see there the example configuration for Telegraf (telegraf.conf), InfluxDB (influxdb.env) and Grafana (grafana-provisioning/datasources). Please note that this is only for dev/demo purposes, and real production environments need to be set up more securely (including the use of HTTPS and secure password handling).

Airflow

On the Airflow side, there are two significant points to be addressed.

  1. Airflow requires the Apache-airflow [statsd] package to have the statsd client available – you can do it via pip.
  2. In the airflow configuration file (usually: Airflow.cfg), the [metrics] section must be configured to enable the statsd, point it to the Telegraf instance and prefix the metrics (very useful in case of multiple Airflow instances).

There are several metrics available to be sent in Airflow. You can see the complete list in the Airflow documentation here. There, you can also limit which ones are actually reported to the Telegraf. Be aware that even though most of the metrics are said to be reported in seconds, you need to validate them yourself (as in this example).

Dashboards

The last step is to configure Grafana and set up some dashboards. The UI offers a WYSIWYG editor that you can use to tailor it to your needs. The example available on the GitHub might serve as a starting point, as it shows:

  • state of the Airflow executors (queues and tasks being run at the moment) to see whether any processing is going on,
  • state of the pools (default and the custom ones) to check potential bottlenecks if you use multiple pools,
  • task times to identify unexpected stragglers (and compare instances’ run times and find any performance challenges early on).
An example dashboard, available on our GitHub. It contains metrics useful to monitor the load on the Airflow instance and identify potential overload or bottlenecks in the data pipeline processing.

Summary

In conclusion, for Airflow monitoring, you can use specialized metrics tools like TIG stack and aggregate all the metrics from multiple AF instances. The stack can accommodate AF system metrics and data from other applications, including your custom ones. An example of sending data quality metrics is what we will look into in the second part. There is a demo environment on our GitHub to see whether this works for you.

The post Monitoring Airflow jobs with TIG 1: system metrics appeared first on TantusData.

]]> Databricks – Photon https://tantusdata.com/insights/databricks-photon/ Tue, 02 Apr 2024 12:52:56 +0000 https://tantusdata.com/?post_type=insights&p=1903 The Databricks platform offers two execution engines for the clients: the standard Apache Spark (available as an open-source application) and one with Photon enhancement that brings a performance improvement (as well as extra pricing). Have you ever wondered where this speedup comes from and how it affects designing Apache Spark jobs? This article is based […]

The post Databricks – Photon appeared first on TantusData.

]]>

The Databricks platform offers two execution engines for the clients: the standard Apache Spark (available as an open-source application) and one with Photon enhancement that brings a performance improvement (as well as extra pricing). Have you ever wondered where this speedup comes from and how it affects designing Apache Spark jobs?

This article is based on Berlkey’s paper on Photon by Databricks people, as that publication is the closest to the source, as the Photon engine is not an open-source project, and its source code is not available to the public.

The general idea

In a nutshell, Photon replaces the standard (bundled) query engine available in Apache Spark, using the same API. For some operations, mostly CPU-heavy ones, the Catalyst optimizer may decide to send the job to Photon instead of using the default execution path, all for performance reasons. In other words, it is an alternative way to execute Spark DAG tasks, not skipping/reordering/reorganizing the data engine.

SIMD

The SIMD stands for Single Instruction, Multiple Data and is one of the ground ideas for job optimisations in Photon. With current architectures running the same operation on multiple instances (values, rows, etc.), data can be optimised via vectorisation, even within a single thread. Photon is said to utilise those optimisations.

C++ and Code generation

The Photon engine is implemented in C++ instead of the JVM native languages (Scala, Java or others). The source paper points to performance reasons, including ‘hitting performance ceilings’. The communication with the rest of the Apache Spark is implemented via JNI. Databricks’ internal benchmarks indicate that the performance hit due to moving data in and out of JVM is not noticeable. 

Internal data format

Photon engine uses columnar data representation (same as, e.g. Parquet data format) instead of row data (like the rest of Apache Spark). This is due to SIMD optimizations – kernel implementation that works best on columnar data. The memory management (calling, freeing, etc) is still done via Apache Spark’s memory manager. The data is kept off-heap, so transferring from Photon to Spark does not require copying the data.

When a shuffle operation is necessary, Photon writes a shuffle file and uses Spark API to execute the exchange. However, the data format is not compatible with vanilla spark, and a Photon shuffle read must follow the Photon shuffle write.

When does it help?

Photon is meant to address the CPU-heavy loads. This includes joins (especially hash join) and aggregations. On the other hand, being a non-JVM implementation, Photon obviously does not support UDFs or RDD API. Exact benchmarks and precise speedups are mentioned in the source paper.

Sources

I hope this helps. Moreover, if you know any other good sources, do let us know on social media so that everyone can see them.

The post Databricks – Photon appeared first on TantusData.

]]> Airflow — pools and mutexes. https://tantusdata.com/insights/airflow-pools-and-mutexes/ Tue, 19 Mar 2024 13:47:32 +0000 https://tantusdata.com/?post_type=insights&p=1898 Although the ideal data pipeline is made of idempotent and independent tasks, there are some cases when setting up a mutex (a.k.a. part of the job that cannot be run concurrently) is necessary. Fortunately, Airflow supports such cases and offers a few tools varying by complexity to implement such a pipeline. In this article, we […]

The post Airflow — pools and mutexes. appeared first on TantusData.

]]>

Although the ideal data pipeline is made of idempotent and independent tasks, there are some cases when setting up a mutex (a.k.a. part of the job that cannot be run concurrently) is necessary. Fortunately, Airflow supports such cases and offers a few tools varying by complexity to implement such a pipeline.

In this article, we will look at the following DAG in AF. The graph itself is relatively simple; the catch is that load_1 and load_2 operators cannot have concurrently running task instances. We will look at treating loads separately and looking at them as a group.

Therefore, there are three scenarios we will look into:

  1. Only one instance of the operator can be running.
  2. One instance is that the operator cannot run before the older ones are successful.
  3. Only one instance of a group of operators.

The examples are using annotation syntax for Airflow. Still, the concept stays the same for the 1.0 compatible approach – instead of annotation params, use them in any Operator class constructor.

Mutex on an operator

The first scenario is that only one operator instance can be running at a time. If there are multiple runs ready to be scheduled, it does not matter which one goes first. One solution would be using the max_active_tis_per_dag option with the value of 1. 

@task(
   max_active_tis_per_dag=1
)

Dependency on past runs

The second example assumes that the nth batch cannot start before the nth-1 one is completed successfully (or at least marked so in Airflow). For this use case, AF offers depends_on_past flag. In this case, you have to be careful and pay some attention to the state of the latest runs. One failed, upstream-failed, or waiting task can halt all future DAG runs.

@task(
   depends_on_past=True
)

Pools – mutex across multiple operators

The most complex approach is required if we need to group multiple operators to make them share a mutex. One way to do it is to put them into one custom pool and limit it to accommodate only one task simultaneously – either by setting a pool with 1 slot or assigning a high number of required slots to each operator.

load_pool = Pool.create_or_update_pool(
   name="load_pool",
   slots=1,
   description="Pool for data load tasks."
)

@task(
   pool=load_pool.pool
)

Source

The source code for all examples and the docker environment to run them is available on GitHub.

The post Airflow — pools and mutexes. appeared first on TantusData.

]]> Passing information between DAGs in Airflow. https://tantusdata.com/insights/passing-information-between-dags-in-airflow/ Thu, 08 Feb 2024 13:34:54 +0000 https://tantusdata.com/?post_type=insights&p=1893 There are data pipelines where you must pass some values between tasks – not complete datasets, but ~ kilobytes. This can be managed even within the Airflow itself. As always, multiple options are available – let’s review some of them. In this article, we are looking at sharing data between DAGs, which are connected via […]

The post Passing information between DAGs in Airflow. appeared first on TantusData.

]]>

There are data pipelines where you must pass some values between tasks – not complete datasets, but ~ kilobytes. This can be managed even within the Airflow itself. As always, multiple options are available – let’s review some of them.

In this article, we are looking at sharing data between DAGs, which are connected via run dependencies. Let’s assume that each DAG needs to be run daily, and the first DAG generates some important data for the second DAG. 

XCom

XCom would be the first and the recommended approach. It works well with out-of-the-box features like ExternalTaskSensor:

with DAG(
       dag_id="xcom-sink",
       schedule_interval="@daily",
       start_date=datetime(2023, 7, 1),
       catchup=True,
) as xcom_sink:
   ExternalTaskSensor(
       task_id="wait-for-dependency",
       external_dag_id="xcom-source",
       external_task_id="update-hive-table-events-triggers"
   ) >> BashOperator(
       task_id="show-xcom",
       bash_command="echo {{ ti.xcom_pull(dag_id='xcom-source', task_ids='update-hive-table-events-triggers') }}"
   )

XCom vs run id

The XCom is identified by the DAG ID, task ID and run ID. If you want to run a single DAG with a custom run ID, you have to ensure there is an XCom value for that run ID already created. This complicates issuing manual runs.

When not to XCom?

Rather than discussing cases that are tailored to use XCom, let’s focus on examples that are not well supported. Basically, XCom does not work well with Datasets, and you might get some quirky results here: https://github.com/apache/airflow/discussions/33069. 

With datasets, you need to refer to the last past value of XCom, effectively losing all benefits of tight coupling. Using that parameter, you will need to deal with race conditions: if the not-latest sink DAG is restarted, it will receive an incorrect value from the source. Moreover, let’s consider a design with multiple sources and a single sink:

Suppose the events dataset gets updated twice and the users’ dataset only once. In that case, the events-with-users will receive only the latest value of the former one. It is up to you to decide whether this behaviour is expected or unacceptable.

Variables

In rare cases, you can use Variables from Airflow to solve the issue. Variables are built-in mechanisms in Airflow that provide a global state (or configuration) for all DAGs to read and write. You can use it to implement the Registry code pattern:

  1. Put a hashmap of the date of run -> metadata into a variable
  2. Use the execution date (named logical_date in the newer versions) as the hashmap key.
  3. Use PythonOperator to update the variable. Read can be done by either Python code or templating.
  4. Use sensors or datasets to schedule DAGs in the correct order.

Beware!

Before you go with the variable route, please keep in mind that compared to XComs, variables are way more costly to maintain. You might need to keep track of the variables’ sizes, implement error handling in your PythonOperators and be mindful of any unsolicited changes in the variables’ values.

External system?

There is always an option of using an external data service to synchronise, similarly to using built-in variables. This may come in the form of data paths on HDFS / S3 / other storage, batch load dates in the database or such. However, this solution creates an inferior design, as you would end up with disadvantages of the variables approach and new implicit dependencies between DAGs and external services.

Summary

XCom + datasetXCom + sensorVariablesExternal
+ no tight coupling between DAGs+ supports 1:1 task run relationships+ works with both sensors and datasets+ works even with 3rd party systems
– race conditions,

– no guarantee for 1:1 run relationships		– a bit tighter coupling

– for manual run one must supply by hand the exec_date		– handles only data sharing, does not handle orchestration

– requires way more effort than XCom		– requires even more effort

– hidden dependencies outside AF

– complicated architecture
Comparison

Source

The source code for both XCom and Variable examples and the docker environment to run them is available on GitHub.

The post Passing information between DAGs in Airflow. appeared first on TantusData.

]]> Managing inter-DAG dependencies in Airflow https://tantusdata.com/insights/managing-inter-dag-dependencies-in-airflow/ Fri, 01 Sep 2023 11:00:00 +0000 https://tantusdata.com/?post_type=insights&p=1731 In the real world, data pipelines only sometimes come as a completely independent sequence of operations. Usually, they share dependences on one another, occasionally easy 1:1 ones, sometimes more complicated. Here is a short list of what Apache Airflow has to offer for handling those relationships. The general setting We will look into a situation […]

The post Managing inter-DAG dependencies in Airflow appeared first on TantusData.

]]>

In the real world, data pipelines only sometimes come as a completely independent sequence of operations. Usually, they share dependences on one another, occasionally easy 1:1 ones, sometimes more complicated. Here is a short list of what Apache Airflow has to offer for handling those relationships.

The general setting

We will look into a situation where parent data pipelines (called DAGs in Airflowish) create data used (consumed) by children pipelines. The dependency looks like this:

The silent assumption is that we cannot just look at the metadata (file creation time, last row’s created timestamp or alike), as it might show that data is being loaded but not exactly fully loaded – therefore, we are waiting for the entire dataset to be available.

External tools

Using an external tool is always available, not only for Airflow data pipelines. The most straightforward implementation would be using an 0-byte file on NFS, HDFS or any other shareable filesystem to mark success for a given dataset.  

This example is based on HDFS, but you could also use:

  • NFS,
  • FTP/SFTP location,
  • metadata stored in a SQL or NoSQL database,
  • any other accessible non-airflow related data storage.

Source code

For the current (4.1.0) version of the apache-airflow-providers-apache-hdfs, there is no HDFS operator; only sensors are available. Therefore a bit of walkaround (e.g. BashOperator) has to be used to create a marker file.

from __future__ import annotations


from datetime import datetime


from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.apache.hdfs.sensors.web_hdfs import WebHdfsSensor
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor


HDFS_CONNECTION_ID = "hdfs-14"
SFTP_CONNECTION_ID = "sftp-21"


with DAG(
	dag_id="external-events",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events:
	SFTPSensor(
    		task_id="wait-for-csv",
	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    		sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="csv-to-parquet",
    		application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/events/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_users:
	SFTPSensor(
    		task_id="wait-for-csv",
	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    		sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="csv-to-parquet",
    		application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/users/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-events-with-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events_with_users:
	[
    	WebHdfsSensor(
        	task_id="wait-for-marker-events",
        	filepath="/marker/events/{{ dag_run.logical_date | ds }}.success",
        	webhdfs_conn_id=HDFS_CONNECTION_ID
    	),
    	WebHdfsSensor(
        	task_id="wait-for-marker-users",
        	filepath="/marker/users/{{ dag_run.logical_date | ds }}.success",
        	webhdfs_conn_id=HDFS_CONNECTION_ID
    	)
	] >> SparkSubmitOperator(
	    	task_id="compute-events-with-users",
    		application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/events-with-users/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-reports-1",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_reports_1:
	WebHdfsSensor(
	    	task_id="wait-for-events-with-users",
    		filepath="/marker/events-with-users/{{ dag_run.logical_date | ds }}.success",
	    	webhdfs_conn_id=HDFS_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="compute-report",
    		application="/spark/applications/compute-report-1.py"
	)


with DAG(
	dag_id="external-reports-2",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_reports_2:
	WebHdfsSensor(
    		task_id="wait-for-events-with-users",
	    	filepath="/marker/events-with-users/{{ dag_run.logical_date | ds }}.success",
    		webhdfs_conn_id=HDFS_CONNECTION_ID
	) >> SparkSubmitOperator(
    		task_id="compute-report",
	    	application="/spark/applications/compute-report-2.py"
	)

This is a helpful technique if Airflow is to be integrated with other tools. On the other hand, the dependency is very implicit. It must be refactored carefully to maintain the contract between data pipelines.

Airflow datasets

Since version 2.4, Airflow has offered data-aware scheduling based on the concept of datasets. The general idea is:

1. Operators define datasets to which they publish data. The dataset in Airflow is just metadata – AF does not handle the data itself.

2. DAGs specify datasets that they depend on. Each time all dependency datasets are updated (once or many times), the dependent DAG is triggered. This setting replaces the time-based schedule.

Although this feature does not give any control over the time of the execution nor tie specific runs (there is no guarantee that DAG gets run on every separate dependency run – several runs may get squashed into a single dependent run), a user receives out-of-the-box a friendly UI for browsing datasets and DAGs relationships. 

Source code

from __future__ import annotations


from airflow import DAG, Dataset
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from pendulum import datetime




dataset_users = Dataset("af://datasets/users")
dataset_events = Dataset("af://datasets/events")
dataset_events_with_users = Dataset("af://datasets/events-with-users")
dataset_reports_1 = Dataset("af://datasets/reports-1")
dataset_reports_2 = Dataset("af://datasets/reports-2")


SFTP_CONNECTION_ID = "sftp-21"


with DAG(
	dag_id="dataset-events",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule="@daily",
	max_active_runs=1,
) as dag_producer_events:
	SFTPSensor(
        	task_id="wait-for-csv",
    	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_events]
	)


with DAG(
	dag_id="dataset-users",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule="@daily",
	max_active_runs=1,
) as dag_producer_users:
	SFTPSensor(
        	task_id="wait-for-csv",
    	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_users]
	)


with DAG(
	dag_id="dataset-events-with-users",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events, dataset_users],
	max_active_runs=1,
) as dag_processor_events_with_users:
	SparkSubmitOperator(
    	    	ask_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_events_with_users]
	)


with DAG(
	dag_id="dataset-reports-1",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events_with_users],
	max_active_runs=1,
) as dag_processor_reports_1:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py",
    	    	outlets=[dataset_reports_1]
	)


with DAG(
	dag_id="dataset-reports-2",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events_with_users],
	max_active_runs=1,
) as dag_processor_reports_2:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py",
    	    	outlets=[dataset_reports_2]
	)

Trigger the external DAG

Suppose the functionality of the datasets is too limited or an older version of the AF is used. In that case, there are alternative ways to orchestrate the data pipelines. TriggerDagRunOperator represents the push approach. The idea behind this operator is to trigger a run of a specified DAG with the option of supplying custom parameters. Obviously, this feature does not allow to create a dependency on more than one source dataset. However, one dataset can still run multiple dependents.

Since this approach uses the DAG name (id) to identify the dependent pipeline, you must be careful when refactoring those names. One way to look up the dependent (and depending) DAGs and tasks is to use the Browse > DAG Dependencies tab that shows a graph of trigger and sensor relationships.

Source code

from __future__ import annotations


from datetime import datetime
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.utils.task_group import TaskGroup
from airflow.utils.trigger_rule import TriggerRule




SFTP_CONNECTION_ID = "sftp-21"




with DAG(
	dag_id="trigger-events-with-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events_with_users:
	with TaskGroup(group_id="gather-events") as group_events:
    	SFTPSensor(
        	task_id="wait-for-csv",
        	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
        	sftp_conn_id=SFTP_CONNECTION_ID
    	) >> SparkSubmitOperator(
        	task_id="csv-to-parquet",
        	application="/spark/applications/csv-to-parquet.py"
    	) >> SparkSubmitOperator(
        	task_id="update-hive-table",
        	application="/spark/applications/update-hive-table.py"
    	)


	with TaskGroup(group_id="gather-users") as group_users:
    	SFTPSensor(
        	task_id="wait-for-csv",
        	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
        	sftp_conn_id=SFTP_CONNECTION_ID
    	) >> SparkSubmitOperator(
        	task_id="csv-to-parquet",
        	application="/spark/applications/csv-to-parquet.py"
    	) >> SparkSubmitOperator(
        	task_id="update-hive-table",
        	application="/spark/applications/update-hive-table.py"
    	)


	[group_events, group_users] >> SparkSubmitOperator(
    	    	task_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	) >> [
    	    	TriggerDagRunOperator(
        	    	task_id="trigger-reports-1",
    	        	trigger_dag_id="trigger-reports-1",
        	    	trigger_rule=TriggerRule.ALL_SUCCESS,
    	    	),
    	    	TriggerDagRunOperator(
        	    	task_id="trigger-reports-2",
    	        	trigger_dag_id="trigger-reports-2",
           	 	trigger_rule=TriggerRule.ALL_SUCCESS,
    	    	)
	]


with DAG(
	dag_id="trigger-reports-1",
	catchup=True,
	start_date=datetime(2023, 7, 1),
) as dag_processor_reports_1:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py",
	)


with DAG(
	dag_id="trigger-reports-2",
	catchup=True,
	start_date=datetime(2023, 7, 1),
) as dag_processor_reports_2:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py",
	)

The sensor for external DAG

Using the sensor is exactly the opposite of triggering some other DAG. In this approach, we tell DAG to pause and wait until some other DAG (or a specific task in the DAG) is completed. We still need to specify appropriate schedules for both pipelines, however.

The sensor approach is very flexible. Unlike the datasets approach, each sensor instance can wait for a specific DAG/task/execution date combination, which allows waiting with specific time offset or model aggregation relationships (hourly to daily, daily to weekly, etc.). Unlike the triggering approach, a DAG can model waiting for all preconditions with the sensor approach.

Precisely, like when using DAG triggering, sensor dependency can be checked on the Browse > DAG Dependencies tab. 

Source code

from __future__ import annotations


from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.sensors.external_task import ExternalTaskSensor




SFTP_CONNECTION_ID = "sftp-21"




with DAG(
    	dag_id="sensor-events",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as sensor_events:
	SFTPSensor(
    	    	task_id="wait-for-csv",
    	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensor-users",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as sensor_users:
	SFTPSensor(
    	    	task_id="wait-for-csv",
    	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensor-events-with-users",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_daily:
	[
    	    	ExternalTaskSensor(
         	   	task_id="wait-for-events",
        	    	external_dag_id="sensor-events",
    	        	check_existence=True
    	    	),
    	    	ExternalTaskSensor(
         	   	task_id="wait-for-users",
        	    	external_dag_id="sensor-users",
    	        	check_existence=True
    	    	)
	] >> SparkSubmitOperator(
    	    	task_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensors-reports-1",
    	schedule="@daily",
    	start_date=datetime(2023, 7, 1),
) as dag_reports_1:
	ExternalTaskSensor(
        		task_id="wait-for-events-with-users",
    	    	external_dag_id="sensor-events-with-users",
    	    	check_existence=True
	) >> SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py"
	)


with DAG(
    	dag_id="sensors-reports-2",
    	schedule="@daily",
    	start_date=datetime(2023, 7, 1),
) as dag_reports_2:
	ExternalTaskSensor(
    	    	task_id="wait-for-events-with-users",
    	    	external_dag_id="sensor-events-with-users",
    	    	check_existence=True
	) >> SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py"
	)

Short comparison

ExternalDatasetTriggerSensor
Multiple triggers for a single DAGYesNoYesYes (although a bit obscure)
Multiple requirements to run a single DAGYesYesNoYes
Dependencies on different time intervalYesNoNoYes
Strong pointsIntegration with 3rd party tools.Good lineage support in the UI.Push scheduling.Push scheduling.Allows execution time manipulation (daily to weekly DAGs).
Weak pointsSensitive to refactoring.No control over schedule intervals.Weak error handling. Any DAG can depend on only up to one DAG.Pull scheduling.
The options compared

Example scenarios

The end goal of running Airflow is to orchestrate an actual pipeline, so let’s discuss some actual-life examples where DAG dependencies are necessary.

Breaking down complex DAG

First, DAG is an example of a complex DAG used to calculate multiple outputs (exports) from some external inputs. For clarity, all operations are split into waiting (sensors), preprocessing and reports computation. The screenshot shows them in the form of a Graph view from Airflow.

The assumption is that all operations consume whole input datasets – they are not restricted to, e.g. current hour only. Otherwise, using sensors or external markers would be the right option.

This situation is where datasets (or, according to the docs – data-aware scheduling) show good results. For each branching point, we define an outlet – abstraction over the dataset produced by each operator (remember that it is only the metadata – handling the actual data is not done by Airflow and never should be). Then we define each chain of operations that takes those datasets as a separate DAG with datasets defined as the schedule. As a result, the DAGs are short & simple, and we got a bit of data lineage in the Datasets tab of Airflow for free:

Source code

from __future__ import annotations


from datetime import datetime


from airflow import Dataset
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator
from airflow.sensors.bash import BashSensor


dataset_clients = Dataset("af://dataset/clients")
dataset_events = Dataset("af://dataset/events")
dataset_transactions = Dataset("af://dataset/transactions")
dataset_users = Dataset("af://dataset/users")


with DAG(
	dag_id="complex-dag-clients",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_clients:
	sensor_clients = BashSensor(task_id="wait-clients", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4", outlets=[dataset_clients])


	sensor_clients >> op_preprocess_step_1 >> \
    	[op_preprocess_step_2 >> op_preprocess_step_3] >> \
    	op_preprocess_step_4


with DAG(
	dag_id="complex-dag-events",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_events:
	sensor_events = BashSensor(task_id="wait-events", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3", outlets=[dataset_events])


	sensor_events >> op_preprocess_step_1 >> op_preprocess_step_2 >> op_preprocess_step_3


with DAG(
	dag_id="complex-dag-transactions",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_transactions:
	sensor_gateway_01 = BashSensor(task_id="wait-gateway-01", bash_command="true")
	sensor_gateway_02 = BashSensor(task_id="wait-gateway-02", bash_command="true")
	sensor_gateway_03 = BashSensor(task_id="wait-gateway-03", bash_command="true")
	sensor_gateway_04 = BashSensor(task_id="wait-gateway-04", bash_command="true")
	sensor_gateway_05 = BashSensor(task_id="wait-gateway-05", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4")
	op_preprocess_step_5 = EmptyOperator(task_id="preprocess-step-5", outlets=[dataset_transactions])


	[
    	sensor_gateway_01,
    	sensor_gateway_02,
    	sensor_gateway_03,
    	sensor_gateway_04,
    	sensor_gateway_05
	] >> op_preprocess_step_1 >> op_preprocess_step_2 >> op_preprocess_step_3 >> \
    	op_preprocess_step_4 >> op_preprocess_step_5


with DAG(
	dag_id="complex-dag-users",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_users:
	sensor_users = BashSensor(task_id="wait-users", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4")
	op_preprocess_step_5 = EmptyOperator(task_id="preprocess-step-5")
	op_preprocess_step_6 = EmptyOperator(task_id="preprocess-step-6", outlets=[dataset_users])


	sensor_users >> [op_preprocess_step_1, op_preprocess_step_2] >> op_preprocess_step_3 >> \
    	[op_preprocess_step_4, op_preprocess_step_5] >> op_preprocess_step_6


with DAG(
	dag_id="complex-dag-report-clients",
	schedule=[dataset_clients, dataset_events],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_clients:
	op_rich_clients = EmptyOperator(task_id="enrich-clients")
	op_report_clients = BashOperator(task_id="report-clients", bash_command="sleep 3")


	op_rich_clients >> op_report_clients


with DAG(
	dag_id="complex-dag-report-users",
	schedule=[dataset_events, dataset_users],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_users:
	op_rich_users = EmptyOperator(task_id="enrich-events")
	op_report_users = BashOperator(task_id="report-users", bash_command="sleep 3")


	op_rich_users >> op_report_users


with DAG(
	dag_id="complex-dag-report-full",
	schedule=[dataset_events, dataset_transactions, dataset_users],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_full:
	op_report_users = BashOperator(task_id="report-full", bash_command="sleep 3")

Daily to weekly transition

This example covers any time-based aggregation of the data. Regardless of the actual time intervals, the general idea is that multiple runs of one DAG are used as input for the other one:

  • 24 runs of the hourly pipeline for daily one,
  • 7 runs of the daily DAG for the weekly one.

This setting limits available options to using an external tool or implementing DAG-run sensors. Let’s take a look at the latter one.

The idea behind this diagram is simple – the daily pipeline is unaffected by the relationship. At the same time, the weekly one starts with seven sensors of ExternalTaskSensor, each with a different execution date offset. Those have been waiting for daily DAGs from 1 to 7 days ago. Apart from that set of sensors, the DAG is similar to others.

Source code

from __future__ import annotations


from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.bash import BashSensor
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.utils.task_group import TaskGroup


with DAG(
    	dag_id="dag-daily",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_daily:
	BashSensor(task_id="some-input-wait", bash_command="true") >> \
    	BashOperator(task_id="some-data-processing-1", bash_command="sleep 3") >> \
    	BashOperator(task_id="some-data-processing-2", bash_command="sleep 3") >> \
    	BashOperator(task_id="exporting-to-hdfs", bash_command="sleep 3")


with DAG(
    	dag_id="dag-weekly",
    	schedule_interval="@weekly",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_weekly:
	# TaskGroup is purely optional, but makes DAG in the UI a bit clearer.
	with TaskGroup(group_id="dag-daily-sensors") as sensors:
    	[
        	ExternalTaskSensor(
            		task_id=f"wait-{days_offset}d",
	            	external_dag_id="dag-daily",
       		     	timeout=24 * 60 * 60,
            		mode="reschedule",
	            	execution_delta=timedelta(days=days_offset)
        	) for days_offset in range(1, 8)
    	]


	sensors >> \
    		BashOperator(task_id="some-data-processing", bash_command="sleep 3") >> \
	    	BashOperator(task_id="exporting-to-hdfs", bash_command="sleep 3")

The source code for all examples and the docker environment to run them is available on GitHub.

The post Managing inter-DAG dependencies in Airflow appeared first on TantusData.

]]>