Do you want to see the number of partitions? Or the partition size in rows from within the job? Or maybe you just like some low-level hacking? The RDD API is still there and accessible via the .rdd method. 

Why is RDD API still around?

Despite deprecating the RDD API, the engine of Apache Spark (at least its Open Source part – see our article on Photon), the RDDs are still used in its internals. It is also available via the .rdd method of Datasets, and, therefore, DataFrames as well. Keep in mind, though, that the number of actually useful operations not available from Dataset API is really low, and currently, excluding some low-level or Spark internals hacking, it boils down to partitions count and size checking – using getNumPartitions method or glom operator, respectively.

Check number of partitions

RDD API still keeps the getNumPartitions method for that use case:

println(inputData.rdd.getNumPartitions)

Check number of rows per partition

The glom function coalesces all rows in each partition into an array. It can be used to check the number of rows per partition. In the case of wide rows, consider using select to limit the number of columns—RDD are not optimized by most of Spark’s mechanisms.

inputData.rdd.glom().map(_.length).collect().foreach(println _)

The post RDD in Apache Spark appeared first on TantusData.

With the deprecation of the public use of the old good RDD API, Spark users are left with two options: typed Datasetsand untyped DataFrames (that are actually a specific case of Datasets). The API also allows users to freely cast one to another – e.g. using the .as[T] method to cast an untyped DataFrame to a Dataset[T]. It does not change the underlying data though and can result in surprising results if one is not aware of that.

What does .as[T] do?

Let’s start by looking at the source (code):

Returns a new Dataset where each record has been mapped on to the specified type. The method used to map columns depend on the type of U:
When U is a class, fields for the class will be mapped to columns of the same name (case sensitivity is determined by spark. sql. caseSensitive).
When U is a tuple, the columns will be mapped by ordinal (i. e. the first column will be assigned to _1).
When U is a primitive type (i. e. String, Int, etc), then the first column of the DataFrame will be used.
If the schema of the Dataset does not match the desired U type, you can use select along with alias or as to rearrange or rename as required.
Note that as[] only changes the view of the data that is passed into typed operations, such as map(), and does not eagerly project away any columns that are not present in the specified class.
Since:
1.6.0

The last note is crucial: casting to a Dataset does not change the underlying data. Any columns not present in the T (e.g., columns without a corresponding field in the case class) will not be discarded.

Why bother?

There are a few situations where having extra columns may be surprising and create problems with a job run (or even worse – silently introduce data quality issues):

• running union or unionAll transformations on non-aligned data,
• calling distinct (it will check for hidden columns’ uniqueness as well),
• saving data (will include extra columns).

A defensive version of .as[T] 

The simple version of a defensive (meaning: adjusting the schema to the provided domain class) would be one with a .select() transformation call:

case class Artist(id: String, name: String, location: String)

def toArtistsDefensive(input: DataFrame): Dataset[Artist] = { input
  .select("id", "name", "location")
  .as[Artist]
}

This is a very DRY-unfriendly implementation, as each modification of the Artists class requires searching for all related select instances and updating them. Fortunately, with a bit of reflection, it can be refactored into a generic solution. This generic transformation will trim the Dataset to contain only the expected columns.

import scala.reflect.runtime.universe._

def toTDefensive[T <: Product: TypeTag](input: DataFrame): Dataset[T] = { 
  val caseClassFields = typeOf[T].members
    .collect { case m: MethodSymbol if m.isCaseAccessor => m.name.toString }
    .toSeq
  
  val columns = caseClassFields
    .map(F.col _)
    .reverse

  input
    .select(columns: _*)
    .as[T]
}

The post Datasets and DataFrames appeared first on TantusData.

In the first article on Monitoring Airflow jobs with TIG, “System Metrics”, we have seen an example of Airflow installation with a TIG stack set up to monitor it. To fully utilize this stack, we should enrich the raw system metrics with statistics on the processed data. Without this, the metrics would tell if the data pipelines are doing anything but not whether they are working on the correct data.

What to look for?

What can be realistically monitored is a pretty deep topic without a one-fits-all answer. The safe starting point is to look for the size of the data, duplicates (or unique rows), null/missing columns’ values and basic aggregates (count per some enumerated type or min/max values). The nice part of this issue is it is not limited by any software, and you can report any numeric value into an InfluxDB database.

Equally important is not limiting the monitoring to the output of the whole pipeline only. Being able to check the data volume and basic traits on the input and in intermediate steps is crucial, as it enables one to check, identify and react to problems early on (and avoid painful backtracking and recomputing of the whole pipeline).

Computing the metrics

Technically speaking, such monitoring requires two things in the pipeline: a code (or job) to compute the metric and a wrapper to send it to the metrics database. We did not cover the former here – it may vary from a simple SQL query run via Hive / Impala to a side output of a Spark job.

Storing the data for graphs

The second part to be done in Airflow is sending the data to the database. At the time of writing this article, the built-in InfluxDB connector allows only querying the database. Please see our demo (especially the plugins directory) for an example implementation of InfluxDB write. You can also use the REST API or BashOperator to call the influx command there.

Merging both steps or not?

Both compute and send metrics steps may be squashed into a single bash step instead of scheduling them separately and stitching them via XComs. However, the more complicated or time-consuming the calculation may be, the better the separated approach would seem. This is a decision for you to make; we provide an example of the former approach.

Summary

After the first step, the example stack has monitoring of the system, and an operator can see whether the system is working and does not have an overload or some kind of bottleneck. This step adds a base monitoring of the data quality. Adding those on multiple points in the data pipeline will also enable verification during the processing – in a centralized place (or, in this case, WebUI). Once again, a development/demo environment is available on our GitHub.

The post Monitoring Airflow jobs with TIG 2: data quality metrics appeared first on TantusData.

Like many server applications, Airflow can – and should – be monitored for metrics and logs. In this article, we will look into the former and the integration with the TIG stack. The goal is to pull example metrics into a time series database and visualize it in a web application. This article will focus on AF system metrics, like its performance, load or health. We will cover the topic of reporting the data metrics into the same database in an upcoming article, “TITLE”. So, stay tuned. 

Why?

The Airflow itself finally got a Cluster status dashboard built-in. Therefore, it is valid to question the need for a separate dashboard that requires additional effort and maintenance. Why bother, then? There are several possible reasons to introduce a separate monitoring stack:

• Aggregated view: looking into one app is easier than browsing through 10 of them. TIG (or any other monitoring stack) can be used for monitoring multiple instances and apps and is easily integrated with custom ones.

• Security: there is no need to give access to the AF itself (or any other application) to be able to see the metrics and pinpoint errors. It is aligned with the data mesh and any other data democratization approach.
• Support friendly: your 1st line support has a starting point to check the status of the data processing and be able to call the right people in case of a problem.

The toolkit

Out of the box (but with the right pip packages) Airflow supports sending its internal metrics to a statsd server. We can leverage that and set up such a server via Telegraf to proxy the metrics further into a time series database: InfluxDB. As a convenient UI, Grafana can be used.

TIG stack configuration

Firstly, let’s configure the TIG stack to accept the metrics:

1. Install an InfluxDB instance. Nothing fancy here.
2. Install the Telegraf. In the configuration, look for the statsd input (which has to be enabled) and InfluxDB output. The latter will require at least authentication. Note that the Telegraf supports many more inputs and allows monitoring of, e.g. system resources (CPU, memory, disk space, etc.). This is a good idea to configure in a production environment.
3. Install Grafana and configure the data source to point it to the InfluxDB.

We have prepared a docker compose-based demo in the GitHub repository with a pre-configured environment. You can see there the example configuration for Telegraf (telegraf.conf), InfluxDB (influxdb.env) and Grafana (grafana-provisioning/datasources). Please note that this is only for dev/demo purposes, and real production environments need to be set up more securely (including the use of HTTPS and secure password handling).

Airflow

On the Airflow side, there are two significant points to be addressed.

1. Airflow requires the Apache-airflow [statsd] package to have the statsd client available – you can do it via pip.
2. In the airflow configuration file (usually: Airflow.cfg), the [metrics] section must be configured to enable the statsd, point it to the Telegraf instance and prefix the metrics (very useful in case of multiple Airflow instances).

There are several metrics available to be sent in Airflow. You can see the complete list in the Airflow documentation here. There, you can also limit which ones are actually reported to the Telegraf. Be aware that even though most of the metrics are said to be reported in seconds, you need to validate them yourself (as in this example).

Dashboards

The last step is to configure Grafana and set up some dashboards. The UI offers a WYSIWYG editor that you can use to tailor it to your needs. The example available on the GitHub might serve as a starting point, as it shows:

• state of the Airflow executors (queues and tasks being run at the moment) to see whether any processing is going on,
• state of the pools (default and the custom ones) to check potential bottlenecks if you use multiple pools,
• task times to identify unexpected stragglers (and compare instances’ run times and find any performance challenges early on).

Summary

In conclusion, for Airflow monitoring, you can use specialized metrics tools like TIG stack and aggregate all the metrics from multiple AF instances. The stack can accommodate AF system metrics and data from other applications, including your custom ones. An example of sending data quality metrics is what we will look into in the second part. There is a demo environment on our GitHub to see whether this works for you.

The post Monitoring Airflow jobs with TIG 1: system metrics appeared first on TantusData.

The Databricks platform offers two execution engines for the clients: the standard Apache Spark (available as an open-source application) and one with Photon enhancement that brings a performance improvement (as well as extra pricing). Have you ever wondered where this speedup comes from and how it affects designing Apache Spark jobs?

This article is based on Berlkey’s paper on Photon by Databricks people, as that publication is the closest to the source, as the Photon engine is not an open-source project, and its source code is not available to the public.

The general idea

In a nutshell, Photon replaces the standard (bundled) query engine available in Apache Spark, using the same API. For some operations, mostly CPU-heavy ones, the Catalyst optimizer may decide to send the job to Photon instead of using the default execution path, all for performance reasons. In other words, it is an alternative way to execute Spark DAG tasks, not skipping/reordering/reorganizing the data engine.

SIMD

The SIMD stands for Single Instruction, Multiple Data and is one of the ground ideas for job optimisations in Photon. With current architectures running the same operation on multiple instances (values, rows, etc.), data can be optimised via vectorisation, even within a single thread. Photon is said to utilise those optimisations.

C++ and Code generation

The Photon engine is implemented in C++ instead of the JVM native languages (Scala, Java or others). The source paper points to performance reasons, including ‘hitting performance ceilings’. The communication with the rest of the Apache Spark is implemented via JNI. Databricks’ internal benchmarks indicate that the performance hit due to moving data in and out of JVM is not noticeable. 

Internal data format

Photon engine uses columnar data representation (same as, e.g. Parquet data format) instead of row data (like the rest of Apache Spark). This is due to SIMD optimizations – kernel implementation that works best on columnar data. The memory management (calling, freeing, etc) is still done via Apache Spark’s memory manager. The data is kept off-heap, so transferring from Photon to Spark does not require copying the data.

When a shuffle operation is necessary, Photon writes a shuffle file and uses Spark API to execute the exchange. However, the data format is not compatible with vanilla spark, and a Photon shuffle read must follow the Photon shuffle write.

When does it help?

Photon is meant to address the CPU-heavy loads. This includes joins (especially hash join) and aggregations. On the other hand, being a non-JVM implementation, Photon obviously does not support UDFs or RDD API. Exact benchmarks and precise speedups are mentioned in the source paper.

Sources

• Source paper: Photon: A Fast Query Engine for Lakehouse Systems

I hope this helps. Moreover, if you know any other good sources, do let us know on social media so that everyone can see them.

The post Databricks – Photon appeared first on TantusData.

Although the ideal data pipeline is made of idempotent and independent tasks, there are some cases when setting up a mutex (a.k.a. part of the job that cannot be run concurrently) is necessary. Fortunately, Airflow supports such cases and offers a few tools varying by complexity to implement such a pipeline.

In this article, we will look at the following DAG in AF. The graph itself is relatively simple; the catch is that load_1 and load_2 operators cannot have concurrently running task instances. We will look at treating loads separately and looking at them as a group.

Therefore, there are three scenarios we will look into:

1. Only one instance of the operator can be running.
2. One instance is that the operator cannot run before the older ones are successful.
3. Only one instance of a group of operators.

The examples are using annotation syntax for Airflow. Still, the concept stays the same for the 1.0 compatible approach – instead of annotation params, use them in any Operator class constructor.

Mutex on an operator

The first scenario is that only one operator instance can be running at a time. If there are multiple runs ready to be scheduled, it does not matter which one goes first. One solution would be using the max_active_tis_per_dag option with the value of 1. 

@task(
   max_active_tis_per_dag=1
)

Dependency on past runs

The second example assumes that the nth batch cannot start before the nth-1 one is completed successfully (or at least marked so in Airflow). For this use case, AF offers depends_on_past flag. In this case, you have to be careful and pay some attention to the state of the latest runs. One failed, upstream-failed, or waiting task can halt all future DAG runs.

@task(
   depends_on_past=True
)

Pools – mutex across multiple operators

The most complex approach is required if we need to group multiple operators to make them share a mutex. One way to do it is to put them into one custom pool and limit it to accommodate only one task simultaneously – either by setting a pool with 1 slot or assigning a high number of required slots to each operator.

load_pool = Pool.create_or_update_pool(
   name="load_pool",
   slots=1,
   description="Pool for data load tasks."
)

@task(
   pool=load_pool.pool
)

Source

The source code for all examples and the docker environment to run them is available on GitHub.

The post Airflow — pools and mutexes. appeared first on TantusData.

There are data pipelines where you must pass some values between tasks – not complete datasets, but ~ kilobytes. This can be managed even within the Airflow itself. As always, multiple options are available – let’s review some of them.

In this article, we are looking at sharing data between DAGs, which are connected via run dependencies. Let’s assume that each DAG needs to be run daily, and the first DAG generates some important data for the second DAG. 

XCom

XCom would be the first and the recommended approach. It works well with out-of-the-box features like ExternalTaskSensor:

with DAG(
       dag_id="xcom-sink",
       schedule_interval="@daily",
       start_date=datetime(2023, 7, 1),
       catchup=True,
) as xcom_sink:
   ExternalTaskSensor(
       task_id="wait-for-dependency",
       external_dag_id="xcom-source",
       external_task_id="update-hive-table-events-triggers"
   ) >> BashOperator(
       task_id="show-xcom",
       bash_command="echo {{ ti.xcom_pull(dag_id='xcom-source', task_ids='update-hive-table-events-triggers') }}"
   )

XCom vs run id

The XCom is identified by the DAG ID, task ID and run ID. If you want to run a single DAG with a custom run ID, you have to ensure there is an XCom value for that run ID already created. This complicates issuing manual runs.

When not to XCom?

Rather than discussing cases that are tailored to use XCom, let’s focus on examples that are not well supported. Basically, XCom does not work well with Datasets, and you might get some quirky results here: https://github.com/apache/airflow/discussions/33069. 

With datasets, you need to refer to the last past value of XCom, effectively losing all benefits of tight coupling. Using that parameter, you will need to deal with race conditions: if the not-latest sink DAG is restarted, it will receive an incorrect value from the source. Moreover, let’s consider a design with multiple sources and a single sink:

Suppose the events dataset gets updated twice and the users’ dataset only once. In that case, the events-with-users will receive only the latest value of the former one. It is up to you to decide whether this behaviour is expected or unacceptable.

Variables

In rare cases, you can use Variables from Airflow to solve the issue. Variables are built-in mechanisms in Airflow that provide a global state (or configuration) for all DAGs to read and write. You can use it to implement the Registry code pattern:

1. Put a hashmap of the date of run -> metadata into a variable
2. Use the execution date (named logical_date in the newer versions) as the hashmap key.
3. Use PythonOperator to update the variable. Read can be done by either Python code or templating.
4. Use sensors or datasets to schedule DAGs in the correct order.

Beware!

Before you go with the variable route, please keep in mind that compared to XComs, variables are way more costly to maintain. You might need to keep track of the variables’ sizes, implement error handling in your PythonOperators and be mindful of any unsolicited changes in the variables’ values.

External system?

There is always an option of using an external data service to synchronise, similarly to using built-in variables. This may come in the form of data paths on HDFS / S3 / other storage, batch load dates in the database or such. However, this solution creates an inferior design, as you would end up with disadvantages of the variables approach and new implicit dependencies between DAGs and external services.

Summary

Source

The source code for both XCom and Variable examples and the docker environment to run them is available on GitHub.

The post Passing information between DAGs in Airflow. appeared first on TantusData.

Most programming languages have their SDK available for non-system-wide installation. Python offers venv, while Java Development Kit needs only the JAVA_HOME variable to be set, etc. At first glance, this does not seem to be the case with Node.js for JavaScript, where developers are expected to install all tools system-wide, but there are ways to achieve it. One is to use a tool called n – a simple-to-use wrapper and manager for node.js, as described below.

Installation

Three short steps that work for MacOSX:

# Install n

brew install n

# Set N_PREFIX to avoid writing into system directories

export N_PREFIX=$HOME/.n

export PATH=$PATH:$N_PREFIX/bin

# Install latest node.js

n latest

The full installation guide is available on the project’s GitHub page: (https://github.com/tj/n#third-party-installers). 

IntelliJ IDEA

I want to use an IDE to work with node.js apps. Unfortunately, editors like IntelliJ fail to find the node command that is not on the system $PATH. One can work around this issue and use the terminal to execute the framework’s command. However, there is also a more comfortable approach – updating the environment variables.

N_PREFIX

This one is pretty simple: create or update the `/etc/launchd.conf` file (source: https://stackoverflow.com/questions/135688/setting-environment-variables-on-os-x): 

setenv N_PREFIX <absolute_path_to_n_prefix>

PATH

In case of $PATH variable there is a different way to set it than using launchd:

# Create a file in /etc/paths.d/ with any file name

# and value of directory you want to append to $PATH

echo "/path/to/.n/bin" | sudo tee -a /etc/paths.d/n

The post NPM & N on MacOSX appeared first on TantusData.

In the real world, data pipelines only sometimes come as a completely independent sequence of operations. Usually, they share dependences on one another, occasionally easy 1:1 ones, sometimes more complicated. Here is a short list of what Apache Airflow has to offer for handling those relationships.

The general setting

We will look into a situation where parent data pipelines (called DAGs in Airflowish) create data used (consumed) by children pipelines. The dependency looks like this:

The silent assumption is that we cannot just look at the metadata (file creation time, last row’s created timestamp or alike), as it might show that data is being loaded but not exactly fully loaded – therefore, we are waiting for the entire dataset to be available.

External tools

Using an external tool is always available, not only for Airflow data pipelines. The most straightforward implementation would be using an 0-byte file on NFS, HDFS or any other shareable filesystem to mark success for a given dataset.  

This example is based on HDFS, but you could also use:

• NFS,
• FTP/SFTP location,
• metadata stored in a SQL or NoSQL database,
• any other accessible non-airflow related data storage.

Source code

For the current (4.1.0) version of the apache-airflow-providers-apache-hdfs, there is no HDFS operator; only sensors are available. Therefore a bit of walkaround (e.g. BashOperator) has to be used to create a marker file.

from __future__ import annotations


from datetime import datetime


from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.apache.hdfs.sensors.web_hdfs import WebHdfsSensor
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor


HDFS_CONNECTION_ID = "hdfs-14"
SFTP_CONNECTION_ID = "sftp-21"


with DAG(
	dag_id="external-events",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events:
	SFTPSensor(
    		task_id="wait-for-csv",
	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    		sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="csv-to-parquet",
    		application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/events/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_users:
	SFTPSensor(
    		task_id="wait-for-csv",
	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    		sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="csv-to-parquet",
    		application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/users/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-events-with-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events_with_users:
	[
    	WebHdfsSensor(
        	task_id="wait-for-marker-events",
        	filepath="/marker/events/{{ dag_run.logical_date | ds }}.success",
        	webhdfs_conn_id=HDFS_CONNECTION_ID
    	),
    	WebHdfsSensor(
        	task_id="wait-for-marker-users",
        	filepath="/marker/users/{{ dag_run.logical_date | ds }}.success",
        	webhdfs_conn_id=HDFS_CONNECTION_ID
    	)
	] >> SparkSubmitOperator(
	    	task_id="compute-events-with-users",
    		application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
	    	task_id="update-hive-table",
    		application="/spark/applications/update-hive-table.py"
	) >> BashOperator(
	    	task_id="create-hdfs-marker-file",
    		bash_command="hdfs dfs -touch /marker/events-with-users/{{ dag_run.logical_date | ds }}.success"
	)


with DAG(
	dag_id="external-reports-1",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_reports_1:
	WebHdfsSensor(
	    	task_id="wait-for-events-with-users",
    		filepath="/marker/events-with-users/{{ dag_run.logical_date | ds }}.success",
	    	webhdfs_conn_id=HDFS_CONNECTION_ID
	) >> SparkSubmitOperator(
	    	task_id="compute-report",
    		application="/spark/applications/compute-report-1.py"
	)


with DAG(
	dag_id="external-reports-2",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_reports_2:
	WebHdfsSensor(
    		task_id="wait-for-events-with-users",
	    	filepath="/marker/events-with-users/{{ dag_run.logical_date | ds }}.success",
    		webhdfs_conn_id=HDFS_CONNECTION_ID
	) >> SparkSubmitOperator(
    		task_id="compute-report",
	    	application="/spark/applications/compute-report-2.py"
	)

This is a helpful technique if Airflow is to be integrated with other tools. On the other hand, the dependency is very implicit. It must be refactored carefully to maintain the contract between data pipelines.

Airflow datasets

Since version 2.4, Airflow has offered data-aware scheduling based on the concept of datasets. The general idea is:

1. Operators define datasets to which they publish data. The dataset in Airflow is just metadata – AF does not handle the data itself.

2. DAGs specify datasets that they depend on. Each time all dependency datasets are updated (once or many times), the dependent DAG is triggered. This setting replaces the time-based schedule.

Although this feature does not give any control over the time of the execution nor tie specific runs (there is no guarantee that DAG gets run on every separate dependency run – several runs may get squashed into a single dependent run), a user receives out-of-the-box a friendly UI for browsing datasets and DAGs relationships. 

Source code

from __future__ import annotations


from airflow import DAG, Dataset
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from pendulum import datetime




dataset_users = Dataset("af://datasets/users")
dataset_events = Dataset("af://datasets/events")
dataset_events_with_users = Dataset("af://datasets/events-with-users")
dataset_reports_1 = Dataset("af://datasets/reports-1")
dataset_reports_2 = Dataset("af://datasets/reports-2")


SFTP_CONNECTION_ID = "sftp-21"


with DAG(
	dag_id="dataset-events",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule="@daily",
	max_active_runs=1,
) as dag_producer_events:
	SFTPSensor(
        	task_id="wait-for-csv",
    	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_events]
	)


with DAG(
	dag_id="dataset-users",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule="@daily",
	max_active_runs=1,
) as dag_producer_users:
	SFTPSensor(
        	task_id="wait-for-csv",
    	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_users]
	)


with DAG(
	dag_id="dataset-events-with-users",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events, dataset_users],
	max_active_runs=1,
) as dag_processor_events_with_users:
	SparkSubmitOperator(
    	    	ask_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py",
    	    	outlets=[dataset_events_with_users]
	)


with DAG(
	dag_id="dataset-reports-1",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events_with_users],
	max_active_runs=1,
) as dag_processor_reports_1:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py",
    	    	outlets=[dataset_reports_1]
	)


with DAG(
	dag_id="dataset-reports-2",
	catchup=True,
	start_date=datetime(2023, 7, 1, tz="UTC"),
	schedule=[dataset_events_with_users],
	max_active_runs=1,
) as dag_processor_reports_2:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py",
    	    	outlets=[dataset_reports_2]
	)

Trigger the external DAG

Suppose the functionality of the datasets is too limited or an older version of the AF is used. In that case, there are alternative ways to orchestrate the data pipelines. TriggerDagRunOperator represents the push approach. The idea behind this operator is to trigger a run of a specified DAG with the option of supplying custom parameters. Obviously, this feature does not allow to create a dependency on more than one source dataset. However, one dataset can still run multiple dependents.

Since this approach uses the DAG name (id) to identify the dependent pipeline, you must be careful when refactoring those names. One way to look up the dependent (and depending) DAGs and tasks is to use the Browse > DAG Dependencies tab that shows a graph of trigger and sensor relationships.

Source code

from __future__ import annotations


from datetime import datetime
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.utils.task_group import TaskGroup
from airflow.utils.trigger_rule import TriggerRule




SFTP_CONNECTION_ID = "sftp-21"




with DAG(
	dag_id="trigger-events-with-users",
	schedule="@daily",
	start_date=datetime(2023, 7, 1),
) as dag_events_with_users:
	with TaskGroup(group_id="gather-events") as group_events:
    	SFTPSensor(
        	task_id="wait-for-csv",
        	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
        	sftp_conn_id=SFTP_CONNECTION_ID
    	) >> SparkSubmitOperator(
        	task_id="csv-to-parquet",
        	application="/spark/applications/csv-to-parquet.py"
    	) >> SparkSubmitOperator(
        	task_id="update-hive-table",
        	application="/spark/applications/update-hive-table.py"
    	)


	with TaskGroup(group_id="gather-users") as group_users:
    	SFTPSensor(
        	task_id="wait-for-csv",
        	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
        	sftp_conn_id=SFTP_CONNECTION_ID
    	) >> SparkSubmitOperator(
        	task_id="csv-to-parquet",
        	application="/spark/applications/csv-to-parquet.py"
    	) >> SparkSubmitOperator(
        	task_id="update-hive-table",
        	application="/spark/applications/update-hive-table.py"
    	)


	[group_events, group_users] >> SparkSubmitOperator(
    	    	task_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	) >> [
    	    	TriggerDagRunOperator(
        	    	task_id="trigger-reports-1",
    	        	trigger_dag_id="trigger-reports-1",
        	    	trigger_rule=TriggerRule.ALL_SUCCESS,
    	    	),
    	    	TriggerDagRunOperator(
        	    	task_id="trigger-reports-2",
    	        	trigger_dag_id="trigger-reports-2",
           	 	trigger_rule=TriggerRule.ALL_SUCCESS,
    	    	)
	]


with DAG(
	dag_id="trigger-reports-1",
	catchup=True,
	start_date=datetime(2023, 7, 1),
) as dag_processor_reports_1:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py",
	)


with DAG(
	dag_id="trigger-reports-2",
	catchup=True,
	start_date=datetime(2023, 7, 1),
) as dag_processor_reports_2:
	SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py",
	)

The sensor for external DAG

Using the sensor is exactly the opposite of triggering some other DAG. In this approach, we tell DAG to pause and wait until some other DAG (or a specific task in the DAG) is completed. We still need to specify appropriate schedules for both pipelines, however.

The sensor approach is very flexible. Unlike the datasets approach, each sensor instance can wait for a specific DAG/task/execution date combination, which allows waiting with specific time offset or model aggregation relationships (hourly to daily, daily to weekly, etc.). Unlike the triggering approach, a DAG can model waiting for all preconditions with the sensor approach.

Precisely, like when using DAG triggering, sensor dependency can be checked on the Browse > DAG Dependencies tab. 

Source code

from __future__ import annotations


from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.sensors.external_task import ExternalTaskSensor




SFTP_CONNECTION_ID = "sftp-21"




with DAG(
    	dag_id="sensor-events",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as sensor_events:
	SFTPSensor(
    	    	task_id="wait-for-csv",
    	    	path="/inbox/events/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensor-users",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as sensor_users:
	SFTPSensor(
    	    	task_id="wait-for-csv",
    	    	path="/inbox/users/{{ dag_run.logical_date | ds }}.csv",
    	    	sftp_conn_id=SFTP_CONNECTION_ID
	) >> SparkSubmitOperator(
    	    	task_id="csv-to-parquet",
    	    	application="/spark/applications/csv-to-parquet.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensor-events-with-users",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_daily:
	[
    	    	ExternalTaskSensor(
         	   	task_id="wait-for-events",
        	    	external_dag_id="sensor-events",
    	        	check_existence=True
    	    	),
    	    	ExternalTaskSensor(
         	   	task_id="wait-for-users",
        	    	external_dag_id="sensor-users",
    	        	check_existence=True
    	    	)
	] >> SparkSubmitOperator(
    	    	task_id="compute-events-with-users",
    	    	application="/spark/applications/compute-events-with-users.py"
	) >> SparkSubmitOperator(
    	    	task_id="update-hive-table",
    	    	application="/spark/applications/update-hive-table.py"
	)


with DAG(
    	dag_id="sensors-reports-1",
    	schedule="@daily",
    	start_date=datetime(2023, 7, 1),
) as dag_reports_1:
	ExternalTaskSensor(
        		task_id="wait-for-events-with-users",
    	    	external_dag_id="sensor-events-with-users",
    	    	check_existence=True
	) >> SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-1.py"
	)


with DAG(
    	dag_id="sensors-reports-2",
    	schedule="@daily",
    	start_date=datetime(2023, 7, 1),
) as dag_reports_2:
	ExternalTaskSensor(
    	    	task_id="wait-for-events-with-users",
    	    	external_dag_id="sensor-events-with-users",
    	    	check_existence=True
	) >> SparkSubmitOperator(
    	    	task_id="compute-report",
    	    	application="/spark/applications/compute-report-2.py"
	)

Short comparison

Example scenarios

The end goal of running Airflow is to orchestrate an actual pipeline, so let’s discuss some actual-life examples where DAG dependencies are necessary.

Breaking down complex DAG

First, DAG is an example of a complex DAG used to calculate multiple outputs (exports) from some external inputs. For clarity, all operations are split into waiting (sensors), preprocessing and reports computation. The screenshot shows them in the form of a Graph view from Airflow.

The assumption is that all operations consume whole input datasets – they are not restricted to, e.g. current hour only. Otherwise, using sensors or external markers would be the right option.

This situation is where datasets (or, according to the docs – data-aware scheduling) show good results. For each branching point, we define an outlet – abstraction over the dataset produced by each operator (remember that it is only the metadata – handling the actual data is not done by Airflow and never should be). Then we define each chain of operations that takes those datasets as a separate DAG with datasets defined as the schedule. As a result, the DAGs are short & simple, and we got a bit of data lineage in the Datasets tab of Airflow for free:

Source code

from __future__ import annotations


from datetime import datetime


from airflow import Dataset
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator
from airflow.sensors.bash import BashSensor


dataset_clients = Dataset("af://dataset/clients")
dataset_events = Dataset("af://dataset/events")
dataset_transactions = Dataset("af://dataset/transactions")
dataset_users = Dataset("af://dataset/users")


with DAG(
	dag_id="complex-dag-clients",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_clients:
	sensor_clients = BashSensor(task_id="wait-clients", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4", outlets=[dataset_clients])


	sensor_clients >> op_preprocess_step_1 >> \
    	[op_preprocess_step_2 >> op_preprocess_step_3] >> \
    	op_preprocess_step_4


with DAG(
	dag_id="complex-dag-events",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_events:
	sensor_events = BashSensor(task_id="wait-events", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3", outlets=[dataset_events])


	sensor_events >> op_preprocess_step_1 >> op_preprocess_step_2 >> op_preprocess_step_3


with DAG(
	dag_id="complex-dag-transactions",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_transactions:
	sensor_gateway_01 = BashSensor(task_id="wait-gateway-01", bash_command="true")
	sensor_gateway_02 = BashSensor(task_id="wait-gateway-02", bash_command="true")
	sensor_gateway_03 = BashSensor(task_id="wait-gateway-03", bash_command="true")
	sensor_gateway_04 = BashSensor(task_id="wait-gateway-04", bash_command="true")
	sensor_gateway_05 = BashSensor(task_id="wait-gateway-05", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4")
	op_preprocess_step_5 = EmptyOperator(task_id="preprocess-step-5", outlets=[dataset_transactions])


	[
    	sensor_gateway_01,
    	sensor_gateway_02,
    	sensor_gateway_03,
    	sensor_gateway_04,
    	sensor_gateway_05
	] >> op_preprocess_step_1 >> op_preprocess_step_2 >> op_preprocess_step_3 >> \
    	op_preprocess_step_4 >> op_preprocess_step_5


with DAG(
	dag_id="complex-dag-users",
	schedule_interval="@daily",
	start_date=datetime(2023, 7, 1),
) as complex_dag_users:
	sensor_users = BashSensor(task_id="wait-users", bash_command="true")


	op_preprocess_step_1 = EmptyOperator(task_id="preprocess-step-1")
	op_preprocess_step_2 = EmptyOperator(task_id="preprocess-step-2")
	op_preprocess_step_3 = EmptyOperator(task_id="preprocess-step-3")
	op_preprocess_step_4 = EmptyOperator(task_id="preprocess-step-4")
	op_preprocess_step_5 = EmptyOperator(task_id="preprocess-step-5")
	op_preprocess_step_6 = EmptyOperator(task_id="preprocess-step-6", outlets=[dataset_users])


	sensor_users >> [op_preprocess_step_1, op_preprocess_step_2] >> op_preprocess_step_3 >> \
    	[op_preprocess_step_4, op_preprocess_step_5] >> op_preprocess_step_6


with DAG(
	dag_id="complex-dag-report-clients",
	schedule=[dataset_clients, dataset_events],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_clients:
	op_rich_clients = EmptyOperator(task_id="enrich-clients")
	op_report_clients = BashOperator(task_id="report-clients", bash_command="sleep 3")


	op_rich_clients >> op_report_clients


with DAG(
	dag_id="complex-dag-report-users",
	schedule=[dataset_events, dataset_users],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_users:
	op_rich_users = EmptyOperator(task_id="enrich-events")
	op_report_users = BashOperator(task_id="report-users", bash_command="sleep 3")


	op_rich_users >> op_report_users


with DAG(
	dag_id="complex-dag-report-full",
	schedule=[dataset_events, dataset_transactions, dataset_users],
	start_date=datetime(2023, 7, 1),
) as complex_dag_report_full:
	op_report_users = BashOperator(task_id="report-full", bash_command="sleep 3")

Daily to weekly transition

This example covers any time-based aggregation of the data. Regardless of the actual time intervals, the general idea is that multiple runs of one DAG are used as input for the other one:

• 24 runs of the hourly pipeline for daily one,
• 7 runs of the daily DAG for the weekly one.

This setting limits available options to using an external tool or implementing DAG-run sensors. Let’s take a look at the latter one.

The idea behind this diagram is simple – the daily pipeline is unaffected by the relationship. At the same time, the weekly one starts with seven sensors of ExternalTaskSensor, each with a different execution date offset. Those have been waiting for daily DAGs from 1 to 7 days ago. Apart from that set of sensors, the DAG is similar to others.

Source code

from __future__ import annotations


from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.bash import BashSensor
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.utils.task_group import TaskGroup


with DAG(
    	dag_id="dag-daily",
    	schedule_interval="@daily",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_daily:
	BashSensor(task_id="some-input-wait", bash_command="true") >> \
    	BashOperator(task_id="some-data-processing-1", bash_command="sleep 3") >> \
    	BashOperator(task_id="some-data-processing-2", bash_command="sleep 3") >> \
    	BashOperator(task_id="exporting-to-hdfs", bash_command="sleep 3")


with DAG(
    	dag_id="dag-weekly",
    	schedule_interval="@weekly",
    	start_date=datetime(2023, 7, 1),
    	catchup=True,
) as dag_weekly:
	# TaskGroup is purely optional, but makes DAG in the UI a bit clearer.
	with TaskGroup(group_id="dag-daily-sensors") as sensors:
    	[
        	ExternalTaskSensor(
            		task_id=f"wait-{days_offset}d",
	            	external_dag_id="dag-daily",
       		     	timeout=24 * 60 * 60,
            		mode="reschedule",
	            	execution_delta=timedelta(days=days_offset)
        	) for days_offset in range(1, 8)
    	]


	sensors >> \
    		BashOperator(task_id="some-data-processing", bash_command="sleep 3") >> \
	    	BashOperator(task_id="exporting-to-hdfs", bash_command="sleep 3")

The source code for all examples and the docker environment to run them is available on GitHub.

The post Managing inter-DAG dependencies in Airflow appeared first on TantusData.

When working on more advanced analytics (or reports), you may stumble upon the problem of doing a self-join. Be it all pairs of available products, account connections in social media or anything else; it requires creating a heavy self-join on the data. Since this pattern is not as commonly documented as aggregation by group or filtering, let’s look into possible implementations of it. In this case, we use BigQuery service on the Google Cloud Platform.

The challenge

We have a table of products available in shops: 

1. One row represents one product in one shop.

2. If a product is available in multiple stores, it will have multiple rows, each with an appropriate shop ID.

As a result, we need a table of pairs of products:

+ All to all product pairs are computed for each shop and month, and we do not want cross-month or cross-store pairs (if the first product is from shop 38 and month 2001-04, the second one must be from the same period and store),

+ The table will be queried with a month predicate – only one single month per query will be requested,

+ Each pair of products within one month & shop should appear only once (if (A, B) pair is present, there should not be a row of (B, A) for a given month and shop).

All queries must run successfully in GCP BigQuery on-demand pricing mode.

Input data

In this case, we are going to look into the following table containing input data:

CREATE TABLE IF NOT EXISTS db.products AS
 SELECT
   product,
   shop,
   transactions,
   transaction_date
 FROM
   db.products_ingestion;

This table is the input for self-join. The available columns are narrowed to a minimum for the sake of clarity:

• Product (containing product id – those will be cross-joined).
• Shop (shop id).
• Transactions (number of transactions involving a given product).
• Transaction_date (effectively truncated to year-month). 

Naive approach

The first thing that comes to mind when doing the self-join is to do the self-join. Let’s test the simplest approach:

CREATE TABLE db.products_pairs AS
 SELECT
   lhs.product AS product_1,
   rhs.product AS product_2,
   lhs.transactions AS transactions_1,
   rhs.transactions AS transactions_2,
   lhs.shop AS shop,
   lhs.transaction_date AS transaction_date
 FROM (SELECT * FROM db.products) lhs
 INNER JOIN (SELECT * FROM db.products) rhs
 USING (shop, transaction_date)
 WHERE
   lhs.transaction_date = DATE(2022, 1, 1) 
      AND rhs.transaction_date = DATE(2022, 1, 1) 
      AND lhs.product < rhs.product;

The quiet assumption here is to run data filtering at the input level (before the join) and select only a single month of data on each query. The last predicate is to filter out product duplicates. Although this query looks good, we end up with either of those error messages:

Resources exceeded during query execution: Your project or organization exceeded the maximum disk and memory limit available for shuffle operations. Consider provisioning more slots, reducing query concurrency, or using more efficient logic in this job.

The query exceeded resource limits. This query used 1666502 CPU seconds but would charge only 2394M Analysis bytes. This exceeds the ratio supported by the on-demand pricing model. Please consider moving this workload to the flat-rate reservation pricing model, which does not have this limit. 1666502 CPU seconds were used, and this query must use less than 612800 CPU seconds.

The BigQuery engine seems to look into input data size and estimate the join cost based on that data. This case appears to be a corner case, though – we use only one table, but twice. This is clearly not considered in this computation, and using only one table fools the engine. Let’s fool it deeper and make it estimate the cost correctly. 

Complicated approach

We are going to look into two approaches: 

• tweaking the query to make the BigQuery engine calculate the cost correctly,
• recreating the query to get rid of self-join.

Along with comparing those two ideas, we will look at the impact of using the two most apparent techniques for fine-tuning queries: partitioning and clustering. They should improve both the execution time and cost – let’s see how effective they are.

Fixed join approach

Changing the cartesian join goes first. One trick must be applied: each side of the join must come from a different table. This can be accomplished by creating a 1:1 copy of the input table. We cannot use a view here – BQ is smart enough to look at the source of the view.

CREATE TABLE IF NOT EXISTS db.products_aux AS
 SELECT * FROM db.products;
CREATE TABLE db.products_pairs_join AS
 SELECT
   lhs.product AS product_1,
   rhs.product AS product_2,
   lhs.transactions AS transactions_1,
   rhs.transactions AS transactions_2,
   lhs.shop AS shop,
   lhs.transaction_date AS transaction_date
 FROM (SELECT * FROM db.products) lhs
 INNER JOIN (SELECT * FROM db.products_aux) rhs
 USING (shop, transaction_date)
 WHERE
   lhs.transaction_date = DATE(2022, 1, 1) 
      AND rhs.transaction_date = DATE(2022, 1, 1) 
      AND lhs.product < rhs.product;

One downside of this approach is that Google does not recommend cartesian joins. It also creates a duplicate of the input table, which must be managed and synchronized with the input. Nevertheless, it is possible to run such a query successfully.

Windowing approach

Aggregation and windowing can be employed to compute the output table to avoid doing a self-join. It is a bit similar to the computation of rolling sum (or average), but instead of a sum function, rows will be collected into an array: for each shop and transaction date, we want all the following products. 

The second step is splitting array elements into separate rows via UNNEST instruction to produce the final output. It also clears null rows that cannot be joined when unnesting.

CREATE TABLE IF NOT EXISTS db.products_pair_aggregate AS
 WITH root AS (
   SELECT *,
       ARRAY_AGG(STRUCT (product, transactions)) OVER (product_window) AS product_2
       FROM db.products
       WHERE transaction_date = DATE(2022, 1, 1)
       WINDOW product_window
         AS (PARTITION BY shop, transaction_date 
             ORDER BY product ROWS BETWEEN 1 FOLLOWING AND UNBOUNDED FOLLOWING)
 )
 SELECT
   shop,
   transaction_date,
   root.product AS product_1,
   product_2.product AS product_2,
   root.transactions AS transactions_1,
   product_2.transactions AS transactions_2
   FROM root, UNNEST(product_2) product_2
 WHERE root.transaction_date = DATE(2022, 1, 1);

The final query is more complicated than one in the join approach, but it does not require creating any ‘hacky’ mirror tables.

Results

Both solutions can be implemented as unpartitioned, partitioned or partitioned and clustered tables (and queries creating them); therefore, there are six implementations to compare. Five of them work with BQ on-demand pricing, and the aggregation approach with only partitioning fails.

The results are presented in the table below. Each approach has been measured in three configurations: vanilla (no optimisation), partitioning (P) as well as partitioning and clustering (P+C). The join benchmark includes creating the mirror table (upper row) and the join itself (lower).

Main points

1. Using partitioning on filtered tables is necessary to bring the cost of a query into acceptable range. 

2. Both join, and aggregate-based queries can produce the output in a reasonable amount of time and money. The aggregate one will further halve the cost, as it does not require creating a mirror table.

3. Apart from the pure benchmarking results, it is good to compare the query complexity when deciding on the approach. The join query may be trickier to maintain due to the extra table, but it is way easier to understand than the aggregate one.

The post Big cartesian join in Big Query appeared first on TantusData.