data pipelines

obracajacy sie tekst
podwojna strzalka w dol
all articles
airflow DAG dependencies data pipelines job orchestration
The technicalities 7 min read

Passing information between DAGs in Airflow.

There are data pipelines where you must pass some values between tasks – not complete datasets, but ~ kilobytes. This can be managed even within…

check out.
Apache Airflow DAG dependencies data pipelines data-aware scheduling performance tuning
The technicalities 15 min read

Managing inter-DAG dependencies in Airflow

In the real world, data pipelines only sometimes come as a completely independent sequence of operations. Usually, they share dependences on one another, occasionally easy…

check out.