Menu
e-mail us at contact@tantusdata.com
social media

Select your language

Amadeusz Kosik

Amadeusz Kosik

Big Data Engineer

obracajacy sie tekst
podwojna strzalka w dol
all of Amadeusz’s articles
Apache Spark data pipelines
The technicalities 3 min read

RDD in Apache Spark

Learn how to utilize the RDD API in Apache Spark to check partition details or perform low-level operations. Despite being deprecated, the RDD API is accessible via the .rdd method on Datasets and DataFrames. Discover how to check the number of partitions with the getNumPartitions method and determine partition sizes using the glom function. Explore the remaining useful operations that RDD API offers for low-level hacking and internal Spark tasks.

check out.
Apache Spark data pipelines
The technicalities 5 min read

Datasets and DataFrames

Understanding Spark's .as[T] Method: Best Practices and Defensive Programming

check out.
data pipelines data quality monitoring
The technicalities 3 min read

Monitoring Airflow jobs with TIG 2: data quality metrics

In the first article on Monitoring Airflow jobs with TIG, “System Metrics”, we have seen an example of Airflow installation with a TIG stack set…

check out.
Apache Airflow data pipelines devops monitoring sysops
Apache Airflow min read

Monitoring Airflow jobs with TIG 1: system metrics

Like many server applications, Airflow can – and should – be monitored for metrics and logs. In this article, we will look into the former…

check out.
Apache Airflow DAG dependencies data pipelines job orchestration
Apache Airflow 5 min read

Databricks – Photon

The Databricks platform offers two execution engines for the clients: the standard Apache Spark (available as an open-source application) and one with Photon enhancement that…

check out.
Apache Airflow DAG dependencies data pipelines job orchestration
Apache Airflow 6 min read

Airflow — pools and mutexes.

Although the ideal data pipeline is made of idempotent and independent tasks, there are some cases when setting up a mutex (a.k.a. part of the…

check out.
Apache Airflow DAG dependencies data pipelines job orchestration
Apache Airflow 7 min read

Passing information between DAGs in Airflow.

There are data pipelines where you must pass some values between tasks – not complete datasets, but ~ kilobytes. This can be managed even within…

check out.
IntelliJ IDEA node.js node.js installation non-system-wide installation
The technicalities 5 min read

NPM & N on MacOSX

Most programming languages have their SDK available for non-system-wide installation. Python offers venv, while Java Development Kit needs only the JAVA_HOME variable to be set,…

check out.
Apache Airflow DAG dependencies data pipelines data-aware scheduling performance tuning
Apache Airflow 15 min read

Managing inter-DAG dependencies in Airflow

In the real world, data pipelines only sometimes come as a completely independent sequence of operations. Usually, they share dependences on one another, occasionally easy…

check out.
big query cartesian join
The technicalities 15 min read

Big cartesian join in Big Query

When working on more advanced analytics (or reports), you may stumble upon the problem of doing a self-join. Be it all pairs of available products,…

check out.
AWS query performance redshift storage organisation
The technicalities 10 min read

How does storage organisation affect query performance?

Despite great efforts to separate interface from the implementation (like SQL), the pesky details always come up important when deploying to production, either when performance…

check out.
storage_organisation_vs_query_performance_TantusData
query performance schema distribution storage organisation
The technicalities 10 min read

Storage organisation vs query performance – examples

The article How does storage organisation affect query performance described a number of principles on how to model data in Amazon…

check out.