Data in various formats and via multiple protocols.

Case study

Data in various formats and via multiple protocols.

Bespoke framework to apply the same pattern for ingesting data into Data Lake regardless of access method and data format.

Challenge

The aim was data ingestion for Quants to support their decisions as well as enabling the Consumption of structured and flat data coming from upstream systems in various formats and via various protocols. Although the first version of a solution supporting few use cases was already created. It was far from being ready to be shared with a wider audience. It required standardisation, automation, and heavy refactoring. Additionally, some activities had to be put behind the algorithm instead of doing them manually.

Solutions

The time of delivery was shortened and the use-cases were easier to reason about. Both with lowered manual work required.

Technology & Tools

Airflow

Bash

Scala

Python

Groovy

Kafka

Jenkins

Spark

Impala

Client

One of the biggest ecommerce companies in Sweden.

Opportunity

Permitting ingestion of multiple data formats and from various sources facilitates the process, cutting the logistics and supporting decision making. At the same time the costs were reduced as one solution was able to efficiently handle the data and manual labour decreased. The value for the end clients was shorter delivery time.

Delivery

The CI/CD and deployment scripts were introduced. Selected module functionalities were moved into better suited places. A generative approach to transformations and schema defining was introduced which allowed for further and more precise customization of its artefacts.

Effect

The robustness of releasing new versions was improved significantly. The unified implementation made the projects of all use-cases easier to reason about. The time in which a given use case is delivered to a client is shortened. Overall the manual activities required during development of a use case were minimised.

Testimonial

Director of Data & Analytics, Retail Company