Fixing the other side of custom made.

Case study

Fixing the other side of custom made.

Customer visitation in physical stores: the refactored data backend for effective bespoke solution.

Challenge

The in-house built replacement for legacy footfall tracking solution turned out to come with very high and ever rising data processing costs. On top of that, the deadline for decommissioning the legacy solution was fast approaching and the data product was not finished.

Solutions

The customer expected to receive a data product compliant with the company-wide Data Mesh architecture. The remaining deadline for the basic version was just 1 month. In order to meet those expectations we had to refactor data pipelines and model data to rapidly reduce the processing costs, increase ease-of-use and usage costs for consumers. Consumer-facing tables were enriched with basic information relevant for most common use-cases so they can be used both as stand-alones as well as with other data products within the mesh.

Technology & Tools

BigQuery

Dataflow,

Pub/Sub

Cloud Run

Cloud Functions

Github

Python

Data Mesh

Client

Global scale leader in furniture retailing in both ecommerce and brick-and-mortar.

Opportunity

The solution is the sole global source of information for how many customers visit the stores. Footfall is one of the key KPIs for the stores. Other uses of the tool support planning with calculation of the required floor cleaning frequency, manpower for checkout lines, reducing food wastage in restaurants and alerting in case of sudden spikes in visitation. It was also used to comply with store occupancy regulations related to COVID-19 pandemic.

Delivery

The starting point was a comprehensive analysis of all data sources, models and pipelines existing within the project. We have verified these against the requirements related to both the data mesh architecture and end goals. Working together with the in-house team responsible for the solution, we’ve refactored existing components and operations on them to better suit project needs. These were made to comply with best practices for technologies used. Enabling good cost management involved designing the data pipelines and BigQuery tables to process just the required amounts of data. Additionally we’ve introduced a set of data quality metrics and critical issue alerts. After implementing those changes we’ve brought in the historical data from the legacy system. By transforming it according to the design we’ve stream-lined footfall data. The last step consisted of documenting the data in the Data Catalog and making it available to all users.

Effect

Data Product for footfall data was well received by the stakeholders and the wider audience it was introduced to through Data Catalog and other means. The cost of data processing was reduced by ~85% compared to the initial state. It has since remained at a stable level.