Introducing Databricks for a fintech startup

Case study

Introducing Databricks for a fintech startup

Migration from MS SQL to Databricks

Challenge

A finance company has to migrate its analytics to a big data solution, as its to-date approach becomes very cost-inefficient. The previous solution was also becoming a bottleneck as onboarding significantly bigger clients was impossible. The IT team is very proficient in both application development and cloud administration but is new to the Apache Spark ecosystem, so needed proper onboarding.

Solutions

The TantusData team worked closely with the client’s team to discover and implement the analytics algorithms. There was a very short loop of development - testing - validation cycle to identify any differences between the expected and actual results early on and correct them. The client’s team was also up to date with the created data jobs to get familiar with it and be able to take it over and extend it if required.

Technology & Tools

Apache Spark

Azure

Databricks

MS SQL (migration data source)

Client

Tradeteq is a fintech startup based in the UK.

Opportunity

The Databricks platform is more cost-efficient and offers more flexible scaling options than an SQL database (including cloud offerings). Keeping the SQL solution would ramp up the infrastructure cost by an order of magnitude, while the Databricks & Spark platform scales linearly and is significantly more cost-efficient in our client’s case.

Delivery

As a startup, the company focuses on short delivery time, quick feedback loop, and transparency regarding the current state of features in progress. TantusData has successfully accommodated this approach and delivered accordingly. Within a month and a half, all major data jobs were migrated from Azure Functions applications to Apache Spark jobs. The process includes onboarding and familiarising the client’s engineers with the solution.

Effect

The implemented data platform resolved the challenge of scaling the existing MSSQL-backed batch processing. The platform is ready for a 100x increase in events without affecting the transactional backend system. It not only enables future growth but is significantly cheaper than the previous solution.