SESSION

Streamline Real Time Ingestion and Serving with DLT and Brickstore

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Engineering and Streaming
INDUSTRYPublic Sector
TECHNOLOGIESDelta Lake, Developer Experience, ETL
SKILL LEVELIntermediate
DURATION40 min

Västtrafik is the public transport authority responsible for public transport involving buses, ferries, trains, and trams in West Sweden. BI and Data Analysis Team aims to increase the efficiency and responsiveness of its services by making it more data-driven. We will implement a near real-time ingestion framework for its Travel Planner App to improve service reliability, travel experience, and customer satisfaction. Development and maintenance of such low latency pipelines quickly become difficult as issues like small files, infrastructure cost due to inefficient autoscaling, monitoring, complex setup for serving, etc., are hard to manage. Addressing these challenges requires solutions that reduce complexity and optimize for the right balance between cost and latency. In this talk, we will present our solution, which resolves these challenges using Delta Live Tables and Brickstore. We will showcase DLT’s out-of-the-box capabilities and Brickstore’s simple, fast, scalable serving.

SESSION SPEAKERS

Sanjeev Kumar

/SSA
Databricks

Magnus Johannesson

/Solutions Architect and Data Engineer
Västtrafik