Streamline Real Time Ingestion and Serving with DLT and Online Tables

Västtrafik is the public transport authority responsible for public transport involving buses, ferries, trains, and trams in West Sweden. BI and Data Analysis Team aims to increase the efficiency and responsiveness of its services by making it more data-driven. We will implement a near real-time ingestion framework for its Travel Planner App to improve service reliability, travel experience, and customer satisfaction. Development and maintenance of such low latency pipelines quickly become difficult as issues like small files, infrastructure cost due to inefficient autoscaling, monitoring, complex setup for serving, etc., are hard to manage. Addressing these challenges requires solutions that reduce complexity and optimize for the right balance between cost and latency. In this talk, we will present our solution, which resolves these challenges using Delta Live Tables and Brickstore. We will showcase DLT’s out-of-the-box capabilities and Brickstore’s simple, fast, scalable serving.