Modernizing Data Pipelines with Databricks: IAS Journey

Our legacy pipelines struggled to handle data complexity, hindering insights and slowing innovation. We needed a modern platform built for the future. Embracing the Data Mesh paradigm on a single enterprise Lakehouse, we migrated hundreds of complex Airflow pipelines to Databricks Workflows, simplifying orchestration and boosting efficiency. With Databricks Unity Catalog as the central governance, many teams now share data and insights seamlessly, fostering cross-functional analytics and faster decision-making. Our goal is to fully transition from batch model to real-time, and Databricks has helped us beat the waiting game immensely. In this talk, we will cover our lessons learned: How we migrated Airflow pipelines to Databricks workflows with minimal disruption—managing sensitive data without affecting development timelines and building sensors and operators for managing pipeline dependencies Unleashing the power of automation and CI/CD to maximize engineers productivity.