Applying Software Development Principles to a Data-Driven World: Automating SDP from Dev to Prod

Are you a data architect struggling to bring rigor and best practices to your data operations? This session highlights how the Intermountain Health Data Teams transformed their data platform by overcoming cultural resistance and technical gaps to fully automate production-grade data pipeline deployments. By combining Databricks Asset Bundles, GitHub Actions, and Spark Declarative Pipelines, they built a reliable CI/CD workflow supporting nearly all workloads. Parameterized deployments manage dev and prod seamlessly, while a trunk based Git strategy avoids the complexity of multi branch models. The impact: over 90% improvement in deployment reliability, fewer hotfixes, and 20% faster processing. And because transformation depends on people, we’ll share how Intermountain guided teams away from GUI only ETL habits, building trust, clarity, and excitement for this modern, maintainable approach.