Pella Next Generation: Implement Optimization Techniques and Automated Deployments for Cost Savings

In this session, we will explore the “Pella Next Generation” project, towards performance optimization and cost savings using various tools within Databricks data analytics in an Azure cloud. Liquid clustering, managed schema/table within Unity Catalog, BI analytical dashboard to monitor job performance and cost — compute and storage — are used towards performance optimization. Auto Loader and DLT with data quality checks/constraints and a common framework to support data ingestion into the medallion layers of the data lake. Databricks Asset Bundles for workflows/jobs deployment resulted in significant cost savings of $30K/year — conservative number — with a potential savings of up to $40K/year for the organization. This presentation will provide valuable takeaways for professionals looking to implement similar solutions in their own organizations.