Pella Next Generation: Implement Optimization Techniques and Automated Deployments for Cost Savings
Overview
Experience | In Person |
---|---|
Type | Breakout |
Track | Data Engineering and Streaming |
Industry | Manufacturing, Professional Services |
Technologies | Databricks Workflows, DLT, Unity Catalog |
Skill Level | Intermediate |
Duration | 40 min |
In this session, we will explore the “Pella Next Generation” project, towards performance optimization and cost savings using various tools within Databricks data analytics in an Azure cloud. Liquid clustering, managed schema/table within Unity Catalog, BI analytical dashboard to monitor job performance and cost — compute and storage — are used towards performance optimization. Auto Loader and DLT with data quality checks/constraints and a common framework to support data ingestion into the medallion layers of the data lake. Databricks Asset Bundles for workflows/jobs deployment resulted in significant cost savings of $30K/year — conservative number — with a potential savings of up to $40K/year for the organization. This presentation will provide valuable takeaways for professionals looking to implement similar solutions in their own organizations.
Session Speakers
IMAGE COMING SOON
Ragavan Ramasamy
/Pella Corporation