Session

Pella Next Generation: Implement Optimization Techniques and Automated Deployments for Cost Savings

Overview

ExperienceIn Person
TypeBreakout
TrackData Engineering and Streaming
IndustryManufacturing, Professional Services
TechnologiesDatabricks Workflows, DLT, Unity Catalog
Skill LevelIntermediate
Duration40 min

In this session, we will explore the “Pella Next Generation” project, towards performance optimization and cost savings using various tools within Databricks data analytics in an Azure cloud. Liquid clustering, managed schema/table within Unity Catalog, BI analytical dashboard to monitor job performance and cost — compute and storage — are used towards performance optimization. Auto Loader and DLT with data quality checks/constraints and a common framework to support data ingestion into the medallion layers of the data lake. Databricks Asset Bundles for workflows/jobs deployment resulted in significant cost savings of $30K/year — conservative number — with a potential savings of up to $40K/year for the organization. This presentation will provide valuable takeaways for professionals looking to implement similar solutions in their own organizations.

Session Speakers

IMAGE COMING SOON

Ragavan Ramasamy

/Pella Corporation