Sponsored by: Zipher | Dynamic Compute Allocation: Runtime Placement to Reduce Costs by up to 60%
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology, Communications, Media & Entertainment, Transportation |
| Technologies | Databricks SQL, Lakeflow, Unity Catalog |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
Choosing compute for data workloads is often treated as a static decision: this job runs on Serverless, that pipeline on Classic compute, this workload uses a pool, and this stream runs on an all-purpose cluster. But production environments are dynamic: load changes by run, day, week, and business cycle. The same job may process twice as much data during business hours while facing the same tight SLA constraints. Supply conditions also shift constantly: spot pricing, instance availability, and spare capacity in pools or clusters. This session frames compute selection as a runtime placement problem and gives teams a practical toolbox for efficient allocation across SKUs.
Session Speakers
Dan Coster
/Co-Founder & CEO
Zipher
Samuel Sterling
/Database Architect
Hertz
Full Summary
Why static compute configurations are costing data teams millions
Data and AI teams are shipping more workloads than ever, yet most still size clusters once and leave them unchanged. Static configurations, often tuned to a peak that rarely occurs, drive chronic over-provisioning and unexpected spend while masking performance headroom. On the Databricks Platform, workload counts are rising sharply as copilots and agentic apps make creating jobs and queries cheap and fast, which makes one-time tuning untenable. Teams evaluating where to focus optimization efforts can find a useful starting point in the data and AI use cases Databricks customers have documented, which illustrates the breadth of workload types now running on modern platforms.
FAQ
Hourly swings, weekday versus weekend seasonality, sliding-window growth within a month, parameter-driven variance across runs, and structured streaming micro-batch fluctuations all create mismatches with fixed clusters. Provisioning for peaks leaves most runs over-provisioned and expensive.
In the TPC-DS benchmarks presented, serverless cost almost three times more than on-demand classic compute and much more than spot-backed classic clusters. Serverless can still be the right fit for specific SLAs, but teams should evaluate it workload by workload rather than adopt it as a blanket default.
Photon cut durations substantially on larger transformations but increased cost by two to six times. On smaller datasets, runtime differences were minor and non-Photon was cheaper. Photon is best used when tight SLAs mandate faster completion, and avoided when timing is flexible.
Spot reclaim probabilities vary by hour, availability zone, weekday versus weekend, and month. The speakers advocate runtime policies that factor in current spot conditions and proactively reshuffle data when termination signals appear, which reduces retries and failures while capturing savings.
The collector gathers operational metadata only, such as resource utilization and Spark logs. No underlying tables, PII, or application code are accessed, and placement decisions are derived from these runtime signals.