Migrating to Serverless: The Databricks Journey
Overview
| Experience | In Person |
|---|---|
| Track | Data Warehousing |
| Industry | Enterprise Technology |
| Technologies | Databricks SQL, Unity Catalog |
| Skill Level | Beginner |
| DOWNLOAD SESSION SLIDES | |
The future of the data intelligence platform is serverless, and Databricks is leading by example. We aren’t just telling customers to migrate; the Data Platform team is actively moving our own mission-critical ETL and BI workloads to Databricks Serverless SQL and Compute first. This session chronicles our internal migration journey. We will share the raw “before and after” metrics on cost, latency and engineering overhead. We will be transparent about the challenges we faced as early adopters and the specific best practices we developed to run a massive-scale, serverless-first platform that powers the business.
Session Speakers
Alan Jackoway
/Sr. Staff Software Engineer
Databricks
Zack Loebel
/Staff Software Engineer
Databricks
Full Summary
Inside Databricks' migration to serverless: lessons from moving 10,000+ jobs
Databricks' internal team migrated more than 10,000 production jobs from classic compute to serverless, confronting the same quota fights, upgrade churn, and oversizing habits that challenge many data platforms. The outcome, shared candidly, shows a large-scale migration that reduced operational toil and cost while surfacing the real blockers that still need attention.
FAQ
Usually no. The team's rule was to change configuration only, not code. Exceptions include structured streaming workloads that hit serverless feature gaps and Python UDFs that exceed the 1 GB memory ceiling, which may require refactoring.
Expect a small percentage to regress, especially on standard mode. Consider switching those to serverless performance mode for tighter SLAs, roll back using the captured prior configuration, or raise patterns to your account team so product engineering can address them.
Make serverless the default, restrict who can create classic compute with policies, and require an exception process for classic usage. Closing the inflow is essential or new classic jobs will outpace your migration.
Databricks SQL and Lakeflow Declarative Pipelines change with a single setting. All-purpose clusters are next if you already use cluster policies. Scheduled jobs are highest value and hardest, so start with low-SLA dev and test workloads on platforms your team owns.
Watch for ANSI SQL enforcement tied to DBR upgrades, which can be disabled if needed, the 1 GB memory limit on Python UDFs including map and mapPartitions, any use of RDD APIs, and structured streaming feature gaps. The internal team did not hit DBFS issues because they operate on Unity Catalog.