Session

First‑Class Containers Across Databricks: A Unified, Portable Dependency Management Strategy

Overview

ExperienceIn Person
TrackData Engineering & Streaming
IndustryEnterprise Technology
TechnologiesLakeflow, Unity Catalog, Databricks Apps
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES

Databricks is bringing first‑class Docker support everywhere. This session shows how we’re making containers a core part of the Databricks developer experience, delivering faster startup, tighter governance, and clearer observability.

 

We’ll walk through the product architecture, the UX investments that reduce time‑to‑first‑result, and how we approach compatibility and safe rollout for critical workloads at scale.

Session Speakers

Justin Breese

/Staff Product Manager
Databricks

Speaker placeholderIMAGE COMING SOON

Xiao Chen

/Software Engineer
Databricks

Full Summary

Container support comes to standard compute: a new chapter for Databricks dependency management

Dependency management has been a persistent drag on reliability and speed for production workloads on the Databricks Platform. Teams have wrestled with brittle init scripts, unpinned packages, internet hiccups during startup, and security surprises, all of which undermine reproducibility and slow delivery.

FAQ


No. The new standard compute container support intentionally blocks init scripts and cluster libraries on top of the image. The goal is an immutable, reproducible artifact, and allowing mutable layers would undermine both performance and supportability.

Standard compute supports Amazon ECR, Azure ACR, and Docker Hub with the same authentication model as the prior container offering. The early serverless GPU preview starts with Docker Hub and will expand over time.

DBR 18 beta or above is required during the beta period. Clusters must run in standard mode and be configured at creation with the Docker image URI. The beta designation is expected to drop so that DBR 18 or later will suffice.

Yes, that is the design. Standard compute uses the same base image family as serverless, so images built for the current beta are intended to be forward compatible with upcoming serverless notebooks and jobs container support.

By storing images in Unity Catalog and converting them to a lazy load format, Databricks expects cold starts to drop from about 15 minutes to roughly 10 seconds. Artifacts also inherit lineage and ACLs automatically because they live in Unity Catalog.