Databricks Network Configuration delivery to Tens of Millions of Serverless VMs

Summary

• Databricks' serverless platform launches tens of millions of VMs daily, and each VM needs network configuration such as allowed destinations and private endpoints, before serving customer workloads. With each node fetching config at startup and polling for updates throughout its lifetime, this translates to billions of network config requests per day. The old architecture fetched this from multiple upstream services synchronously, creating latency, and availability bottlenecks.
• We re-architected network config delivery using event-driven pipelines and snapshot pre-computation, reducing RPC latency by 97.5% (5,000ms → 125ms), achieving 99.99% service availability.

Problem Statement

Databricks' serverless compute platform powers virtually all of our data and AI products, such as, SQL warehouses, notebooks, ML serving endpoints, and more. The platform launches tens of millions of VMs daily across AWS, Azure, and GCP.

Before any serverless workload can execute, the VM needs to know its network configuration: What storage destinations can it access? Are there private link endpoints through which it should route traffic? Is there recent changes in Unity Catalog that grants access to new storage destinations? Do we start consuming new destinations shared via Delta Sharing?

The challenge is that network configuration is not stored in any single place. It must be assembled from multiple upstream services, each contributing a piece of the full picture.

The Old Architecture

In the original design, every time a serverless cluster started, our network configuration service would synchronously call all upstream services, aggregate their responses, compute the per-workspace network configuration, and return it to the serverless dataplane. This happened on the critical path of cluster creation.

While the old architecture was simple and worked well with small scale, this architecture suffered from fundamental problems, reflected in the following metrics we track on our operational dashboard:

1. Latency: With multiple upstream services on the critical path, the RPC latency for serving network configuration was 5,000ms at p99. This impacted serverless cluster start up latency.
2. Server Success Rate: Each upstream service has its own availability characteristics. With several services in series, the compound availability drops quickly, translating to increased likelihood of severless cluster launch failures per year.

As serverless usage continued its rapid growth, the synchronous model became increasingly unsustainable. Each synchronous call triggered expensive operations across all workspaces, often doing duplicated computation. This added load that grew proportionally with the number of tenants and their configured resources.

Solution: Event Driven Precomputation

We performed a ground-up re-architecture of how Databricks delivers network configuration. It is built on the core principles:

1. Event-driven pipeline: Instead of making synchronous calls to all upstream services, the new system subscribes to change events via a message queue. When a customer creates a new Unity Catalog connection or modifies a network policy, the upstream service emits an event. The system processes it and updates the pre-computed configuration.
2. Snapshot pre-computation: Network configurations are computed asynchronously in the background and stored in a pre-computed snapshot store. The serving path becomes a single, thin storage fetch, completely decoupled from the upstream services.
3. Static stability: In the event of any upstream service outage, we can maintain a static config, providing static stability for the serverless clusters.

The architecture cleanly separates two paths. The management path runs asynchronously in the background: upstream services emit change events to a message queue, which an event processor consumes to resolve which workspaces are affected and fan out per-workspace update notifications. A local event manager then fetches the relevant details from upstream, recomputes the workspace's network configuration, and stores the result in a pre-computed snapshot store. A periodic reconciler also re-syncs all workspaces in the background, ensuring eventual consistency even if events are missed. The serving path, by contrast, is critical and fast: when a serverless cluster starts up and needs network configuration, the network configuration service serves it directly from the snapshot store with a single storage read, requiring no upstream service calls and meaningfully reducing load on upstream services.

Key Design Decisions

• Upstream services push change events to the message queue. The system processes these events in the background. A low-frequency reconciler periodically re-syncs all workspaces as a safety net, providing the reliability of synchronous framework with the efficiency of push.
• Network configurations are computed and stored locally within each service partition, co-located with the workspaces they serve. This distributes computation, reduces blast radius during incidents, and eliminates cross-partition dependencies on the serving path.
• Events carry only workspace and resource identifiers. This keeps events lightweight, makes them idempotent (they can be replayed in any order), and avoids transferring sensitive customer data through the messaging pipeline.

How Events Flow

When a customer creates a new Unity Catalog connection, Unity Catalog emits a change event to the message queue. The event processor then receives the event, determines which workspaces are attached to the affected metastore, and fans out a per-workspace update notification. In each workspace's partition, the event manager receives this notification, fetches the updated connection details, recomputes the workspace's network configuration, and stores it with a new version mark. From that point on, when a serverless cluster requests the network config, it is served directly from the snapshot store with no upstream calls needed.

Impact

After rolling out the new architecture, the results were transformative across all operational metrics:

Beyond the topline metrics:

1. Upstream call volume reduced by 86%. The system only calls upstream services when an event indicates a change, not on every request.
2. We observed meaningful improvement in the freshness of the networking configuration.
3. Legacy synchronous framework fully deprecated.

Conclusion

This project taught us several lessons about operating network infrastructure at cloud scale:

Pre-computation decouples critical paths. By moving expensive aggregation to the background, the serving path becomes trivially simple and fast. This is the single most impactful architectural decision. It turned a multi-service dependency chain into a single storage read.

Event-driven architecture trades consistency for scalability and reconciliation provides the safety net. Event-based push handles the common case efficiently, while a periodic reconciler catches anything that falls through the cracks.

Design for extensibility from day one. The modular, stage-based architecture means adding support for a new upstream data source requires only a new stage implementation with zero changes to the core pipeline. As Databricks' product surface expands, the network configuration system scales with it.

Today, this system serves billions of network config requests per day across Databricks' global serverless fleet, with ~125ms latency and 99.99% availability. As serverless compute continues its rapid growth, the event-driven architecture ensures that network configuration delivery scales right alongside it.

We're always looking for engineers who enjoy tackling distributed systems challenges at global scale. If problems like these excite you, we'd love to hear from you, please check out open roles at databricks.com/careers!