Rethinking Distributed Systems for Serverless Performance and Reliability

Building truly serverless compute for Apache Spark required solving fundamental architectural challenges that have existed since Spark’s inception. The complexity goes far beyond simply creating warm pools of machines or implementing basic autoscaling. It required rethinking core assumptions about how distributed computing systems should operate.

Traditional Spark deployments expose infrastructure directly to users, creating tight coupling between applications and compute. Workloads compete for shared resources, small inefficiencies can cascade into failures, and users are forced to manually balance performance, cost, and reliability. As demand changes, systems struggle to maintain both high utilization and predictable performance.

Serverless compute takes a different approach by fully managing the infrastructure so that  the user can focus on the data and insights. Stability becomes a system property rather than a user responsibility, enabled by architectures that isolate workloads, intelligently place them, and dynamically adapt resources. 

Serverless compute is designed to improve stability, performance, and operational simplicity. Three core systems make this possible:

1. Spark Connect, which separates user applications from compute infrastructure
2. The Serverless Gateway, which intelligently routes workloads across compute resources
3. An adaptive autoscaler, which continuously optimizes cluster size for performance and cost 

Together, these systems enable a model where performance is achieved by first ensuring stability across the system.

Spark Connect: Stability Through Isolation 

Spark Connect represents the most significant architectural transformation in Spark's history, a complete departure from the monolithic design that has defined distributed computing for over a decade. In traditional architectures, user applications run directly on the same machine as the Spark driver, creating tight coupling that introduces critical limitations. When multiple applications compete for resources on the same cluster or when user code consumes excessive memory or CPU, the system becomes unstable, leading to failures that can cascade across workloads. 

Spark Connect introduces a client-server architecture in which applications communicate with the Spark driver over gRPC, and the driver executes queries on behalf of the client rather than running user processes directly. This shifts the unit of execution from application processes to queries and enables a clean separation between user applications and infrastructure.

This decoupling significantly improves reliability and allows the platform to manage drivers independently of user workloads. By isolating applications from compute, Spark Connect creates the foundation required for stable multi-tenant execution and enables more advanced resource management across the system.

This architecture enables Databricks to deliver more than 25 major Spark runtime upgrades per year with a 99.998% success rate across more than 4.5 billion workloads, with no user action required.¹

The Gateway: Balancing Efficiency and Predictability

Distributed systems have long faced a fundamental tension between efficiency and predictability. Maximizing utilization often leads to resource contention, while isolating workloads can result in underutilized capacity. Traditional cluster models force users to navigate this tradeoff manually, often resulting in unpredictable performance or unreliable execution as workloads change.

Consider what happens when dozens of queries land simultaneously: some small exploratory scans running against sample data, others large production ETL jobs processing hundreds of gigabytes. A naive router treats them identically, forcing large jobs to wait behind small ones or letting workloads compete for the same cluster, leading to unpredictable performance degradation. This dynamic makes it difficult to deliver both high utilization and consistent performance in shared environments.

The Databricks gateway routes each workload by evaluating three real-time signals: estimated query size (derived from the logical plan), current utilization across the cluster pool, and latency profile: whether a session is interactive and latency-sensitive or a batch job optimized for throughput. A small exploratory query gets routed to a lightly loaded cluster that can respond in seconds; a heavy ETL job gets directed to a cluster with available headroom for its data volume, or the autoscaler is signaled to provision one. When conditions shift (a cluster fills up, a long-running job finishes, a new cluster comes online), the gateway continuously re-evaluates placements and corrects routing without user intervention. The result: workloads are insulated from each other. A runaway query on one cluster doesn't delay queries on another, and the system maintains high utilization without sacrificing predictability.

Autoscaling: Optimizing the Cost-Performance Curve

Dynamic cluster sizing is the primary mechanism for optimizing price-performance in distributed systems, but determining the optimal amount of compute is inherently complex. The optimal configuration depends on workload characteristics, data size, and the relative importance of latency versus cost, with no single configuration working across all scenarios. Databricks serverless offers two modes to fit different needs: Standard, which uses less compute to reduce costs, and Performance-Optimized, which delivers faster startup and execution for time-sensitive workloads. 

Startup is a priority for us, and serverless Notebooks and Workflows have made a huge difference. Serverless compute for notebooks makes it easy with just a single click. — Chiranjeevi Katta, Data Engineer at Airbus

Databricks helped us move to serverless compute, while eliminating redundant workflows. These efficiencies put us in position to lower operational costs by 25%. Pipelines on our legacy infrastructure previously took hours to process. Now, they run 2 to 5 times faster.  — Evan Cherney, Senior Data Science Manager at Unilever

Traditional autoscaling approaches rely on static rules and reactive thresholds, which often fail to capture these nuances. As a result, clusters are frequently under or over-provisioned, leading to inefficiency, instability, or both. 

Serverless autoscaling takes a more adaptive approach. By continuously analyzing workload patterns and system-wide signals, the autoscaler positions each workload on the optimal cost-performance curve, where most manually configured clusters fall short, delivering worse performance and higher cost due to the difficulty of correctly sizing distributed systems. It dynamically adjusts compute capacity by scaling horizontally and vertically as needed, preventing out-of-memory failures and maintaining stability as workloads grow. When a task encounters an out-of-memory error, the autoscaler automatically detects it, restarts the task on a larger VM, and continues the job with no manual intervention or job failure required. 

The impact is measurable. CKDelta reported jobs completing in 20 minutes that previously ran for 4–5 hours. Unilever saw pipelines running 2–5x faster with operational costs down 25%. HP realized cloud savings of over 32% and decreased combined job runtime by 36%.

Together, Spark Connect, the gateway, and the autoscaler enable a fundamentally different operating model for Spark. Workloads are isolated, intelligently placed, and dynamically resourced without user intervention. By addressing stability at the architectural level, serverless compute can deliver strong performance while maintaining reliability, allowing users to focus on building data and AI workloads rather than managing infrastructure.

¹ Justin Breese et al., "Blink Twice: Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries," SIGMOD/PODS '25, pp. 103–106. https://doi.org/10.1145/3722212.3725084

Start Your Serverless Journey Today

• Sign up to a Serverless Databricks account
• Serverless Compute for Notebooks, Lakeflow Jobs, and Spark Declarative Pipelines
• Practitioner’s Guide to Serverless Compute