Spark on Databricks

Databricks serverless compute lets you run many Spark workloads without provisioning or managing compute infrastructure. Databricks handles the underlying compute lifecycle, including provisioning and scaling, so you can focus on code and data.

Serverless compute is available for supported notebooks, jobs, and Lakeflow pipelines. Some Spark APIs and workload types still require classic compute, so the right option depends on how your application is built.

Photon is the Databricks native vectorized query engine. It accelerates SQL workloads and DataFrame API calls and can reduce runtime and total cost per workload. Because Photon is compatible with Apache Spark APIs, many existing workloads can benefit without application code changes.

Apache Spark supports batch and streaming workloads through a common set of APIs and execution concepts. Spark Structured Streaming lets teams build streaming applications with the same data engineering tools they already use for batch processing.

On Databricks, Delta Lake provides a common storage layer for batch and streaming data. The same data can also be used for analytics, machine learning, and AI.

Real-Time Mode (RTM) is a Structured Streaming capability introduced in Apache Spark 4.1. It is designed for workloads that need continuous processing and much lower latency than traditional microbatch execution.

Spark Declarative Pipelines (SDP) is an open source framework in Apache Spark for building reliable batch and streaming pipelines. You define datasets and transformations, while Spark manages the execution graph, dependency ordering, parallelism, checkpoints, and retries. It adds Databricks capabilities for production operations, including managed compute, data quality expectations, observability, event logging, and integration with Unity Catalog.

Transformation code is designed to remain portable to other runtimes that support Apache Spark Declarative Pipelines.

Lakeflow is the Databricks data engineering solution for ingestion, transformation, and orchestration:

• Lakeflow Connect brings data from databases, SaaS applications, and other sources into Databricks.
• Spark Declarative Pipelines manages batch and streaming transformations.
• Lakeflow Jobs orchestrates production workflows and dependencies.
• Lakeflow can run on serverless or classic compute, depending on the pipeline and workload requirements.

Unity Catalog provides centralized access control, auditing, lineage, and discovery for data and AI assets on Databricks.

For Spark workloads, this gives teams one governance layer across tables, files, models, and other governed assets, with consistent policies across supported workspaces and clouds.

Spark Connect is an Apache Spark client and server protocol that separates client applications from the Spark runtime. A client can communicate with a remote Spark server instead of running inside the same process as the driver.

Databricks uses Spark Connect in experiences such as serverless compute and Databricks Connect. Databricks Connect is the Databricks product that lets local applications and IDEs connect to Databricks compute.

Use Lakeflow Jobs to schedule and orchestrate Spark workloads in production. Databricks also provides monitoring and observability tools to help teams understand job performance, failures, and resource use.

Many existing Spark workloads can reuse the same Spark APIs on Databricks, which can reduce the amount of application code that needs to change.

Migration requirements still depend on the workload. Compute type, libraries, language, runtime dependencies, and API usage can require changes. For example, serverless compute supports Spark Connect APIs but does not support RDD APIs or R workloads.