Apache Spark™ Tutorial: Spark Connect
Spark Connect lets a Spark client run anywhere and still reach a cluster. Your code builds the same DataFrame plan it always would, but the client sends that plan to a server instead of running it locally, and the server returns results as they're produced. The client stays thin as a result: no JVM, no matching package set, and nothing that has to run next to the cluster.
If you completed the Quick Start module, you've already been using Spark Connect. Databricks serverless notebooks run on it: your code builds a query locally and sends it to Spark running elsewhere. This module explains what that split is, why it exists, and how to use it from your own applications.
Learning objectives
By the end of this module, you can:
- Describe the client and server split that Spark Connect introduces.
- Explain why Databricks serverless already uses it, and what that buys you.
- Connect to Spark from a thin Python client.
- Run your own Connect server and point a client at it.
- Switch an existing Spark application to Connect with one configuration.
Using a coding agent? Copy this prompt to get started quickly:
What Spark Connect is
Spark Connect is a gRPC-based protocol that lets a client application talk to a remote Spark server. The client builds an unresolved query plan as you call select, where, and other operations, and sends that plan to the server. The server analyzes it, optimizes it with Catalyst, runs it, and returns the result as Arrow data.
That split removes the tight coupling the older model had. Before Connect, your application and the Spark driver ran in the same process, so your Python version, your packages, and the JVM all had to agree, and a client could not connect unless it matched the cluster. Connect puts a network boundary between the two, so the client stays small and the server owns the engine.
You are already using it
On Databricks, Spark Connect is not something you have to manually enable. Serverless compute uses it, Databricks Connect is built on it, and standard compute uses it for Python notebooks on Databricks Runtime 14.3 and above and Scala notebooks on 13.3 and above. Every query you have run in this tutorial has gone through this client-server path already, which is why you never created a SparkContext or matched a cluster version.
Connect from Python
Outside a Databricks notebook, the thin client is a small package that connects to an existing server. It uses the normal pyspark imports but needs no Spark JARs and no local Java:
pyspark-client only connects; it cannot start a server of its own. The sc:// URI names the host and port, and port 15002 is the default. You can also set the address in the environment with SPARK_REMOTE, so the same script runs against whichever server that variable names. Because the client is this small, a web service, a CI job, or a local script can open a session the same way a notebook does.
Run your own Connect server
To run a server yourself, you use a full Spark installation rather than the thin client. A Spark download ships the server scripts:
The server owns the JVM, the packages, and the static configuration, so it starts with whatever the workload needs. Dependencies belong on the server, not the client:
Match the connector version to the server's Spark version exactly, or the streaming source will fail to load. Package and JVM settings now live on the server, where they belong, and the client keeps only the settings that vary per session.
Move an existing job to Connect
An existing application that uses a full Spark install can switch to Connect with one configuration. Set spark.api.mode to connect and keep your master; Spark runs the job through a local Connect server:
Because the change is one line, a team can migrate one job, watch it in production, and move it back if anything is wrong. That's a far lower-risk path than rewriting every job at once.
Clients in other languages
A client implements the gRPC protocol, not the engine, so it can be written in any language. A Rust client embeds Spark without a JVM, and Go and Swift clients exist as well. They all speak the same protocol to the same server, so one server can serve clients in several languages. This lets tools, services, IDEs, or even agents work with Spark without embedding a Spark installation.
Why use Spark Connect?
- An application talks to Spark over a small client, so Spark can be embedded in any app, notebook, IDE, or service.
- The server upgrades independently of clients, so a client no longer has to match the cluster version.
- The protocol is language agnostic, so Python, Scala, Rust, Go, and Swift clients all reach the same server.
- On Databricks, Spark Connect is the foundation that lets external tools and drivers connect to compute, and it is why serverless works the way it does.
Try it yourself
- In a notebook, run spark.range(5) and note that no SparkContext was created.
- Read the Connect versus Classic page and list which Databricks compute types use Spark Connect.
- Sketch where the client ends and the server begins for a query that reads a table and aggregates it.
Recap
- Spark Connect is a gRPC protocol that splits a thin client from the Spark server.
- Databricks serverless and notebooks already run on it, which is why there is no SparkContext to manage.
- pyspark-client connects to a server; a full Spark install runs a server or uses spark.api.mode=connect.
- The protocol is language agnostic, so one server serves Python, Scala, Rust, Go, and Swift clients.
What's next
That completes the tutorial. You have read and transformed data with DataFrames and SQL, streamed it in microbatches and in Real-Time Mode, built declarative pipelines, trained a model, and connected to Spark from a thin client. For deeper references, see the Apache Spark on Databricks documentation and the Spark Connect overview.