Apache Spark™ Tutorial: Spark Declarative Pipelines
Spark Declarative Pipelines (SDP) allows you to write Spark code without worrying about orchestration, letting you focus on the transformations. Instead of providing a list of steps in order, you describe the tables you want and the query that produces each one, and Spark reads those queries to work out the dependency order on its own. Add a table or point one at a new source, and the graph updates on the next run, with no orchestration list to maintain.
Declarative pipelines are built to be easy to use. You write each table as an ordinary query, and the engine does the rest.
Learning objectives
By the end of this module, you can:
- Explain how a declarative pipeline infers its dependency graph from the queries.
- Mix streaming and batch processing, and SQL and Python, in a single pipeline.
- Tell a streaming table from a materialized view, and choose the right one.
- Map a medallion architecture onto dataset types.
- Apply a change feed to a table with Auto CDC.
Using a coding agent? Copy this prompt to get started quickly:
Overview
Declarative pipelines are flexible and intuitive. A single pipeline can combine streaming and batch processing: a streaming table ingests new rows incrementally, while a materialized view recomputes an aggregate from the full dataset, with both in the same graph. You can also combine SQL and Python in one pipeline, choosing whichever language fits each step. The queries reference tables by name, so it doesn’t matter which language defines them.
Because the dependency graph is derived from the code, a pipeline can be reorganized or extended without rewriting orchestration. Add a dataset or change a read, and the graph updates on the next run. On Databricks, Lakeflow Declarative Pipelines builds on Spark Declarative Pipelines, adding platform capabilities such as integrated governance, an editor, and monitoring.
Streaming tables and materialized views
A pipeline produces datasets in the form of streaming tables and/or materialized views, and the two kinds differ in how the source is read:
- A streaming table ingests data incrementally, one batch of new rows at a time. It suits sources that keep growing, such as a Kafka topic, a directory of new files, or an upstream streaming table.
- A materialized view recomputes its result from the full dataset. It suits aggregations, joins across all rows, and batch sources.
The source determines the type, and the engine checks the choice when the pipeline runs. This helps catch early mistakes, such as using a streaming table for an aggregation that must reflect changes to existing rows. Streaming tables don’t support updates to existing records by default, so a materialized view is a better choice here.
Medallion mapping
The medallion layout follows directly from that split:
- Bronze lands raw events in a streaming table.
- Silver cleans and conforms them, still incrementally, in a streaming table.
- Gold aggregates across the full dataset, which requires a materialized view.
Declare a pipeline in Python
Decorate a function with @dp.table() for a streaming table or @dp.materialized_view() for a materialized view. The function returns the query, and the decorator publishes the result. Pipeline functions stay pure and return a DataFrame; the runtime handles the writes.
Because silver_trips reads bronze_trips and daily_revenue reads silver_trips, the dependencies are in the code, and the engine orders the three on every run. daily_revenue reads with spark.read (batch) because an aggregation has to see all the relevant rows.
Declare a pipeline in SQL
Here’s the SQL form, which declares the same datasets. CREATE OR REFRESH is the pipeline convention. STREAM(...) marks a read as incremental, which makes the target a streaming table. A query without STREAM produces a materialized view.
Running this pipeline over the sample data builds the three tables and the gold result:
Result: daily_revenue
pickup_date | revenue | trips |
|---|---|---|
2016-01-01 | 3998.0 | 326 |
2016-01-02 | 4074.0 | 338 |
2016-01-03 | 3405.5 | 276 |
2016-01-04 | 4011.3 | 325 |
2016-01-05 | 4056.5 | 358 |
Auto CDC
Auto CDC keeps a table in sync with a change feed: a stream of inserts, updates, and deletes from a source system. It’s a compelling reason to reach for Spark Declarative Pipelines on its own, because it replaces the handwritten merge logic that change data capture usually demands, including the ordering and deduplication that make that logic fragile. For details on configuration, see the Databricks AUTO CDC documentation.
Declare the target table, then declare the flow that applies the changes to it:
keys identifies the row to match, and the flow applies each change in sequence_by order, so late or out-of-order events still land correctly. The stored_as_scd_type choice decides what the target keeps:
- stored_as_scd_type=1 keeps only the latest row per key, overwriting it in place.
- stored_as_scd_type=2 keeps the full history of each key, so you can query what a row looked like at any point in time.
Either way, you declare the intent, and the engine maintains the table as the change feed advances.
Run and monitor
Run a pipeline with the spark-pipelines command in open source Spark, or with Lakeflow on Databricks. On Databricks, the pipeline editor shows the live DAG and the update history for each dataset, so you can see which table updated, how long it took, and where any failures occurred, without building a separate observability layer. See the documentation on configuring pipelines for details.
The gold table is an ordinary table, so a query reads the latest result:
Try it yourself
- Add a gold materialized view that aggregates revenue by pickup_zip instead of by date.
- Add a column to silver_trips, for example F.hour("tpep_pickup_datetime") as pickup_hour, and rerun; watch the DAG update.
- Change silver_trips from a streaming table to a materialized view, read the validation error, then change it back.
Recap
- You declare datasets and queries; the engine infers the dependency graph from the reads.
- Streaming tables ingest data incrementally; materialized views recompute from the full dataset.
- Medallion layers map onto those types: typically, bronze and silver are streaming tables, while gold is a materialized view.
- Python decorators and SQL CREATE OR REFRESH declare the same datasets.
- Auto CDC applies a change stream to a target table declaratively.
What's next
Machine Learning continues the tutorial: it shows you how to read a prepared table like the gold dataset here, assemble features, train a model with MLlib, and track the run with MLflow.