Apache Spark™ Tutorial: Structured Streaming

Structured Streaming treats a stream as a table that keeps growing. The select, where, groupBy, and join operations you already used on a batch table work the same way on a stream, and Spark reads only the rows that arrived since the last run, so the work is incremental rather than a full recompute. Once that clicks, the rest is about triggers, state, and where output lands.

In the Spark SQL module, you ran queries over a table that contained static data. Structured Streaming lets you run the same queries over data that is continuously arriving. Because Spark is a real-time engine with exactly-once guarantees, and it uses the same DataFrame and SQL APIs you already know, most of what you learned carries straight over.

By the end of this module, you can:

• Describe the streaming model and why a streaming query is evaluated incrementally. 
• Read a stream, transform it, and write it to a table with a checkpoint. 
• Choose a trigger and an output mode for a query. 
• Use a watermark to bound late data and let the engine drop old state. 

• Run windowed aggregation and deduplication as stateful streaming operations.

   

Using a coding agent? Copy this prompt to get started quickly

Structured Streaming treats a stream as a table that keeps growing. Each time the query runs, Spark reads the rows that arrived since the last run and updates the result, rather than recomputing it from scratch. The DataFrame and SQL operations you use on a batch table apply unchanged to a stream: select, where, groupBy, and join function the same way. Spark schedules the work, keeps whatever state the query needs, and writes to a table that you can query like any other.

Three choices shape every streaming query:

• A trigger decides when new rows are processed. availableNow=True processes everything available right now and then stops, which is ideal for scheduled, batch-like runs. processingTime="30 seconds" runs a microbatch on a fixed interval. Real-Time Mode, covered in the next module, pushes latency below a second. 
• An output mode decides what the sink receives. append writes only new rows, update writes rows whose values changed, and complete rewrites the whole result each time. 
• A sink decides where results land. A Delta table is the common target on Databricks; Kafka forwards a stream downstream; foreachBatch runs custom logic per microbatch.

Reading a Delta table as a stream picks up new commits incrementally. The transformations after readStream are ordinary DataFrame code, identical to their batch form:

display(stream) renders a live result in the notebook that updates as data arrives; stop it when you are done exploring.

A Kafka source is the common production case. It needs a running broker, so it is shown here for reference rather than to run as is. The payload arrives in value as bytes, and from_json turns it into named columns:

Writing a stream requires a checkpoint. Keep checkpoints in a Unity Catalog volume, and give every query its own directory:

The checkpoint records which input offsets the query has committed and stores the state for stateful queries. On restart, Spark reads the checkpoint and resumes from the last committed offset instead of reprocessing the input. Exactly-once delivery comes from committing the offsets and the output together: a batch whose commit is missing is replayed, and one whose commit is present is skipped.

A watermark bounds how long the engine waits for late rows. Events older than the watermark are dropped, and state for windows below the watermark is released:

The watermark tracks the maximum event time seen in the data, not wall-clock time. A longer watermark accepts more late data at the cost of holding more state. Most stateful operations need a watermark; without one, the engine keeps state for every key it has ever seen.

Windowed aggregation groups events by a time window and a key, then aggregates within each window:

Deduplication keeps one row per key. dropDuplicatesWithinWatermark lets a key reappear after the watermark passes, which suits identifiers that are unique only within a bounded period:

State lives under the checkpoint and grows with the number of distinct keys and the length of the watermark. For large state, Databricks keeps it in a RocksDB state store that spills to disk.

The Delta sink is an ordinary table, so SQL and BI tools read it with no streaming client:

Result

These are the same rows you aggregated in Quick Start, now produced by a streaming write and queried as a plain table. 

The trigger trades latency against cost. availableNow processes the backlog and releases the compute between runs, which fits a schedule. A short processingTime interval keeps compute busy for lower latency. When the business action has to happen within a second, Real-Time Mode is the next step. Whatever you choose, keep each query's checkpoint in a volume so it survives restarts, and never share a checkpoint between queries.

• A stream is a table that keeps growing; the query runs incrementally. 
• A trigger decides when rows are processed, an output mode decides what the sink receives, and a sink decides where results land. 
• A checkpoint stores offsets and state, supporting exactly-once processing and allowing a query to resume after a failure. 
• A watermark bounds late data and lets the engine release old state. 
• Windowed aggregation and deduplication are stateful operations that depend on a watermark.