Skip to main content

What is Spark Structured Streaming?

How Spark's streaming engine covers everything from hourly incremental jobs to millisecond pipelines with one API.

by Databricks Staff

  • Structured Streaming is Apache Spark's stream processing engine. It uses the same DataFrame and SQL APIs as batch, so one codebase covers both.
  • Real-Time Mode brings Spark streaming to sub-second, end-to-end latency, so most teams no longer need a separate engine like Apache Flink.
  • Pick the trigger by latency need, from availableNow for small incremental jobs to Real-Time Mode for fraud detection and AI features.

Structured Streaming is the stream processing engine built into Apache Spark™. You write a streaming job with the same DataFrame and SQL APIs you use for batch, and Spark runs it incrementally as new data arrives. With Real-Time Mode, the same engine now reaches sub-second, end-to-end latency. That means one engine and one codebase cover nightly batch, near-real-time ETL and millisecond operational workloads.

Most teams used to pair Spark for batch with a second engine for low-latency streaming. Each engine had its own code, skills and failure modes. Spark 4 removes the reason for that split.

How does Structured Streaming work?

Structured Streaming treats a live data stream as a table that keeps growing. You write a query against that table, and Spark runs it incrementally, processing only new records and updating the results. Spark tracks progress in a checkpoint, so a restarted query picks up where it left off with exactly-once processing for supported sources and sinks.

Diagram of a data stream appended as rows to an unbounded input table, processed incrementally by a Spark query into an updating result table.
Figure 1. Structured Streaming treats a stream as a table that keeps growing. Each trigger processes only the new rows and updates the result.

Because the streaming API is the batch API, you can prototype a transformation on a static table and then point it at a stream with minimal code changes. Sources include Apache Kafka, cloud object storage and Delta Lake or Apache Iceberg tables.

Which trigger should I use: availableNow, micro-batch or Real-Time Mode?

Choose the trigger by the latency you need, not by the engine. All three run the same query code. Changing latency tiers means changing one line, not rewriting the pipeline.

TriggerTypical latencyBest forCost profile
availableNowMinutes to hours (scheduled)Small or incremental workloads, backfills, cost-sensitive ETLCompute runs only while there is new data
Micro-batch (processingTime)SecondsStreaming ingestion and ETL into the lakehouseAlways-on, high throughput
Real-Time ModeSub-second, end to endFraud detection, personalization, real-time AI featuresAlways-on, latency-optimized

Table 1. Structured Streaming triggers compared. Sources: Apache Spark Real-Time Mode docs and Databricks Real-Time Mode reference. Latency depends on source, operators, state and sink.

What is Real-Time Mode in Spark Structured Streaming?

Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of waiting for micro-batch boundaries. It first shipped in Apache Spark 4.1 for stateless Scala queries, and Apache Spark 4.2 added the PySpark trigger. Stateful support, including streaming aggregation, deduplication and transformWithState, has been merged into open source Spark for the upcoming Apache Spark 4.3 release. On Databricks, Real-Time Mode has been generally available since March 19, 2026, with support for stateful workloads, including transformWithState.

Real-Time Mode gets its speed from long-running tasks, continuous data flow and a streaming shuffle that lets stages exchange data without blocking. For the full design, read Breaking the microbatch barrier: the architecture of Apache Spark Real-Time Mode.

Timeline comparing Spark micro-batch execution with gaps between batches to Real-Time Mode, where long-running tasks process records continuously between checkpoints.
Figure 2. Micro-batch waits for batch boundaries and blocking shuffles. Real-Time Mode runs stages continuously and checkpoints in the background.

You turn it on by changing the trigger on an existing query:

The duration in the trigger controls how often progress is checkpointed, with a 5-second minimum. Records don't wait for it. They're processed as they arrive. Real-Time Mode requires update output mode. It reads from Kafka and writes to Kafka or a row-level foreach sink. To land results in a Delta table, run a separate micro-batch query. See the Real-Time Mode reference for the full list of supported sources, sinks and operators.

Is Spark Structured Streaming competitive with Apache Flink?

Yes. Real-Time Mode closes the latency gap that once made Flink the default for operational streaming, and Spark keeps a single engine for batch, streaming, SQL and machine learning. MakeMyTrip moved its personalization pipeline to Real-Time Mode. P50 latency dropped from about 1.23 seconds to 44 ms, and P99 dropped from more than a minute to about 500 ms. The faster pipeline drove a 7% lift in click-through rates. The team chose Spark specifically to avoid running two engines.

Bar chart showing MakeMyTrip P50 latency falling from about 1.23 seconds to 44 ms and P99 from over a minute to about 500 ms with Spark Real-Time Mode.
Figure 3. MakeMyTrip cut P50 latency from about 1.23 seconds to 44 ms and P99 from over a minute to about 500 ms after adopting Real-Time Mode.
PercentileBefore (micro-batch)After (Real-Time Mode)
P50~1.23 s44 ms
P99>1 min~500 ms

Coinbase and DraftKings also run Real-Time Mode in production, as described in the Real-Time Mode GA announcement. For most teams, the deciding factor is operational. With Spark, the same code, people and governance cover every latency tier. Flink remains an option for organizations already invested in its runtime for specialized event processing.

Can Structured Streaming handle small workloads?

Yes. Structured Streaming runs in local mode on a laptop with pip install pyspark, and the availableNow trigger processes only the new data, then shuts down. That makes streaming practical for small, incremental jobs that run on a schedule, not just always-on pipelines. When volume grows, the same query moves to a cluster or serverless compute without changes. To try it, follow the Apache Spark tutorial: Real-Time Mode.

REPORT

The agentic AI playbook for the enterprise

How does Structured Streaming support AI and agents?

Structured Streaming is how AI systems get fresh data. Teams use it to compute features continuously, so a fraud or recommendation model scores against what happened seconds ago, not yesterday. DraftKings uses Real-Time Mode with transformWithState to build unified feature pipelines for both ML training and online inference. MakeMyTrip is testing Real-Time Mode to supply its AI agents with the most recent context.

Streaming is also agent-friendly to write. Structured Streaming queries, including transformWithState, run over Spark Connect, the thin client in Apache Spark. A coding agent can author and submit a streaming plan without holding a cluster session or credentials.

How do I manage state in a streaming query?

Use transformWithState for custom stateful logic in Spark 4. It replaces the older mapGroupsWithState and flatMapGroupsWithState APIs. It adds typed state variables, timers, time-to-live eviction for expiring state and the ability to initialize state from an existing table. Pair it with the RocksDB state store for large state. For built-in patterns such as windowed aggregation, deduplication and stream-stream joins, use the standard operators and set a watermark so Spark can bound state and handle late data.

What are Structured Streaming best practices in 2026?

  1. Pick the trigger by latency need. Start with availableNow, move to micro-batch for seconds and Real-Time Mode for sub-second.
  2. Always set a watermark on stateful queries so state stays bounded.
  3. Use transformWithState for custom state, with the RocksDB state store.
  4. Give every query its own checkpoint location and treat it as part of the deployment.
  5. Write to open table formats such as Delta Lake or Apache Iceberg, governed by a catalog.
  6. Declare pipelines with Spark Declarative Pipelines instead of wiring jobs by hand. Spark manages dependencies and incremental refresh. Read Bringing Declarative Pipelines to the Apache Spark open source project.
  7. Watch the latency metrics for Real-Time Mode. Batch-duration metrics don't reflect end-to-end latency.
  8. Don't build new work on DStreams. It's the legacy streaming API.

Structured Streaming on Databricks

Databricks runs Structured Streaming as a managed service built by the original creators of Apache Spark. On top of open source Spark, it adds:

  • Real-Time Mode with stateful support: generally available, including transformWithState and initial state loading
  • Lakeflow Spark Declarative Pipelines: declarative streaming tables
  • Serverless compute: no cluster sizing
  • Unity Catalog: governance for every streaming table

Code written for open source Spark runs on Databricks unchanged. See Structured Streaming on Databricks.

FAQs

What is the difference between Spark Streaming and Structured Streaming?

Spark Streaming (DStreams) is the original, legacy streaming API built on RDDs. Structured Streaming is the current engine, built on DataFrames and SQL. All new streaming features, including Real-Time Mode, ship only in Structured Streaming.

Is Spark streaming real time or micro-batch?

Both. Structured Streaming defaults to micro-batch, and Real-Time Mode processes records continuously with sub-second, end-to-end latency. You choose per query by setting the trigger.

Do I need Apache Flink if I use Spark?

For most workloads, no. Real-Time Mode gives Spark sub-second latency on the same engine you use for batch, so you maintain one system, one codebase and one skill set.

Which Spark version supports Real-Time Mode?

Apache Spark 4.1 introduced Real-Time Mode for stateless Scala queries, and Apache Spark 4.2 added the PySpark trigger. Stateful operators arrive in open source with Apache Spark 4.3. On Databricks, Real-Time Mode is generally available, including stateful operations.

Does Structured Streaming work with Spark Connect?

Yes. Structured Streaming queries, including transformWithState, can be authored and run from a Spark Connect client.

Get started with Real-Time Mode on Databricks

Get the latest posts in your inbox

Subscribe to our blog and get the latest posts delivered to your inbox.