How Spark's streaming engine covers everything from hourly incremental jobs to millisecond pipelines with one API.
Structured Streaming is the stream processing engine built into Apache Spark™. You write a streaming job with the same DataFrame and SQL APIs you use for batch, and Spark runs it incrementally as new data arrives. With Real-Time Mode, the same engine now reaches sub-second, end-to-end latency. That means one engine and one codebase cover nightly batch, near-real-time ETL and millisecond operational workloads.
Most teams used to pair Spark for batch with a second engine for low-latency streaming. Each engine had its own code, skills and failure modes. Spark 4 removes the reason for that split.
Structured Streaming treats a live data stream as a table that keeps growing. You write a query against that table, and Spark runs it incrementally, processing only new records and updating the results. Spark tracks progress in a checkpoint, so a restarted query picks up where it left off with exactly-once processing for supported sources and sinks.
Because the streaming API is the batch API, you can prototype a transformation on a static table and then point it at a stream with minimal code changes. Sources include Apache Kafka, cloud object storage and Delta Lake or Apache Iceberg tables.
Choose the trigger by the latency you need, not by the engine. All three run the same query code. Changing latency tiers means changing one line, not rewriting the pipeline.
| Trigger | Typical latency | Best for | Cost profile |
|---|---|---|---|
| availableNow | Minutes to hours (scheduled) | Small or incremental workloads, backfills, cost-sensitive ETL | Compute runs only while there is new data |
| Micro-batch (processingTime) | Seconds | Streaming ingestion and ETL into the lakehouse | Always-on, high throughput |
| Real-Time Mode | Sub-second, end to end | Fraud detection, personalization, real-time AI features | Always-on, latency-optimized |
Table 1. Structured Streaming triggers compared. Sources: Apache Spark Real-Time Mode docs and Databricks Real-Time Mode reference. Latency depends on source, operators, state and sink.
Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of waiting for micro-batch boundaries. It first shipped in Apache Spark 4.1 for stateless Scala queries, and Apache Spark 4.2 added the PySpark trigger. Stateful support, including streaming aggregation, deduplication and transformWithState, has been merged into open source Spark for the upcoming Apache Spark 4.3 release. On Databricks, Real-Time Mode has been generally available since March 19, 2026, with support for stateful workloads, including transformWithState.
Real-Time Mode gets its speed from long-running tasks, continuous data flow and a streaming shuffle that lets stages exchange data without blocking. For the full design, read Breaking the microbatch barrier: the architecture of Apache Spark Real-Time Mode.
You turn it on by changing the trigger on an existing query:
The duration in the trigger controls how often progress is checkpointed, with a 5-second minimum. Records don't wait for it. They're processed as they arrive. Real-Time Mode requires update output mode. It reads from Kafka and writes to Kafka or a row-level foreach sink. To land results in a Delta table, run a separate micro-batch query. See the Real-Time Mode reference for the full list of supported sources, sinks and operators.
Yes. Real-Time Mode closes the latency gap that once made Flink the default for operational streaming, and Spark keeps a single engine for batch, streaming, SQL and machine learning. MakeMyTrip moved its personalization pipeline to Real-Time Mode. P50 latency dropped from about 1.23 seconds to 44 ms, and P99 dropped from more than a minute to about 500 ms. The faster pipeline drove a 7% lift in click-through rates. The team chose Spark specifically to avoid running two engines.
| Percentile | Before (micro-batch) | After (Real-Time Mode) |
|---|---|---|
| P50 | ~1.23 s | 44 ms |
| P99 | >1 min | ~500 ms |
Coinbase and DraftKings also run Real-Time Mode in production, as described in the Real-Time Mode GA announcement. For most teams, the deciding factor is operational. With Spark, the same code, people and governance cover every latency tier. Flink remains an option for organizations already invested in its runtime for specialized event processing.
Yes. Structured Streaming runs in local mode on a laptop with pip install pyspark, and the availableNow trigger processes only the new data, then shuts down. That makes streaming practical for small, incremental jobs that run on a schedule, not just always-on pipelines. When volume grows, the same query moves to a cluster or serverless compute without changes. To try it, follow the Apache Spark tutorial: Real-Time Mode.
Structured Streaming is how AI systems get fresh data. Teams use it to compute features continuously, so a fraud or recommendation model scores against what happened seconds ago, not yesterday. DraftKings uses Real-Time Mode with transformWithState to build unified feature pipelines for both ML training and online inference. MakeMyTrip is testing Real-Time Mode to supply its AI agents with the most recent context.
Streaming is also agent-friendly to write. Structured Streaming queries, including transformWithState, run over Spark Connect, the thin client in Apache Spark. A coding agent can author and submit a streaming plan without holding a cluster session or credentials.
Use transformWithState for custom stateful logic in Spark 4. It replaces the older mapGroupsWithState and flatMapGroupsWithState APIs. It adds typed state variables, timers, time-to-live eviction for expiring state and the ability to initialize state from an existing table. Pair it with the RocksDB state store for large state. For built-in patterns such as windowed aggregation, deduplication and stream-stream joins, use the standard operators and set a watermark so Spark can bound state and handle late data.
Databricks runs Structured Streaming as a managed service built by the original creators of Apache Spark. On top of open source Spark, it adds:
Code written for open source Spark runs on Databricks unchanged. See Structured Streaming on Databricks.
Spark Streaming (DStreams) is the original, legacy streaming API built on RDDs. Structured Streaming is the current engine, built on DataFrames and SQL. All new streaming features, including Real-Time Mode, ship only in Structured Streaming.
Both. Structured Streaming defaults to micro-batch, and Real-Time Mode processes records continuously with sub-second, end-to-end latency. You choose per query by setting the trigger.
For most workloads, no. Real-Time Mode gives Spark sub-second latency on the same engine you use for batch, so you maintain one system, one codebase and one skill set.
Apache Spark 4.1 introduced Real-Time Mode for stateless Scala queries, and Apache Spark 4.2 added the PySpark trigger. Stateful operators arrive in open source with Apache Spark 4.3. On Databricks, Real-Time Mode is generally available, including stateful operations.
Yes. Structured Streaming queries, including transformWithState, can be authored and run from a Spark Connect client.
Subscribe to our blog and get the latest posts delivered to your inbox.