How Real-Time Mode brings millisecond latency to Spark Structured Streaming with a one-line trigger change, and when to use it.
Real-Time Mode is a trigger for Apache Spark™ Structured Streaming that processes each record as soon as it arrives instead of waiting for the next micro-batch. It brings Spark streaming from seconds down to milliseconds of end-to-end latency, on the same engine and with the same APIs you already use for batch and micro-batch streaming.
Before Real-Time Mode, teams that needed sub-second responses often added a second, stream-only engine. Real-Time Mode removes that requirement for most workloads. You change the trigger on an existing query, and Spark runs it continuously.
Real-Time Mode replaces micro-batch scheduling with long-running tasks that keep reading and processing data. A micro-batch query waits for a batch boundary, schedules tasks and runs a blocking shuffle between stages, which sets a latency floor of roughly a second. Real-Time Mode removes that overhead with three changes, explained in depth in Breaking the microbatch barrier: the architecture of Apache Spark Real-Time Mode.
The trigger interval controls how often Spark commits a checkpoint and reports progress. It doesn't control how long a record waits. Records flow through the long-running task the whole time, and checkpoints commit in the background.
Change the trigger on a Structured Streaming query to realTime and set update output mode. The rest of the query stays the same.
The minimum trigger interval is 5 seconds, and five minutes is a common production setting. On Databricks, enable Real-Time Mode on classic compute running Databricks Runtime 16.4 LTS or later, with Photon, autoscaling and spot instances turned off, and set spark.databricks.streaming.realTimeMode.enabled to true. You can also run it on serverless compute inside Lakeflow Spark Declarative Pipelines in continuous mode.
Size for every stage at once. Because all stages run concurrently, the cluster needs a task slot for every task in the query. A Kafka source with 8 partitions plus one shuffle with 20 partitions needs 28 slots.
Real-Time Mode supports Kafka-based sources and sinks, row-level foreach sinks and most streaming operators, including stateful ones. This table summarizes the current support on Databricks. See the Real-Time Mode reference for the complete list.
| Area | Supported | Not supported |
|---|---|---|
| Sources | Apache Kafka, AWS MSK, Amazon Kinesis, Event Hubs via the Kafka connector | Delta tables as a source |
| Sinks | Kafka, foreach (ForeachWriter) | Delta tables, foreachBatch |
| Output mode | Update | Append, complete |
| Stateless operators | Projections, filters, UDFs, stream-static joins | None of note |
| Stateful operators | Aggregations, tumbling and sliding windows, deduplication, transformWithState (row-based) | Session windows, transformWithStateInPandas, event-time timers |
| Compute (Databricks) | Classic compute, Lakeflow pipelines on classic or serverless | Serverless notebooks and jobs, Free Edition |
Table 1. Real-Time Mode support on Databricks. Source: Databricks Real-Time Mode reference.
To land Real-Time Mode results in a Delta table, write them to Kafka and run a separate micro-batch query from Kafka to Delta.
Real-Time Mode is generally available on Databricks and is rolling out across open source Apache Spark releases. On Databricks, it has been generally available since March 19, 2026, including stateful operators and transformWithState.
| Release | What it added |
|---|---|
| Apache Spark 4.1 | Real-Time Mode for stateless Scala queries |
| Apache Spark 4.2 | PySpark Real-Time Mode trigger |
| Apache Spark 4.3 | Stateful operators: aggregation, deduplication and transformWithState |
| Databricks (GA, March 2026) | Stateless and stateful queries, initial state loading, Lakeflow pipeline support |
Table 2. Real-Time Mode availability. Sources: Spark 4.1.0 release notes, Apache Spark Real-Time Mode docs.
Real-Time Mode reaches single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries in published benchmarks. Actual latency depends on the source, operators, state size, sink and cluster capacity.
| Workload | Published p99 latency | Source |
|---|---|---|
| Stateless transformation | 3 ms | Databricks latency benchmark |
| Stream-static join | 9 ms | Databricks latency benchmark |
| Stateful aggregation (feature computation) | 14 ms | Databricks latency benchmark |
| dropDuplicates | 136 ms | Databricks stateful benchmark |
| Window aggregation | 139 ms | Databricks stateful benchmark |
| transformWithState | 141 ms | Databricks stateful benchmark |
Table 3. Published Real-Time Mode latency results. Sources: latency-benchmarks repository and the Real-Time Mode tutorial.
To monitor a running query, use the Real-Time Mode latency metrics, not batch duration. Batch-duration metrics don't reflect end-to-end latency, and the reported end-to-end metric excludes sink write time. See optimize and monitor Real-Time Mode performance.
In production, MakeMyTrip cut P50 latency from about 1.23 seconds to 44 ms and P99 from more than a minute to about 500 ms, which drove a 7% lift in click-through rates.
Use the standard streaming operators for aggregation and deduplication, and use transformWithState for custom per-key logic. State lives inside the long-running task, so stateful work stays on the low-latency path. Set a watermark so Spark can release old state.
For custom stateful logic in Python, use the row-based transformWithState API, which requires Databricks Runtime 17.3 LTS or later. The pandas variant, transformWithStateInPandas, isn't supported in Real-Time Mode. See build a custom stateful application.
Use Real-Time Mode when the business action has to happen within a second, and use a cheaper trigger when it doesn't. Real-Time Mode runs continuously on dedicated compute, so its cost is steady.
When seconds or minutes are acceptable, micro-batch or the availableNow trigger is the better choice and costs less. Because the code is the same, you can start with a cheaper trigger and switch later.
Yes, for most workloads. Real-Time Mode gives Spark the millisecond latency that once made Apache Flink the default for operational streaming, while keeping one engine for batch, SQL and machine learning. In the Databricks benchmark, Real-Time Mode matched Flink on stateless work and had lower p99 latency on stateful aggregation and stream-static joins, which Databricks summarized as up to 92% faster. For a full comparison, read Apache Spark vs Apache Flink.
Databricks was founded by the original creators of Apache Spark and runs Real-Time Mode as a generally available feature. On top of open source Spark, it adds:
Code written for open source Spark runs on Databricks unchanged. To try it, follow the Apache Spark tutorial: Real-Time Mode.
Databricks Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of in micro-batches. It brings Spark streaming down to millisecond, end-to-end latency without a code rewrite.
No. Continuous processing was an earlier experimental mode with limited operator support. Real-Time Mode is a new design with long-running tasks, pipeline scheduling and a streaming shuffle, and it supports stateful operators.
Yes. On Databricks, Databricks Real-Time Mode supports aggregations, deduplication and transformWithState today. In open source Apache Spark, stateful support arrives with Spark 4.3.
Not directly. Real-Time Mode writes to Kafka or a foreach sink. To land results in a Delta table, run a separate micro-batch query from Kafka to Delta.
Databricks Real-Time Mode runs on classic compute and in Lakeflow Spark Declarative Pipelines, including serverless pipelines. It doesn't run in serverless notebooks or jobs, so it isn't available in Databricks Free Edition notebooks.
Subscribe to our blog and get the latest posts delivered to your inbox.