Apache Spark™ Tutorial: Real-Time Mode

By the end of this module, you can:

• Explain how Real-Time Mode differs from microbatch mode and where its latency comes from.
• Set up a cluster that supports Real-Time Mode.
• Switch a streaming query to the real-time trigger.
• Describe which stateful operations run in Real-Time Mode and the latency to expect.
• Decide when sub-second latency is worth the cost.

Using a coding agent? Copy this prompt to get started quickly:

Real-Time Mode is a trigger for a Structured Streaming query, not a separate API. The source, transformations, checkpoint, and sink all stay the same; the only thing that changes is the trigger. A microbatch query waits for a trigger, collects the rows that arrived, and processes them as a batch. Real-Time Mode starts a long-running task and processes rows inside that task as they arrive.

Latency is set by the slowest operator in the query, not by the trigger. End-to-end P99 latency measures the whole chain, including source queuing and the sink, so if one operator adds time, the rest of the query cannot hide it.

Real-Time Mode runs only on a Classic cluster configured to support it. Before running any snippet in this module, set up the cluster. The requirements are:

• Databricks Runtime 16.4 LTS or newer
• Classic compute (not serverless), with Photon, autoscaling, and spot instances turned off
• The Spark configuration spark.databricks.streaming.realTimeMode.enabled set to true
• Enough task slots to run all stages of the query at once, since each stage runs as a long-running task

Real-Time Mode reads from a streaming message source such as Apache Kafka, Kinesis, Event Hubs, or MSK. It does not read a Delta table as a source, so the examples below use the rate source to generate rows. In a production query, you would replace this with a streaming message source.

NOTE: Real-Time Mode requires a full Databricks workspace with Classic compute. It is not available on Databricks Free Edition, which provides serverless compute only. On Free Edition, use the micro-batch triggers from the Structured Streaming module (such as trigger(availableNow=True)); Real-Time Mode needs a Classic cluster.

Switch a query to the real-time trigger with trigger(realTime=...) and the update output mode. The rest of the query is unchanged:

The realTime option switches the execution mode. The 5 minutes value controls how often progress is reported and the checkpoint is committed, not how quickly rows are read; rows keep moving through the long-running task the whole time. Five minutes is the production setting; a shorter interval helps while debugging. Real-Time Mode supports the update output mode and sinks including Kafka, foreach, console, and memory. foreachBatch is not supported, because it processes a whole batch at once rather than one record at a time.

Stateful operators keep state inside the long-running task, so streaming aggregation and deduplication stay on the low-latency path.

NOTE: A stateful query regroups records with a shuffle, and Real-Time Mode schedules every shuffle partition as its own concurrent long-running task, so it needs one free task slot per shuffle partition. The default spark.sql.shuffle.partitions of 200 fails on a small cluster with CONCURRENT_SCHEDULER_INSUFFICIENT_SLOT, so lower it to fit your cluster's free slots (the aggregation example below does this).

NOTE: Real-Time Mode support for stateful operators is scheduled to arrive in Apache Spark 4.3. It is available in the Databricks Runtime per the setup requirements above.

An aggregation on the real-time trigger:

The groupBy and count operations run inside the long-running task, and the watermark allows old window state to be released.

Deduplication uses the same operators as microbatch mode:

dropDuplicatesWithinWatermark evicts old state as the watermark advances, while dropDuplicates keeps state without that event-time cleanup. Both are supported with the real-time trigger. Custom stateful logic uses transformWithState, which is supported in Python through the row-based API; transformWithStateInPandas is not supported in Real-Time Mode, and transformWithState requires Databricks Runtime 17.3 LTS or newer.

The published stateful benchmarks are 136 ms P99 for dropDuplicates, 139 ms P99 for window aggregation, and 141 ms P99 for transformWithState.

Batch sits at the high end. Microbatch streaming typically has latency measured in seconds. Real-Time Mode targets sub-second latency: as low as 5 ms for a stateless query and typically under 300 ms with state. Watch e2eLatencyMs in query.lastProgress, and measure the sink separately, because that metric does not include sink write time.

Sub-second latency matters when the business action has to happen within a second or two, such as a fraud check, live personalization, or alerting on a sensor. Real-Time Mode requires a continuously running Classic cluster, so the cost is steady and visible. When minutes are acceptable, a scheduled microbatch trigger such as availableNow usually makes more sense and costs less.

Apache Flink is stream-native. Apache Spark is unified, meaning teams don’t need to run separate stacks for batch and streaming workloads. Spark is an industry standard for teams that need one engine to handle streaming, batch, SQL, and machine learning across a lakehouse, regardless of latency requirements.

1. Change realTime="5 minutes" to realTime="30 seconds" and watch how often progress is reported.
2. Add a second aggregation and observe how the task slots are used.
3. Compare query.lastProgress["e2eLatencyMs"] for the stateless and aggregation queries.