How does Real-Time Mode compare to specialized streaming engines like Apache Flink?

RTM removes the need to maintain a separate streaming engine by delivering comparable or superior performance natively within the Spark ecosystem. RTM processed events up to 92% faster than Flink, in benchmarks based on customer feature-computation patterns. Because it uses the familiar Spark API, teams avoid the codebase duplication and logic drift associated with running dual-engine architectures.

Does RTM integrate with Lakebase?

RTM has a native Lakebase sink that allows pipelines to serve results directly into Databricks' managed Postgres. It eliminates the need to build custom reverse-ETL pipelines by automatically handling connection pooling, batching, and idempotency. Every write is handled as an upsert on a defined key, ensuring downstream operational apps and dashboards can instantly read fresh data without reconciliation errors.

What are the ideal use cases for Real-Time Mode?

RTM is built for latency-sensitive, operational applications where a delayed decision is a wrong decision, including:

How do I enable Real-Time Mode for my workloads?

If you are already using Structured Streaming, you can enable it with a single configuration update spark.databricks.streaming.realTimeMode.enabled = true and by applying the RealTime trigger to your streaming query or flow. No major codebase rewrites are necessary.

Real-Time Mode in Apache Spark^™ Structured Streaming

Process data in milliseconds 100X faster than Spark micro-batch without a second engine

Watch Demo

Data pipeline with 2 Kafka sinks, 601K records.

Process streaming data in milliseconds on serverless compute

Real-Time Mode (RTM) delivers up to 100x lower latency than Spark micro-batch mode for operational workloads, eliminating the need for separate engines like Apache Flink. By supporting continuous processing, RTM brings end-to-end latency down to as low as 5ms for time-critical workloads. Natively integrated into Apache Spark™ Declarative Pipelines, it ensures data teams achieve extreme performance and the benefits of fully managed services, including versionless execution, automated infrastructure upgrades and low-to-zero downtime maintenance.

Unlock operational workloads with a unified execution engine

Minimize logic drift and codebase duplication

Use the same Spark API for large-scale batch training and ultra-low-latency real-time inference. RTM enables seamless scalability, allowing you to shift pipelines from hourly batches to continuous streaming with a single code change, completely eliminating the need for complex dual-engine architectures.

Read the launch blog

Spark RTM mostly has lower latency than Flink.

Process events up to 92% faster than Flink

RTM is designed for sub-second decision-making, driving impact in use cases such as live fraud detection and real-time personalization. Through continuous data flow, pipeline scheduling and streaming shuffles, RTM achieves strict P99 latencies between 40ms and 300ms across demanding operational workloads.

Read blog post

Micro-batch vs. Real-time data processing modes.

Deliver up to 100x lower latency than micro-batch

Shift from minute-level ETL to millisecond-level operational streaming with a single-line config change. For supported workloads, RTM delivers up to 100x lower latency than standard micro-batch enabling you to power applications within the same Spark ecosystem.

Read blog post

DIVE DEEPER

Learn more about Real-Time Mode

Documentation

Get the specifics on how to set up, optimize and monitor Real-Time Mode queries.

Watch a Demo

See Real-Time Mode in action with a comprehensive credit card fraud detection walkthrough.

Discover more

Explore other integrated, intelligent offerings on the Databricks Data + AI Platform.

Apache Spark™ Declarative Pipelines

Simplify batch and streaming ETL with automated data quality, change data capture (CDC), data ingestion, transformation and unified governance.

Genie Code

Build and maintain data pipelines with agentic AI that understands your data.

Zerobus Ingest

Push event data directly to your lakehouse in near-real-time. Direct write API reduces your operational burden with high throughput and performance at scale.

Unity Catalog

Seamlessly govern all your data assets with the industry’s only unified and open governance solution for data and AI, built into the Databricks Data + AI Platform.

Take the next step

Getting started with Real-Time Mode

Watch a short video to learn how Real-Time Mode supports mission-critical applications with sub-second latency.

Tutorial: Run a real-time streaming workload

Learn how to run real-time streaming queries in Databricks with end-to-end latency as low as 5ms.

Real-Time Mode (RTM) FAQ

Real-Time Mode is an evolution of Apache Spark Structured Streaming that delivers 100X lower latency than Spark micro-batch execution..

By utilizing pipeline scheduling to run query stages simultaneously and streaming shuffles to pass data immediately between tasks, it eliminates latency bottlenecks.

This architecture delivers up to 100x lower latency than Spark Micro-batch Mode for supported operational streaming workloads, bringing processing down from seconds to milliseconds, with end-to-end latency as low as 5 milliseconds.

Ready to become a data + AI company?

Take the first steps in your transformation

Try for free Contact Sales

Real-Time Mode in Apache Spark™ Structured Streaming