Databricks에서 Apache Spark Structured Streaming의 실시간 모드 일반 공급 발표

For years, Apache Spark Structured Streaming has powered some of the world’s most demanding streaming workloads. However, for ultra-low latency use cases, teams needed to maintain separate, specialized engines — most commonly Apache Flink, alongside Spark, duplicating codebases, governance models, and operational overhead. Now, Databricks removes this burden for customers. 

Today, we are excited to announce the General Availability of Real-Time Mode (RTM) in Spark Structured Streaming, bringing millisecond-level latency to the Spark APIs you already use. Be it detecting fraud in real-time, or generating fresh, real-time context to steer your AI agents, you can now use Spark to power all of these use cases.

Powering industry-leading customers and use cases

RTM has already been adopted by teams at industry-leading organizations across financial services, e-commerce, media, and ad tech to power fraud detection, live personalization, ML feature computation, and ad attribution.

Coinbase, one of the world’s leading cryptocurrency exchanges, uses RTM to scale their high-frequency risk management and fraud detection engines—processing massive volumes of blockchain and exchange events with the sub-100ms latency necessary to secure millions of digital asset transactions.

By leveraging Real-Time Mode in Spark Structured Streaming, we’ve achieved an 80%+ reduction in end-to-end latencies, hitting sub-100ms P99s, and streamlining our real-time ML strategy at massive scale. This performance allows us to compute over 250 ML features all powered by a unified Spark engine.”—Daniel Zhou, Senior Staff Machine Learning Platform Engineer, Coinbase

DraftKings, one of North America's largest sportsbook and fantasy sports platforms, uses real-time mode to power feature computation for their fraud detection models — processing high-throughput betting event streams with the latency and reliability required for real-money wagering decisions.

​​In live sports betting, fraud detection demands extreme velocity. The introduction of Real-Time Mode together with the transformWithState API in Spark Structured Streaming has been a game changer for us. We achieved substantial improvements in both latency and pipeline design, and for the first time, built unified feature pipelines for ML training and online inference, achieving ultra-low latencies that were simply not possible earlier.”—Maria Marinova, Sr. Lead Software Engineer, DraftKings

MakeMyTrip, one of India’s leading online travel platform for hotels, flights, and experiences, adopted Real-Time Mode to power personalized search experiences. RTM processed high-volume traveler searches to deliver real-time recommendations.

In travel search, every millisecond counts. By leveraging Spark Real-Time Mode (RTM), we delivered personalized experiences with sub-50ms P50 latencies, driving a 7% uplift in click-through rates. RTM has also transformed our data operations, enabling a unified architecture where Spark handles everything from high-throughput ETL to ultra-low-latency pipelines. As we move into the era of AI agents, steering them effectively requires building real-time context from data streams. We are experimenting with Spark RTM to supply our agents with the richest, most recent context necessary to take the best possible decisions.” —Aditya Kumar, Associate Director of Engineering, MakeMyTrip

RTM can support any workload that benefits from turning data into decisions in milliseconds. Some example use cases include:

• Personalized experiences in retail and media: An OTT streaming provider updates content recommendations immediately after a user finishes watching a show. A leading e-commerce platform recalculates product offers as customers browse - keeping engagement high with sub-second feedback loops.
• IoT monitoring: A transport and logistics company ingests live telemetry to drive anomaly detection, moving from reactive to proactive decision-making in milliseconds.
• Fraud detection: A global bank processes credit card transactions from Kafka in real time and flags suspicious activity, all within 200 milliseconds - reducing risk and response time without replatforming.

What Is Real-Time Mode (RTM)?

RTM is an evolution of the Spark Structured Streaming engine that enables it to achieve sub-second performance in benchmarking demanding feature engineering customer workloads. 

Structured Streaming’s default microbatch mode (MBM) is like an airport shuttle bus that waits for a certain number of passengers to board before departing. On the other hand, RTM operates like a high-speed moving walkway, eliminating the limitation to wait for the shuttle bus to fill up. RTM processes each event as it arrives, providing end-to-end millisecond latency without leaving the Spark ecosystem. 

From seconds to milliseconds: RTM transforms the Spark engine by replacing periodic batching with a continuous data flow, eliminating the latency bottlenecks of traditional ETL.

RTM’s performance gains come from three key architectural innovations:

• Continuous data flow: Data is processed as it arrives instead of discretized, periodic chunks.
• Pipeline scheduling: Stages run simultaneously without blocking, allowing downstream tasks to process data immediately without waiting for upstream stages to finish.
• Streaming shuffle: Data is passed between tasks immediately, bypassing the latency bottlenecks of traditional disk-based shuffles.

Together, they transform Spark into a high-performance, low-latency engine capable of handling the most demanding operational use cases.

Spark RTM: Up to 92% faster than Flink, enabling teams to operate less infrastructure, move faster

In order to validate the performance of Spark RTM, we benchmarked the performance against a popular specialized engine, Apache Flink based on actual customer workloads performing feature computation. These feature computation patterns are representative of most low-latency ETL use cases, such as fraud detection, personalization, and operational analytics.When comparing Spark RTM with Flink, the results demonstrate that Spark's evolved architecture provides a latency profile comparable to specialized streaming frameworks. For more information, on the data sets and queries referenced, see this GitHub repository.

One engine, up to 92% faster: RTM outpaces specialized engines like Flink, proving that millisecond-level operational analytics no longer requires a separate streaming engine. Source: Internal benchmarks based on customer feature computation patterns. Full queries available on GitHub.

While raw speed matters, Spark RTM’s greatest advantage over engines like Flink is the simplicity it offers builders. It allows teams to use the same Spark API for both batch training and real-time inference, effectively eliminating "logic drift" and codebase duplication. Spark RTM enables seamless scalability, where a single-line code change can shift a pipeline from hourly batches to sub-second streaming without manual infrastructure tuning. Ultimately, by reducing operational complexity and the need for multiple specialized systems, teams can develop and deploy real-time applications significantly faster with Spark RTM.

Getting started with Spark RTM

Getting up and running with RTM is straightforward. If you’re already using Structured Streaming, you can enable it with a single configuration update - no rewrites required.

Step 1: Configure your cluster

RTM is currently available on Classic compute, across both Dedicated and Standard access modes. RTM is supported on Databricks Runtime (DBR) 16.4 and above; however,we recommend DBR 18.1 for the latest features and optimizations. During cluster creation, add the following Spark configuration:

Step 2: Use the new Real-Time Trigger in your streaming query

What’s New with Spark RTM

Since launching in Public Preview in August 2025, Databricks has continued to expand RTM’s capabilities, based on customer feedback. 

이번 GA 릴리스의 새로운 기능은 다음과 같습니다:

• Apache Spark 4.1의 OSS 지원 (상태 비저장 변환): 상태 비저장 변환의 RTM(실시간 모드)이 이제 오픈 소스 Apache Spark 4.1에서 제공됩니다. OSS Spark를 기반으로 구축하는 팀은 실시간 모드를 활용하여 프로젝션, 필터링 및 UDF 기반 파이프라인을 처리할 수 있습니다.
• 표준 액세스 모드 지원: RTM은 이제 Python에서 클래식 컴퓨팅의 전용 및 표준 액세스 모드 모두에서 작동하여 팀이 스트리밍 워크로드 전반에 걸쳐 컴퓨팅 리소스를 활용하는 방식에 더 많은 유연성을 제공합니다.
• 비동기 상태 체크포인팅 및 진행 상황 추적: 상태 및 쿼리 진행 상황 체크포인팅이 이제 비동기적으로 수행되어 이벤트 처리 중요 경로에서 분리됩니다. 이를 통해 상태 비저장 및 상태 저장 파이프라인의 실시간 모드 지연 시간이 개선됩니다.
• transformWithState의 초기 상태 로드: transformWithState는 사용자 지정 상태 저장 로직을 구축하기 위한 강력한 Spark Structured Streaming 연산자입니다. 사용자는 이제 Real-Time Mode와 함께 transformWithState를 사용할 때 기존 쿼리의 체크포인트 또는 델타 테이블에서 초기 상태를 로드할 수 있습니다. 이 기능은 상태 저장 기능 엔지니어링에 중요하며, "제로에서 시작"하지 않고 기록적 맥락을 온라인 쿼리에 미리 채울 수 있도록 합니다.
• UDF에 대한 향상된 메트릭 및 관찰 가능성: StreamingQuery 리스너를 통해 Python UDF 실행에 대한 더 정확한 지연 시간 메트릭이 제공됩니다.
• Python 상태 저장 UDF 성능 향상: 특히 RTM 쿼리의 경우 Python transformWithState에서 상태 저장 작업의 성능을 개선하기 위한 최적화가 추가되었습니다.

결론

RTM은 Apache Spark Structured Streaming을 새로운 종류의 워크로드, 즉 스트리밍 데이터에 대한 즉각적인 응답을 요구하는 운영상의 지연 시간 민감 애플리케이션으로 확장합니다. 팀이 이미 사용하고 있는 Spark API에 초당 1초 미만의 지연 시간을 제공함으로써, 가장 시간 제약적인 파이프라인을 위해 별도의 전문 엔진을 운영할 필요가 없어집니다. 사기 탐지 파이프라인, 개인화 엔진 또는 ML 기능 계산 시스템을 구축하든, 실시간 모드는 Spark의 단순성과 생태계 폭넓음을 활용하여 애플리케이션에 필요한 지연 시간을 제공합니다.

기술 자료

지금 바로 RTM을 시작하는 데 도움이 되는 자료를 확인해 보세요:

• 문서: Structured Streaming의 실시간 모드
• 주문형 동영상: 실시간 모드 시작하기
• 블로그: Databricks Lakebase와 Spark RTM 구성을 통한 실시간 사기 탐지 달성 방법
• 코드 예제: 실시간 모드 예제
• 주문형 웨비나: 실시간 모드 기술 심층 분석: 초당 300밀리초 미만으로 구축한 방법

(이 글은 AI의 도움을 받아 번역되었습니다. 원문이 궁금하시다면 여기를 클릭해 주세요)