Session

Ultra Low Latency Streaming for Real-Time Feature Engineering

Overview

Experience	In Person
Track	Data Engineering & Streaming
Industry	Consulting & Services, Retail & Consumer Goods, Financial Services
Technologies	Lakeflow, Lakebase
Skill Level	Beginner
DOWNLOAD SESSION SLIDES

Feature freshness is critical for fraud-detection, ad tech, real-time recommendations, dynamic pricing, etc. Historically, ML engineers have had to maintain two separate systems for feature engineering, leveraging Spark and Spark Structure Streaming for batch/near real-time features, while building bespoke pipelines when they needed sub-second feature freshness. In this session, we'll learn about Real-Time Mode in Apache Spark Structured Streaming, the engine behind the Databricks Feature Store, and how companies like DraftKings are using it to power real-time features with millisecond-level freshness. You'll learn how to pair Real-Time Mode with Lakebase to build a state-of-the-art online feature store and leave with a practical blueprint for a unified feature engineering platform that eliminates logic spread across different systems and shifts the focus from infrastructure orchestration to semantic feature modeling.

Session Speakers

Abhay Bothra

/Sr. Staff Software Engineer
Databricks

IMAGE COMING SOON

Jerry Peng

/Staff Software Engineer
Databricks

Full Summary

Ultra-low latency streaming for real-time feature engineering on Databricks

Real-time fraud detection, dynamic pricing, travel delay prediction, and personalization hinge on fresh features computed in milliseconds. When pipelines lag or diverge across batch and streaming stacks, accuracy drops and costs rise. The video outlines how recent streaming advances on the Databricks Platform close the gap between offline training and online serving so teams can run both on one governed engine.

The conversation focuses on the root causes of online-offline skew, how Real-time mode in Spark Structured Streaming delivers millisecond latencies without code changes, and how new operators and sinks, plus a managed Postgres service, enable complete real-time feature pipelines. For teams that prefer a packaged option, Databricks Feature Store wraps these pieces behind a declarative interface.

Why freshness and consistency fail in dual stack feature pipelines

Most organizations maintain separate offline and online paths: a high-throughput batch pipeline for training and a low-latency pipeline for serving. Duplicate implementations often introduce subtle logic differences that create online-offline skew. Latency in the online path also makes features stale relative to the point-in-time correct features used for training, degrading production accuracy.

Unifying on one engine for both batch and streaming removes an entire class of drift and reduces operational overhead. When the same code and platform compute features for training and serving, consistency improves and latency expectations align.

Real-time mode in Structured Streaming: millisecond latency without rewrites

Real-time mode (RTM) is a new execution mode for Spark Structured Streaming that achieves end-to-end latencies in the milliseconds with no API changes. Enabling RTM is a one-line trigger change on an existing streaming query.

Two core changes drive the speedup. RTM uses long-lived batches with time-based checkpoint intervals to minimize per-batch overhead. It also runs stages concurrently with an incremental streaming shuffle, sending records directly from mappers to reducers as soon as they are ready. RTM is also available on Lakeflow Declarative Pipelines, both classic and serverless, so teams can pair low latency with managed reliability and operations.

From operator to store: assembling the real-time feature stack

A fast engine is only part of an online feature platform. New building blocks on Databricks address the rest:

Native rolling window operator: Computes aggregations between now and a look-back duration, expiring old events as time advances. It avoids stale lookups from tumbling windows and the overhead of dense sliding windows, and keeps values fresh even when no new events arrive.
Managed Postgres with a lake-based architecture: Designed for write-heavy feature workloads, with disaggregated storage and compute, autoscaling to zero, and Unity Catalog governance. It addresses the scaling and cost pain seen with read-optimized systems such as Redis and DynamoDB.
Native JDBC streaming sink: Handles batching, connection pooling, retries, backpressure, upserts, and authentication out of the box, replacing fragile custom foreach sinks. Reads from the store can be single-digit to tens of milliseconds at scale.

Prefer not to assemble components by hand? Databricks Feature Store packages these capabilities behind a declarative API. A single feature definition provisions an offline Delta Lake replica for training, a batch tiling pipeline, and a continuous RTM-powered pipeline that writes to the online store, all governed in one place.

Results in production: Coinbase and MakeMyTrip

Coinbase reduced tail latency from roughly 850 to 900 milliseconds to around 100 milliseconds by switching triggers to RTM, achieved about 99 percent online-offline feature consistency, and cut costs by nearly 50 percent. MakeMyTrip consolidated separate Spark and Flink stacks onto one engine, improving P99 latency from over a minute to under 500 milliseconds and lifting click-through rate by 7 percent.

Turning feature freshness into a configuration choice

A unified stack that combines RTM, rolling windows, a native streaming sink, and a governed Postgres store makes real-time features practical without a second system. Teams can either compose these primitives on the Databricks Platform or use the Databricks Feature Store to get consistent, fresh, and cost-efficient features with a configuration change instead of a migration.

FAQ

No. RTM is enabled by changing the trigger on an existing query. APIs, transformations, and application code stay the same, which is why teams can validate and roll out quickly.

Tumbling windows have fixed boundaries, so a lookup between boundaries returns stale or partial aggregates. Sliding windows with short slides compute many unnecessary windows, slowing queries and overloading the online store. The rolling window operator computes only the current look-back window and expires events as time advances.

Feature pipelines are write-heavy, while Redis and DynamoDB are optimized and priced for read-heavy workloads. The managed Postgres service supports high-throughput writes, uses disaggregated storage and compute, can autoscale to zero, and is governed by Unity Catalog alongside offline tables.

A single declarative feature definition sets up three pipelines: an offline Delta Lake replica for training and backfills, a batch tiling pipeline to the online store, and a continuous RTM-powered pipeline that reads from the streaming source and writes to the store.

Autoscaling for RTM on Lakeflow Declarative Pipelines is on the roadmap, with stateless support in public preview soon and stateful support following next quarter. The native Postgres sink is moving to public preview on Lakeflow Declarative Pipelines, and foreach sink support in Lakeflow Declarative Pipelines is in progress for teams that still need to write to systems like Redis or DynamoDB.