Streaming Data for Operational Apps: Building the Bridge From Analytics to Serving

Real‑time operational applications need instant query responses, but analytical platforms optimize for throughput over latency. Running separate operational and analytical databases leads to duplicate infrastructure, complex sync logic and data consistency issues. We needed sub‑second dashboards for factory operators to track machine performance in real time, while also enabling historical analysis and ML on the same data. Our previous platform stored only four days of raw data and lacked proper engineering practices. We built a streaming architecture on Databricks that serves operational queries via Lakebase Postgres and analytical workloads via Delta Lake from a single pipeline, eliminating the dual‑database problem. This session covers the architecture, our custom streaming sink, validation of the new platform alongside the legacy system, early mistakes and patterns for complex state logic, exactly‑once delivery, and measurable production outcomes.