From Oracle Cloud to a Governed Lakehouse: Real-Time Streaming With Spark Declarative Pipelines

Flutter SEA invested heavily in OCI, but platform limits—high costs, low observability and operational complexity—became blockers for real‑time engagement, event processing and fast delivery. The previous Spark Structured Streaming architecture used custom code with minimal monitoring, no standardized data quality and limited governance, causing slow iteration and high maintenance.This session shares the migration to a governed, observable Databricks Lakehouse using Spark Declarative Pipelines and Delta Lake, built by a “triforce” team: Flutter SEA, Databricks PS and Reply. We’ll cover architectural choices like single‑plex vs. multi‑plex pipelines and their impact on scalability, cost and simplicity, plus data quality, end‑to‑end monitoring, KPIs - 40% savings and 96% bronze-to-silver performance improvements, lessons learned and best practices for modernizing streaming systems to reduce cost, improve governance and enable AI‑driven real‑time engagement.