Beyond Medallion: Architecting Disney’s DATOS for Complex, Real-Time Data Streams With Databricks

Disney’s Medallion architecture laid a strong foundation for enterprise data governance, but emerging needs—ranging from personalized user experiences to real-time financial insights—required a richer, near real-time view across domains such as subscriptions, billing and payments. DATOS addresses this by serving as a contextual aggregation and abstraction layer built on the Databricks Lakehouse Platform. Using Structured Streaming and Lakeflow Spark Declarative Pipelines, DATOS ingests diverse sources like Kafka, Kinesis, and data lakes while absorbing upstream schema and delivery variability to provide stable, reusable data products. Liquid Clustering enhances performance on high‑volume tables, and Unity Catalog ensures unified governance and security. This session explores how DATOS reduces operational overhead, decouples producers and consumers, accelerates onboarding, and delivers a context-aware source of truth that powers real-time analytics, Unity Dashboards, and advanced AI/ML workloads.