Enabling a Low-Latency Data Mesh at Warner Music Group Using Lakebase, Iceberg and Unity Catalog

Today’s enterprises face daunting challenges when creating impact with data, from fundamental trust issues to dynamically evolving AI use cases. Legacy systems with technical debt impede business value even as users expect trustworthy, low-latency insights for both natural language queries and classic insights. Furthermore, pipelines are often distributed across different engines. This environment demands a flexible approach to drive impact that brings cohesion to an often-messy data environment.

This session describes how WMG Tech leveraged Databricks to develop a declarative data platform that realizes a compositional data mesh designed to tackle this challenge. This architecture includes:

• Iceberg as an engine agnostic data layer for access to diverse sources
• Unity Catalog for compositional access to data products
• Lakebase for native low latency analytical APIs

Together, this platform enables an array of data products that serve AI queries and WMG's applications.