Data Utility Architecture: A Lakehouse Serving the Whole Enterprise

In 2020, Zhamak Dehghani introduced the Data Mesh concept, uniting data teams but keeping operational and analytical data planes separate. The Data Utility Architecture (DUA), powered by Lakehouse and streaming technologies, breaks down the wall between these planes, uniting all enterprise data. DUA envisions data as a commodity and the pipelines as a utility that serves the whole enterprise regardless of role or persona. It ingests, transforms, and delivers data on demand for any purpose, challenging previous norms that saw operational and analytical data as distinct. Instead, DUA handles all data uniformly, simplifying enterprise data architecture and addressing specific consumer needs with appropriate Service Level Agreements (SLAs), models, and storage. This session will delve into how to reimagine enterprise architecture as a utility using tools like Spark Structured Streaming, Delta tables, and Delta Sharing based on production experience by PwC Canada's Tax DataOps team.