Building Production Data Products: HDC Journey from Asset Bundle Automation to Genie Orchestration

Building production data products demands operational excellence beyond architecture. This session reveals MiQ's Historical Data Cube (HDC) transformation: from Unity Catalog design managing 700T+ advertising signals to fully automated Databricks Asset Bundle deployments, powering AI agents and Genie analytics.We demonstrate the complete lifecycle: 1. Unity Catalogue schema supporting AI agent APIs 2. Genie conversational queries across 10-year trends 3. Data quality frameworks ensuring LLM agent accuracy, 4. Asset Bundles achieving infrastructure-as-code with multi-environment promotion— reducing releases from 3 weeks to 4 hours, 5. Genie optimization delivering sub-second performance on petabyte-scale aggregates. Technical deep-dives: medallion architecture in Unity Catalog, incremental processing patterns, DAB orchestration with Jenkins CI/CD, schema evolution maintaining compatibility across 50+ systems, and monitoring frameworks tracking freshness for agent reliability.