SESSION

Building Metrics Store with Incremental Processing

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Warehousing - Analytics and BI
INDUSTRYEnterprise Technology
TECHNOLOGIESDelta Lake
SKILL LEVELIntermediate
DURATION40 min

The traditional BI solution, bundling consuming applications with analytical logic running on a data warehouse, faces scalability and maintenance challenges. Complex metric calculation across multiple places leads to redundant reprocessing, missing review and testing, and inconsistent metrics for business teams. This session explores Instacart’s in-house solution-business metrics store with incremental processing. It decouples analytic logic from BI reporting and visualization, offering unparalleled efficiency, reliability, and cost-effectiveness for robust business analytics. Attendees will gain insights from a case study demonstrating this architecture's application in supporting large-scale ads experiment analysis. Key takeaways:

 

  • Design of a metrics store facilitating reuse of metrics across BI and operational applications
  • Practical insights into incremental processing with DBX structured streaming
  • Future enhancement with Databricks Unity Catalog and Lakehouse Monitoring.

SESSION SPEAKERS

Hang Li

/Senior Software Engineer II
Instacart