Building Metrics Store with Incremental Processing
OVERVIEW
EXPERIENCE | In Person |
---|---|
TYPE | Breakout |
TRACK | Data Warehousing - Analytics and BI |
INDUSTRY | Enterprise Technology |
TECHNOLOGIES | Delta Lake |
SKILL LEVEL | Intermediate |
DURATION | 40 min |
DOWNLOAD SESSION SLIDES |
The traditional BI solution, bundling consuming applications with analytical logic running on a data warehouse, faces scalability and maintenance challenges. Complex metric calculation across multiple places leads to redundant reprocessing, missing review and testing, and inconsistent metrics for business teams. This session explores Instacart’s in-house solution-business metrics store with incremental processing. It decouples analytic logic from BI reporting and visualization, offering unparalleled efficiency, reliability, and cost-effectiveness for robust business analytics. Attendees will gain insights from a case study demonstrating this architecture's application in supporting large-scale ads experiment analysis. Key takeaways:
- Design of a metrics store facilitating reuse of metrics across BI and operational applications
- Practical insights into incremental processing with DBX structured streaming
- Future enhancement with Databricks Unity Catalog and Lakehouse Monitoring.
SESSION SPEAKERS
Hang Li
/Senior Software Engineer II
Instacart