Inside Adobe’s Near Real-Time Cloud Spend Monitoring System

Cloud costs are rising faster than budgets, yet teams often discover expensive operations only after the spend has occurred. We moved from reactive cost reporting to proactive, near real-time monitoring—reducing SQL warehouse costs by 51% and Serverless compute by 14% in less than 30 days.This talk covers how we built a production-grade monitoring system managing millions in annual Databricks spend across petabyte-scale data, 94,000+ jobs, and millions of compute hours, with 15-minute monitoring cycles and zero infrastructure cost.The key challenge was attribution: Databricks assigns SQL warehouse costs to warehouse creators, not query executors, obscuring who drives spend. We solved this by combining Databricks system tables—`billing.usage`, `query.history`, and `compute.warehouse_events`—to enable query-level cost attribution without added compute.Attendees will learn the architecture, implementation patterns and reusable strategies to build proactive cost monitoring at any scale.