Session

Forecasting at Databricks: One Framework Behind Consumption and Infra Cost

Overview

ExperienceIn Person
TrackAnalytics & BI
IndustryEnterprise Technology
TechnologiesAI/BI, Genie
Skill LevelIntermediate
DOWNLOAD SESSION SLIDES

Consumption forecasting and infrastructure cost forecasting at Databricks share three challenges: trust and stakeholder adoption, model accuracy under hierarchy and outliers, and data signal quality. We addressed them on a unified Databricks-native stack. For trust and adoption: a self-serve human-in-the-loop Databricks App lets infra cost owners review recommended forecasts and inject inorganic shifts the model won’t see, while AI/BI dashboards and Genie Agents let users self-serve insights, and decomposition keeps consumption forecasts explainable. For model accuracy: Unified Forecasting Framework, the backend behind AI_Forecast() function scheduled on Lakeflow Jobs validates model selection across the hierarchies. For data and signal quality: Unity Catalog, Metric Views, and Lakeflow form the governed data backbone for cost forecasting and the feature-selection substrate for consumption. Attendees leave with Databricks-native patterns for forecasts domain owners self-serve and leadership defends.

 

Session Speakers

Alex Wang

/Sr. Data Scientist
Databricks

Speaker placeholderIMAGE COMING SOON

Ginger Holt

/Sr Staff Data Scientist
Databricks

Full Summary

Forecasting at Databricks: Building Trusted Consumption and Infrastructure Cost Forecasts

Finance and FP&A teams need accurate forecasts to manage costs, plan investments, and make informed business decisions. But forecast accuracy alone isn't enough. Without clear explanations, consistent definitions, and stakeholder ownership, even accurate forecasts can struggle to earn trust and adoption. In this session, Databricks shares how it builds and operationalizes consumption and infrastructure cost forecasts across hundreds of teams. Using a unified data and AI platform, governed forecasting pipelines, and Genie-powered analytics, Databricks makes forecasts explainable, defensible, and diagnosable — helping teams understand cost drivers, investigate variances, and make more confident planning decisions.

FAQ


Finance teams can improve forecast reliability by combining predictive models with transparent assumptions, stakeholder inputs, and historical validation. Databricks supports governed forecasting workflows that help teams explain changes, evaluate model performance, and build confidence in planning decisions.

Databricks uses a unified forecasting framework that separates infrastructure costs into usage and unit cost, with clear ownership across teams. Governed data pipelines and forecasting models help organizations understand cost drivers, evaluate assumptions, and improve planning visibility.

Finance teams can analyze forecast variances by breaking predictions into their underlying drivers, assumptions, and exceptions. Databricks supports this approach through explainable forecasting models and governed analytics, making it easier to investigate why actual results differ from expectations.

Genie enables users to ask natural-language questions of governed data and explore forecast performance without building a new report for every question. At Databricks, Genie and metric views help teams investigate infrastructure cost anomalies, understand forecast changes, and identify potential drivers.

Start by identifying priority forecasting challenges, such as cost visibility, variance analysis, or financial planning accuracy. Databricks can help Finance and technology teams establish a governed data foundation and apply AI-powered analytics to improve forecasting processes.

Talk to a Databricks expert →