Databricks FinOps Genie — Cost Observability Meets Optimization Insights

A global bank running more than 1,000 Databricks workspaces reframed its approach, moving from reactive reporting to proactive, self-service optimization on the Databricks Platform. The program combined system tables, AI/BI Dashboards, Genie, and an Agent Bricks knowledge agent to create a policy-aware "FinOps Genie" that teams actually used. The Agent Bricks capability was central to grounding recommendations in the bank's own policies, turning generic best practices into actionable, guardrail-aware guidance.

The result was a measurable reduction in wasteful compute across a multimillion-dollar estate. More importantly, ownership shifted to delivery teams, who acted on prioritized recommendations instead of waiting for central enforcement.

Why visibility was not enough

The bank already had dense Power BI dashboards covering spend, jobs, all-purpose compute, and query behavior. Waste was visible and large, with more than $200K per month identified, yet actions stalled. No one clearly owned next steps, best practices sat unread on SharePoint, and guidance rarely translated to a specific, prioritized to-do.

A federated operating model amplified the problem. The central FinOps function had limited workspace-level control and needed to influence hundreds of teams. Publishing more telemetry was not moving the needle, so the design pivoted from completeness to attention and accountability.

Design choices that drove behavior change

Three moves reframed the experience from data to decisions:

• Layered information design: an executive summary on top, a prioritized to-do list in the middle, and drill-downs for teams that wanted to validate or challenge recommendations.
• Visible comparison and accountability: a leaderboard ranked lines of business by efficiency, and the lowest performers entered a mandatory FinOps council review.
• Sequenced recommendations by risk: start with waste reduction, then right-size infrastructure, then tackle data and code optimization.
   

Ordering mattered. About 72 percent of the savings came from shutting down legacy pipelines and unused assets, not from rewriting Spark code. Waste reduction delivered fast, low-risk wins and avoided breaking production.

Inside the "FinOps Genie": structured telemetry meets policy

FinOps Genie was not a product, but a composition of existing capabilities:

• AI/BI Dashboards, powered by system tables, delivered layered views and a leaderboard. System tables provided a common telemetry backbone across billing, compute, queries, lineage, and more.
• A Genie conversational interface let users ask natural-language questions about spend, efficiency, spot instance usage, and team-level recommendations.
• An Agent Bricks knowledge agent grounded advice in the bank's policies stored on SharePoint, including auto-termination rules, required DBR versions, serverless and private link usage, and warehouse standards.
   

In practice, developers asking how to reduce spend saw recommendations tuned to internal guardrails, complete with severity and deadlines. Genie surfaced concrete actions like decommissioning unused tables, shutting down idle clusters and warehouses, enforcing spot instances in non-prod, or migrating from all-purpose clusters to job clusters, while still allowing drill-down for teams that needed detail.

Outcomes and what is next

The redesign yielded roughly 20 percent less wasteful compute. Teams began to self-correct to improve leaderboard standing, and the FinOps review shifted from a reporting cadence to a decision forum. Guidance became self-service and policy-aware, increasing trust and speed to action.

Planned extensions include budget forecasting, anomaly and spike detection, chargeback support, and contracted-price modeling in system tables. The longer-term direction is agentic workflows, where recommendations flow into automated optimizations with a human in the loop, integrated with the Governance Hub.

Reframe FinOps as behavior design, not dashboarding

FinOps success hinges on attention, motivation, and clear ownership. Layered recommendations, peer comparison, and policy-grounded guidance turn observability into action. Teams can replicate the pattern using system tables, AI/BI Dashboards, Genie, and an Agent Bricks knowledge agent without waiting for a new product.