Enterprise AI at Scale: Serving Google Gemini on Databricks for Production Workloads
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Healthcare & Life Sciences, Manufacturing, Financial Services |
| Technologies | Databricks SQL, Unity Catalog, Lakebase |
| Skill Level | Advanced |
| DOWNLOAD SESSION SLIDES | |
Foundation models are moving beyond chatbots and copilots into the core of enterprise data pipelines — enriching records, classifying documents, resolving entities, and powering agentic workflows at massive scale. This session shows how Google Gemini and Databricks are partnering to make this real for production enterprises today.
Google Cloud shows how Gemini is available as a first-party model on Databricks, served natively via ai_query() and Model Serving — with built-in governance via AI Gateway, including cost controls, rate limits, usage tracking, and Unity Catalog logging.
HG Insights then brings this to life with a production use case: a multi-stage AI pipeline processing 7 billion records where rules and custom BERT models handle the predictable cases, and Gemini acts as an agentic enrichment layer — autonomously reasoning over incomplete company profiles and resolving missing attributes across hundreds of millions of inference calls. Orchestrated by Lakeflow and monitored via AI/BI dashboards, this pipeline directly generates the revenue-critical datasets HG Insights delivers to Fortune 500 customers.
Whether you're enriching CRM data, building document classification pipelines, or deploying agentic data quality workflows, this session gives you the architecture patterns and lessons learned to run Gemini on Databricks at enterprise scale.
Session Speakers
Abhishek Bhagwat
/Software Engineer, Cloud AI
Google Cloud
Bhadri Pani
/Senior Machine Learning Engineer
HG Insights
Shailesh Dargude
/AI Fellow
HG Insights
Full Summary
Running Gemini on Databricks: how HG Insights scales agentic pipelines across billions of documents
A joint Google Cloud and HG Insights session shows how to operationalize Gemini models inside the Databricks Platform, moving from single-SQL inference to managed agents that can query enterprise data on demand. The discussion centers on what it actually takes to run agentic AI in production, where cost, governance, and evaluation determine whether systems scale beyond prototypes. For broader context on how the analyst role is changing, Databricks has written about why AI is rewriting the analyst's job description, and teams building toward fully autonomous workflows can also explore what agentic analytics means in practice.
FAQ
Use Flash for high-volume agentic work where speed and cost matter, including row-by-row inference across large tables. Switch to Pro when you need maximum reasoning or more complex, multi-step coding patterns. In most production data workflows at scale, Flash is the default.
AI_QUERY lets you apply a prompt to every row of a Databricks table with a single SQL call. The platform manages batching, checkpointing, scaling, and rate limits, which removes most of the orchestration code teams would otherwise write to handle API quotas and failure recovery across millions of requests.
The agent registry stores an immutable agent configuration, including instructions, tools, and auth, created via a one-time control-plane call. Centralizing definitions improves governance, prevents configuration drift, and ensures updates propagate consistently rather than diverging across runtime instances.
Running frontier models over billions of documents is prohibitively expensive and discards hard-won domain knowledge. A layered approach uses deterministic filters and small fine-tuned Gemma models first, escalating only ambiguous cases to Gemini Flash, with human-in-the-loop for novel products.
The gateway supports usage tracking, inference tables for logging, rate limits for fair usage, and configurable guardrails. Those controls apply uniformly to model endpoints that downstream SQL or agent workloads consume.