Sponsored by: Dataiku | Dataiku + Databricks: Your Stack for Production-Ready AI Agents

Pairing Dataiku's governed, visual layer with the Databricks Platform creates a consistent path from expert knowledge to deployed, monitored agents, drawing on capabilities across the broader Databricks Platform for AI agents.

A fictional travel brand is used to ground the narrative, but the patterns are familiar: a small data team, a growing backlog of AI requests, and domain experts who know the business but do not code. The solution showcased centers on giving those experts a shared, auditable design surface that operates natively on Databricks infrastructure, an approach consistent with how the Databricks Platform unifies data, analytics, and AI under a single governance model.

Why enterprise agent efforts hit a wall

Agents often underperform because they cannot reach operational systems or enterprise data that sit behind strict security boundaries. Many agent builders do not run natively inside a data platform, so custom integration becomes brittle or slow.

Critical institutional knowledge is also hard to encode as rules. Judgment built over years does not fit into if-then logic, so agents miss nuance. Finally, evaluation is frequently informal. Without repeatable tests tied to business-defined traits, teams cannot promote agents confidently or detect regressions over time.

A governed, visual path that runs inside Databricks

Dataiku acts as an orchestration and abstraction layer on top of Databricks. Data remains where it lives. Processing is pushed down to Delta Lake, Databricks compute clusters, and Databricks SQL. Integrations include Databricks Connect for PySpark, MLflow for models, Model Serving endpoints, and Unity AI Gateway for LLM and agent serving. Existing Databricks permissions flow through, so access stays consistent.

A new capability, Dataiku co-build, lets users chat through building pipelines, generating charts, authoring recipes, and documenting steps, while producing a visual, auditable flow. Guardrails can require confirmation before changes. The point is not just convenience. With data workflows, a wrong answer can look right until you inspect intermediate logic, so visibility and documentation matter.

From pipeline to agent: tools, structure, and reach

The same visual philosophy extends to agent assembly. Agents can connect to structured tables, unstructured PDFs in Databricks Volumes, Genie for natural language SQL, Databricks vector search, and machine learning models, all configured in a UI. Dataiku's extensible tools include partner integrations, dataset lookups, and model queries, plus the ability to bring custom MCP tools.

For business-critical behavior, a structured visual agent adds deterministic routing and escalation. Teams can define loops, branching, and human-in-the-loop approvals without writing code, striking a middle ground between prompt-only abstractions and full-code frameworks. Existing external agents, including those hosted in Databricks, can be called as first-class components inside Dataiku projects.

Measurement, promotion, and ongoing operations

Agent reviews provide a consistent interface to evaluate any agent, whether built in Dataiku or running elsewhere. Teams define the traits that matter, such as correctness or tone, then run repeatable tests with human review or LLM-as-judge scoring. Because scheduling and automation are built in, evaluations can run continuously, compare versions, alert on drift, and gate promotion to production.

Once validated, agents can be exposed via an API, surfaced in an Agent Hub landing zone, or embedded directly into Slack or Teams so users work in familiar tools without worrying about where the agent runs.

Shipping useful agents requires more than model choice

The session's core lesson is pragmatic: production-ready agents emerge when three fundamentals work together, broad access to enterprise data and unstructured content, a shared design surface that both coders and non-coders can use, and evaluation baked into the lifecycle. Using Dataiku for the visual and governance layer on top of the Databricks Platform gives domain experts a credible route from idea to deployed, monitored agents.