Domain Intelligence Wins: What “High-Quality” Actually Means in Production AI

As enterprises move from experimenting with generative AI to deploying agentic systems in production, the conversation is shifting. The question executives are asking is no longer “Can this model reason?” but “Can this system be trusted?”

To explore what that shift really means, I sat down with Maria Zervou, Chief AI Officer for EMEA at Databricks. Maria works closely with customers across regulated and fast-moving industries and spends her time at the intersection of AI architecture, governance, and real-world execution.

Throughout the conversation, Maria kept returning to the same point: success with agentic AI isn’t about the model. It’s about the systems around it—data, engineering discipline, and clear accountability.

Catherine Brown: Many executives I speak with still equate AI quality with how impressive the model seems. You’ve argued that’s the wrong frame. Why?

Maria Zervou: The biggest misunderstanding I see is people confusing a model’s cleverness or perceived reasoning ability with quality. Those are not the same thing.

Quality, especially in agentic systems, is about compounding reliability. You’re no longer evaluating a single response. You’re evaluating a system that might take hundreds of steps—retrieving data, calling tools, making decisions, escalating issues. Even small errors can compound in unpredictable ways.

So the questions change. Did the agent use the right data? Did it find the right resources? Did it know when to stop or escalate? That’s where quality really lives.

And importantly, quality means different things to different stakeholders. Technical teams often focus on KPIs like cost, latency, or throughput. End users care about brand compliance, tone, and legal constraints. So, if those perspectives aren’t aligned, you end up optimizing the wrong thing.

Catherine: That’s interesting, especially because many leaders assume AI systems have to be “perfect” to be usable, particularly in regulated environments. How should companies in highly-regulated industries approach AI initiatives?

Maria: In highly regulated sectors, you do need very high accuracy, but the first benchmark should be human performance. Humans make mistakes today, all the time. If you don’t anchor expectations in reality, you’ll never move forward.

What matters more is traceability and accountability. When something goes wrong, can you trace why a decision was made? Who owns the outcome? What data was used? If you can’t answer those questions, the system isn’t production-ready, regardless of how impressive the output looks.

Catherine: You talk a lot about domain-specific agents versus general-purpose models. How should executives think about that distinction?

Maria: A general-purpose model is essentially a very capable reasoning engine trained on very large and diverse datasets. But it doesn’t understand your business. A domain-specific agent uses the same base models, but it becomes more powerful through context. You force it into a predefined use case. You limit the space it can search. You teach it what your KPIs mean, what your terminology means, and what actions it’s allowed to take.

That constraint is actually what makes it better. By narrowing the domain, you reduce hallucinations and increase the reliability of outputs. Most of the value doesn’t come from the model itself. It comes from the proprietary data it can securely access, the semantic layer that defines meaning, and the tools it’s allowed to use. Essentially, it can reason on your data. That’s where competitive advantage lives.

Catherine: Where do you typically see AI agent workflows break when organizations try to move from prototype to production?

Maria: There are three main failure points. The first is pace mismatch. The technology moves faster than most organizations. Teams jump into building agents before they’ve done the foundational work on data access, security, and structure.

The second is tacit knowledge. A lot of what makes employees effective lives in people’s heads or scattered documents. If that knowledge isn’t codified in a form an agent can use, the system will never behave the way the business expects.

The third is infrastructure. Many teams don’t plan for scale or real-world usage. They build something that works once, in a demo, but collapses under production load.

All three issues tend to show up together.

Catherine: You’ve said before that capturing business knowledge is as important as choosing the right model. How do you see organizations doing that well?

Maria: It starts with recognizing that AI systems are not one-off projects. They’re living systems. One practical approach is to record and transcribe meetings and treat that as raw material. You then structure, summarize, and tag that information so the system can retrieve it later. Over time, you’re building a knowledge base that reflects how the business actually thinks.

Equally important is how you design evaluations. Early versions of an agent should be used by business stakeholders, not just engineers. Their feedback—what feels right, what doesn’t, why something is wrong—becomes training data.

Building an effective evaluation system, custom to that agent’s specific purpose, is critical to ensuring high-quality outputs, which is ultimately critical for any AI projects in production. Our own usage data shows that customers who use AI evaluation tools get nearly 6x more AI projects into production than those who don’t.

In effect, you’re codifying the business brain into evaluation criteria.

Catherine: That sounds expensive and time-consuming. How do you balance rigor with speed?

Maria: This is where I talk about minimum viable governance. You don’t solve governance for the entire enterprise on day one. You solve it for the specific domain and use case you’re working on. You make sure the data is controlled, traceable, and auditable for that agent. Then, as the system proves valuable, you expand.

What helps is having repeatable building blocks—patterns that already encode good engineering and governance practices. That’s the thinking behind approaches like Agent Bricks, where teams can start from refined foundations instead of reinventing workflows, evaluations, and controls from scratch each time.

Executives should still insist on a few non-negotiables up front: clear business KPIs, a named executive sponsor, evaluations built with business users, and strong software engineering fundamentals. The first project will be painful—but it sets the pattern for everything that follows and makes subsequent agents much faster to deploy.

If you skip that step, you end up with what I call “demo wear”: impressive prototypes that never quite become real.

Catherine: Can you share examples where agents have materially changed how work gets done?

Maria: Internally at Databricks, we’ve seen this in a few places. In Professional Services, agents are used to scan customer environments during migrations. Instead of engineers manually reviewing every schema and system, the agent generates recommended workflows based on best practices. That dramatically reduces time spent on repetitive analysis.

In Field Engineering, agents automatically generate demo environments tailored to a customer’s industry and use case. What used to take hours of manual prep now happens much faster, with higher consistency.

In both cases, the agent didn’t replace expertise—it amplified it.

Catherine: If you had to distill this for a CIO or CDO just starting down this path, what should they focus on first?

Maria: Start with the data. Trusted agents require a unified, controllable, and auditable data foundation. If your data is fragmented or inaccessible, the agent will fail—no matter how good the model is. Second, be clear about ownership. Who owns quality? Who owns outcomes? Who decides when the agent is “good enough”? And finally, remember that agentic AI is not about showing how smart the system is. It’s about whether the system reliably helps the business make better decisions, faster, without introducing new risk.

Closing Thoughts

Agentic AI represents a real shift—from tools that assist humans to systems that act on their behalf. But as Maria makes clear, success depends far less on model sophistication than on discipline: in data, in governance, and in engineering.

For executives, the challenge is not whether agents are coming. It’s whether their organizations are ready to build systems that can be trusted once they arrive.

To learn more about building an effective operating model, download the Databricks AI Maturity Model.