Skip to main content

Agent Evaluation on Databricks

This course teaches students how to systematically evaluate AI agents using MLflow's evaluation framework, addressing the unique challenges of non-deterministic AI systems that traditional software testing cannot handle. Students evaluate a shared tool-calling agent that is registered in Unity Catalog and calls its models through Unity Gateway, using built-in judges for common criteria like correctness and safety, guideline judges for business-specific requirements, and custom judges for specialized needs. The course covers both offline evaluation using curated datasets and online production monitoring, including storing MLflow traces in Unity Catalog. Students use MLflow tracing to understand agent execution patterns and collect human feedback from developers and from subject matter experts, who review the agent through its Databricks App. Through practical demonstrations and labs, students develop skills in creating evaluation workflows that drive continuous quality improvements throughout the AI agent development lifecycle.



Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Associate
Duration
4h
Prerequisites
In this course, the content was developed for participants with these skills/knowledge/abilities:  

• Intermediate Python programming experience

• Basic SQL knowledge for querying and creating functions

• Familiarity with Databricks Data Intelligence Platform

• Understanding of Unity Catalog concepts including catalogs and schemas

• Basic understanding of large language models (LLMs) and prompt engineering

• Basic knowledge of MLflow



Outline

1. AI Agent Evaluation Fundamentals

• The Challenge of Evaluating AI Agents

• Demo: Agent Setup

• MLflow's Evaluation Framework


2. Built-In and Guideline Judges

• Built-In Judges

• Demo: Using MLflow Built-In Judges

• Guideline Judges

• Demo: Guideline Judges with MLflow

• Custom Judges and Feedback

• Demo: Custom Judges with MLflow

• Lab: Applying Agent Evaluation


3. Custom Judges and Human Feedback

• Offline vs. Online Evaluation Strategies

• Lab: Developer and SME Feedback with MLflow


Upcoming Public Classes

Date
Time
Your Local Time
Language
Price
Dec 10
09 AM - 01 PM (Asia/Kolkata)
-
English
$750.00
Dec 10
01 PM - 05 PM (Europe/Paris)
-
English
$750.00
Dec 10
09 AM - 01 PM (America/New_York)
-
English
$750.00
Jan 14
09 AM - 01 PM (Asia/Singapore)
-
English
$750.00
Jan 14
09 AM - 01 PM (Europe/Paris)
-
English
$750.00
Jan 14
01 PM - 05 PM (America/New_York)
-
English
$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.