Agent Evaluation on Databricks
This course teaches students how to systematically evaluate AI agents using MLflow's evaluation framework, addressing the unique challenges of non-deterministic AI systems that traditional software testing cannot handle. Students learn to implement various evaluation approaches including built-in judges for common criteria like correctness and safety, guideline judges for business-specific requirements, and custom judges for specialized needs. The course covers both offline evaluation using curated datasets and online production monitoring, with hands-on experience using MLflow's tracing capabilities to understand agent execution patterns and collect human feedback from different stakeholder types. Through practical demonstrations and labs, students develop skills in creating evaluation workflows that drive continuous quality improvements throughout the AI agent development lifecycle.
Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.
Languages Available: English | 日本語 | Português BR | 한국어
• Intermediate Python programming experience
• Basic SQL knowledge for querying and creating functions
• Familiarity with Databricks Data Intelligence Platform
• Understanding of Unity Catalog concepts including catalogs and schemas
• Basic understanding of large language models (LLMs) and prompt engineering
• Basic knowledge of MLflow
Outline
AI Agent Evaluation Fundamentals
• The Challenge of Evaluating AI Agents
• Agent Setup
• MLflow's Evaluation Framework
Built-In and Guideline Judges
• Types of Evaluation Judges
• Using MLflow Built-In Judges
• Guideline Judges with MLflow
• Applying Agent Evaluation
• Custom Judges with MLflow
Custom Judges and Human Feedback
• Offline vs. Online Evaluation Strategies
• Best Practices and Practical Application
• Developer and SME Feedback with MLflow
Upcoming Public Classes
Date | Time | Your Local Time | Language | Price |
|---|---|---|---|---|
May 08 | 11 AM - 03 PM (Asia/Singapore) | - | English | $750.00 |
Jun 03 | 01 PM - 05 PM (Europe/London) | - | English | $750.00 |
Jun 05 | 08 AM - 12 PM (Asia/Kolkata) | - | English | $750.00 |
Jul 08 | 09 AM - 01 PM (Australia/Sydney) | - | English | $750.00 |
Jul 08 | 09 AM - 01 PM (America/New_York) | - | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
