Agent Evaluation on Databricks

This course teaches students how to systematically evaluate AI agents using MLflow's evaluation framework, addressing the unique challenges of non-deterministic AI systems that traditional software testing cannot handle. Students learn to implement various evaluation approaches including built-in judges for common criteria like correctness and safety, guideline judges for business-specific requirements, and custom judges for specialized needs. The course covers both offline evaluation using curated datasets and online production monitoring, with hands-on experience using MLflow's tracing capabilities to understand agent execution patterns and collect human feedback from different stakeholder types. Through practical demonstrations and labs, students develop skills in creating evaluation workflows that drive continuous quality improvements throughout the AI agent development lifecycle.

Skill Level

Associate

Duration

Prerequisites

In this course, the content was developed for participants with these skills/knowledge/abilities:

• Intermediate Python programming experience

• Basic SQL knowledge for querying and creating functions

• Familiarity with Databricks Data Intelligence Platform

• Understanding of Unity Catalog concepts including catalogs and schemas

• Basic understanding of large language models (LLMs) and prompt engineering

• Basic knowledge of MLflow

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Customer registration Partner registration

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Warehousing Practitioner

Building Semantic Models in Databricks with UC Metric Views

In this course, you’ll learn how to empower your data consumers to get validated business insights with less burden on you and the rest of your enterprise's data and AI team. Through building a semantic layer on top of your gold layer, you'll offer decision makers and business analysts the opportunity to explore key business performance indicators in novel ways, while still maintaining efficient data operations and enterprise-vetted metrics.

Free

Lab

Associate

Data Warehousing Practitioner

Building Semantic Models in Databricks with UC Metric Views

Languages Available: English | 日本語 | 한국어

Free

Associate

Databricks Data Sharing & Collaboration

In this course, you will learn how to leverage Databricks' data sharing capabilities to enable secure, efficient collaboration across your organization and with external partners. The course covers the fundamentals of modern data sharing, exploring the business value it delivers and examining Databricks' three core solutions: Delta Sharing, Marketplace, and Clean Rooms. You'll understand the architecture of Delta Sharing, including both Databricks-to-Databricks (D2D) and open (D2O) sharing models, and learn how to share diverse asset types such as tables, AI models, and notebooks. The course further explores advanced replication strategies, cross-cloud and cross-region optimization, and implementation of granular access controls using dynamic views and partition filtering. You'll also master security and governance practices, including token management, fine-grained access control, cost optimization, and auditing strategies. By the end, you'll have the skills to design secure Delta Sharing solutions for internal and external use cases, optimize multi-party collaboration while managing costs, and choose the right Databricks sharing solution for your specific business needs.

Free

Associate