Generative AI Application Evaluation and Governance

This course introduces learners to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.

Note:

1. This is the third course in the 'Generative AI Application Evaluation and Governance’ series.

2. Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

Skill Level

Associate

Duration

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

• Familiarity with the Databricks Data Intelligence Platform, specifically Unity Catalog for governance and model management

• Intermediate knowledge of Python programming for configuring evaluation metrics and utilizing API-based tools

• Conceptual understanding of Generative AI architectures, particularly Retrieval Augmented Generation (RAG) systems and their components (chunking, embeddings, vector stores)

• Basic knowledge of Large Language Model (LLM) behavior, including concepts such as tokens, temperature, and hallucinations

• Familiarity with machine learning lifecycle management and experiment tracking using MLflow

• Basic understanding of prompt engineering techniques for constructing queries and system instructions

• General awareness of data governance principles, including data lineage, access control, and data privacy

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Customer registration Partner registration

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Databricks Data Sharing & Collaboration

In this course, you will learn how to leverage Databricks' data sharing capabilities to enable secure, efficient collaboration across your organization and with external partners. The course covers the fundamentals of modern data sharing, exploring the business value it delivers and examining Databricks' three core solutions: Delta Sharing, Marketplace, and Clean Rooms. You'll understand the architecture of Delta Sharing, including both Databricks-to-Databricks (D2D) and open (D2O) sharing models, and learn how to share diverse asset types such as tables, AI models, and notebooks. The course further explores advanced replication strategies, cross-cloud and cross-region optimization, and implementation of granular access controls using dynamic views and partition filtering. You'll also master security and governance practices, including token management, fine-grained access control, cost optimization, and auditing strategies. By the end, you'll have the skills to design secure Delta Sharing solutions for internal and external use cases, optimize multi-party collaboration while managing costs, and choose the right Databricks sharing solution for your specific business needs.

Free

Associate

Model Development at Scale

In this course, you will develop an in-depth understanding of how to design, implement, and govern scalable machine learning systems that operate effectively at enterprise scale. The curriculum is organized into three experiential modules: developing distributed ML workflows with frameworks such as Apache SparkML and Ray, transitioning local ML development to distributed compute using tools like Pandas on Spark, and operationalizing and governing production models with Databricks’ MLOps ecosystem.

Through hands-on projects, you will construct end-to-end distributed ML pipelines using the SparkML workflow, applying Transformers, Estimators, and the fit/transform paradigm for both classification and regression tasks. You will version, compare, and manage experiments using MLflow 3.0 to ensure reproducibility and governance, capturing lineage between data, features, and model artifacts. Additionally, you will apply scalable Hyperparameter Optimization frameworks to improve model performance at scale.

The course concludes by demonstrating complete lifecycle management, from experimentation to production deployment, using Unity Catalog and Model Serving. You will learn to operationalize trained models, monitor their performance, and implement strong governance over models, features, and Delta assets within the Databricks environment.

Free

Professional