Generative AI Application Evaluation and Governance
This is your introduction to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.
Languages Available: English | 日本語 | Português BR | 한국어
- Familiarity with natural language processing concepts
- Familiarity with prompt engineering/prompt engineering best practices
- Familiarity with the Databricks Data Intelligence Platform
- Familiarity with RAG (preparing data, building a RAG architecture, concepts like embedding, vectors, vector databases, etc.)
- Experience with building LLM applications using multi-stage reasoning LLM chains and agents
Outline
Importance of Evaluating GenAI Applications
- Why to Evaluate GenAI Applications
- Exploring Licensing of Datasets
- Prompts and Guardrails Basics
- Implementing and Testing Guardrails for LLMs
Securing and Governing GenAI Applications
- AI System Security
- Implementing AI Guardrails
GenAI Evaluation Techniques
- Evaluation Techniques
- Benchmark Evaluation
- LLM-as-a-Judge
- Domain-Specific Evaluation
End-to-end Application Evaluation
- AI System Architecture
- Custom Metrics
- Offline vs. Online Evaluation
Upcoming Public Classes
Date | Time | Language | Price |
|---|---|---|---|
Oct 17 | 01 PM - 05 PM (America/New_York) | English | $750.00 |
Oct 23 | 11 AM - 03 PM (Asia/Singapore) | English | $750.00 |
Oct 24 | 01 PM - 05 PM (Europe/London) | English | $750.00 |
Oct 31 | 01 PM - 05 PM (America/Los_Angeles) | English | $750.00 |
Nov 07 | 10 AM - 02 PM (Asia/Singapore) | English | $750.00 |
Nov 12 | 01 PM - 05 PM (America/New_York) | English | $750.00 |
Nov 18 | 01 PM - 05 PM (Europe/London) | English | $750.00 |
Nov 20 | 09 AM - 01 PM (Asia/Colombo) | English | $750.00 |
Nov 20 | 08 AM - 12 PM (Europe/London) | English | $750.00 |
Nov 22 | 08 AM - 12 PM (America/Los_Angeles) | English | $750.00 |
Dec 02 | 08 AM - 12 PM (America/New_York) | English | $750.00 |
Dec 04 | 09 AM - 01 PM (Europe/London) | English | $750.00 |
Dec 06 | 01 PM - 05 PM (Asia/Singapore) | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
Upcoming Public Classes
Career Workshop/
March 20
