Generative AI Application Development

In this course, you will learn how to leverage Databricks' data sharing capabilities to enable secure, efficient collaboration across your organization and with external partners. The course covers the fundamentals of modern data sharing, exploring the business value it delivers and examining Databricks' three core solutions: Delta Sharing, Marketplace, and Clean Rooms. You'll understand the architecture of Delta Sharing, including both Databricks-to-Databricks (D2D) and open (D2O) sharing models, and learn how to share diverse asset types such as tables, AI models, and notebooks. The course further explores advanced replication strategies, cross-cloud and cross-region optimization, and implementation of granular access controls using dynamic views and partition filtering. You'll also master security and governance practices, including token management, fine-grained access control, cost optimization, and auditing strategies. By the end, you'll have the skills to design secure Delta Sharing solutions for internal and external use cases, optimize multi-party collaboration while managing costs, and choose the right Databricks sharing solution for your specific business needs.

In this course, you will develop an in-depth understanding of how to design, implement, and govern scalable machine learning systems that operate effectively at enterprise scale. The curriculum is organized into three experiential modules: developing distributed ML workflows with frameworks such as Apache SparkML and Ray, transitioning local ML development to distributed compute using tools like Pandas on Spark, and operationalizing and governing production models with Databricks’ MLOps ecosystem.

Through hands-on projects, you will construct end-to-end distributed ML pipelines using the SparkML workflow, applying Transformers, Estimators, and the fit/transform paradigm for both classification and regression tasks. You will version, compare, and manage experiments using MLflow 3.0 to ensure reproducibility and governance, capturing lineage between data, features, and model artifacts. Additionally, you will apply scalable Hyperparameter Optimization frameworks to improve model performance at scale.

The course concludes by demonstrating complete lifecycle management, from experimentation to production deployment, using Unity Catalog and Model Serving. You will learn to operationalize trained models, monitor their performance, and implement strong governance over models, features, and Delta assets within the Databricks environment.

In this course, you will be provided with a comprehensive understanding of the machine learning lifecycle and MLOps, emphasizing best practices for data and model management, testing, and scalable architectures. It covers key MLOps components, including CI/CD, pipeline management, and environment separation, while showcasing Databricks’ tools for automation and infrastructure management, such as Databricks Asset Bundles (DABs), Workflows, and Mosaic AI Model Serving. You will learn about monitoring, custom metrics, drift detection, model rollout strategies, A/B testing, and the principles of reliable MLOps systems, providing a holistic view of implementing and managing ML projects in Databricks.

Note: 

1. This course is the second in the series of Advanced Machine Learning. 

2. Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!