Data Analysis with Databricks

In this course, participants will build upon their existing knowledge of Apache Spark, Delta Lake, and Delta Live Tables to unlock the full potential of the data lakehouse by utilizing the suite of tools provided by Databricks. This course places a heavy emphasis on designs favoring incremental data processing, enabling systems optimized to continuously ingest and analyze ever-growing data. By designing workloads that leverage built-in platform optimizations, data engineers can reduce the burden of code maintenance and on-call emergencies, and quickly adapt production code to new demands with minimal refactoring or downtime. The topics in this course should be mastered prior to attempting the Databricks Certified Data Engineering Professional exam.

This course offers a practical introduction to the Mosaic AI platform, focusing on its key components and features for building and deploying generative AI systems. Participants will learn how Databricks facilitates the development of scalable generative AI solutions and explore Mosaic AI tools such as Vector Search, the Agent Framework, and MLflow’s generative AI capabilities for model tracking and logging. This course includes hands-on experience in constructing and evaluating Retrieval-Augmented Generation (RAG) pipelines, deploying generative AI agents, and leveraging evaluation frameworks to optimize performance. By the end of the course, learners will be equipped with the skills to design, deploy, and monitor common generative AI applications using Mosaic AI.

This comprehensive course provides a practical guide to developing traditional machine learning models on Databricks, emphasizing hands-on demonstrations and workflows using popular ML libraries. This course focuses on executing common tasks efficiently with AutoML and MLflow. Participants will delve into key topics, including regression and classification models, harnessing Databricks' capabilities to track model training, leveraging feature stores for model development, and implementing hyperparameter tuning. Additionally, the course covers AutoML for rapid and low-code model training, ensuring that participants gain practical, real-world skills for streamlined and effective machine learning model development in the Databricks environment.

Note: This is the second course in the 'Machine Learning with Databricks’ series.