Continued Pre-Training (CPT) and Pre-Training (PT) Large Language Models

• Audience: Machine learning practitioners
• Hands-on labs: Yes
• Learning path: Advanced Generative AI Engineering with Databricks
• Description: This advanced course is designed to teach the intricacies of continued pre-training and pre-training of Large Language Models, a crucial technique of updating pre-trained models with new data without training from scratch. We will go over how to prepare data for continued pre-training and the effective strategies of implementing it, including model selection and hardware selection. We will also introduce necessary inference theory fundamentals that optimize deployment, such as quantization, flash attention, and LLM cascades. The course concludes with a discussion on pre-training and a survey of the landscape of aligning LLMs with human preferences.

By the end of the course, students will have a comprehensive understanding of continued pre-training, enabling them to update and improve LLMs effectively and efficiently.

This is the third and last course in the GenAI Engineer Professional pathway.

Pre-requisites:

• Completed GenAI Engineer Associate pathway or equivalent practical knowledge of:
• Understanding of deep learning, including how neural networks work, what loss functions are, etc.
• Basic understanding of how to build an LLM application that involves prompt engineering, retrieval-augmented generation (RAG), embeddings and foundation models