Skip to main content

DevOps Essentials for Data Engineering - Mandarin Chinese

本课程探讨软件工程最佳实践与 DevOps 原则,专为使用 Databricks 的数据工程师量身设计。学员将在代码质量、版本控制、文档编写和测试等关键主题上打下坚实基础。课程重点介绍 DevOps,涵盖其核心组件、优势,以及持续集成与持续交付(CI/CD)在优化数据工程工作流中所发挥的作用。


您将学习如何在 PySpark 中应用模块化原则,以创建可复用组件并高效组织代码结构。实践环节包括:使用 pytest 框架为 PySpark 函数设计并实现单元测试,以及使用 Spark Declarative Pipeline 和 Jobs 对 Databricks 数据管道进行集成测试,以确保其可靠性。


课程还涵盖 Databricks 中的基本 Git 运营操作,包括使用 Databricks Git Folders 集成持续集成实践。最后,您将从宏观层面了解 Databricks 资产的多种部署方式,例如 REST API、CLI、SDK 以及 Declarative Automation Bundles(DABs),从而掌握部署和管理管道的相关技术知识。


完成本课程后,您将熟练掌握软件工程与 DevOps 最佳实践,从而能够构建可扩展、易维护且高效的数据工程解决方案。


注意:

1. 这是“Data Engineering with Databricks”系列课程中的第四门课程。

2. 对于 SCORM 课程文件,请确保完成内容后关闭 SCORM 窗口。请勿点击“Next Lesson”按钮,否则可能导致 SCORM 模块无法标记为已完成。

Skill Level
Associate
Duration
3h
Prerequisites

本课程的内容是为具备以下技能、知识和能力的学员开发的:

• 熟练掌握 Databricks 平台,包括具备使用 Databricks Workspaces、Apache Spark、Delta Lake 与奖章架构、Unity Catalog、Delta Live Tables 以及 Workflows 的实践经验。同时需要具备 Git 版本控制的基础知识。

• 具备数据摄取与转换经验,熟练使用 PySpark 进行数据处理和 DataFrame 操作。此外,候选人应具备编写中级 SQL 查询以进行数据分析和转换的经验。

• 掌握 Python 编程知识,能够熟练编写中级 Python 代码,包括设计和实现函数与类的能力。用户还应具备创建、导入和有效使用 Python 包的技能。

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Model Development at Scale

In this course, you will learn to design and implement machine learning workflows that scale from a single machine to distributed compute on Databricks. The curriculum is organized into three modules: foundations of model development (when and why to distribute, and how Spark executes work), scalable model development with Spark ML and MLflow, and distributed model training with Ray and PyTorch — closing with a framework for choosing the right tool for a given workload.

Through hands-on demos and labs, you will build distributed Spark ML pipelines using Transformers, Estimators, and the fit/transform paradigm; scale existing pandas code with the Pandas API on Spark; and track, compare, and govern experiments with MLflow 3 and Unity Catalog. You will then train models across a cluster with Ray Train and PyTorch (TorchDistributor and DeepSpeed), and learn to match each workload to the right framework based on data size, model size, hardware, and cost.

The course concludes with a capstone that takes a problem end to end — engineering features, training a model at scale, tracking it with MLflow, and registering it in Unity Catalog with a governance alias. (Production deployment, serving, and monitoring are covered in the separate Advanced MLOps course.)

By the end, you will be able to choose, build, and justify distributed, reproducible, and governed model-development workflows on Databricks.

Note: For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

Paid & Subscription
4h
Lab
Professional

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.