Skip to main content

DevOps Essentials for Data Engineering

This course explores software engineering best practices and DevOps principles, specifically designed for data engineers working with Databricks. Participants will build a strong foundation in key topics such as code quality, version control, documentation, and testing. The course emphasizes DevOps, covering core components, benefits, and the role of continuous integration and delivery (CI/CD) in optimizing data engineering workflows.


You will learn how to apply modularity principles in PySpark to create reusable components and structure code efficiently. Hands-on experience includes designing and implementing unit tests for PySpark functions using the pytest framework, followed by integration testing for Databricks data pipelines with Spark Declarative Pipeline and Jobs to ensure reliability.


The course also covers essential Git operations within Databricks, including using Databricks Git Folders to integrate continuous integration practices. Finally, you will take a high level look at various deployment methods for Databricks assets, such as REST API, CLI, SDK, and Declarative Automation Bundles (DABs), providing you with the knowledge of techniques to deploy and manage your pipelines.


By the end of the course, you will be proficient in software engineering and DevOps best practices, enabling you to build scalable, maintainable, and efficient data engineering solutions.


Note: 

1. This is the fourth course in the 'Data Engineering with Databricks' series.

2. For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Associate
Duration
3h
Prerequisites

• Proficient knowledge of the Databricks platform, including experience with Databricks Workspaces, Apache Spark, Delta Lake and the Medallion Architecture, Unity Catalog, Delta Live Tables, and Workflows. A basic understanding of Git version control is also required.

• Experience ingesting and transforming data, with proficiency in PySpark for data processing and DataFrame manipulations. Additionally, candidates should have experience writing intermediate level SQL queries for data analysis and transformation.

• Knowledge of Python programming, with proficiency in writing intermediate level Python code, including the ability to design and implement functions and classes. Users should also be skilled in creating, importing, and effectively utilizing Python packages.

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Engineer

Databricks Data Privacy - Mandarin Chinese

Databricks 数据隐私课程面向数据工程师,讲解如何在 Databricks Data Intelligence Platform 上发现、保护、删除个人数据并对其负责。课程将隐私视为建立在 Unity Catalog 和 Delta Lake 之上的一组技术决策与控制。课程把隐私与安全、治理和合规区分开来,然后围绕识别、保护、管理、验证的生命周期,以及学习者可应用于每项技术的简短决策模型来组织工作。

学习者首先使用 Data Classification 查找敏感列,并用治理标签记录每列应如何处理。他们用基于标签的属性访问控制(ABAC)策略保护这些列,并在中央策略不适用时使用手动列掩码和行过滤器。去标识化进一步涵盖加盐哈希、通过单独治理的保险库进行令牌化,以及列加密,盐值和密钥保存在 Databricks Secrets 中。对于删除权,学习者跟踪请求,借助分类和血缘查找主体数据,用 DELETE 和 MERGE 删除并传播变更,再用 REORG TABLE ... APPLY (PURGE) 和考虑保留期的 VACUUM 清除底层文件。课程最后将血缘、审计日志、查询历史和 Delta 表历史组合成证据链,证明各项控制是否生效。

所有数据均为合成数据。演示沿着一家跨国零售商的数据资产走完生命周期,并以已掩码的查询结果、空的删除检查或表历史记录等证据结束每项控制。两个基于场景的实验强化该工作流。第一个实验中,学习者修复子公司客户表上损坏的保护。第二个实验中,他们完成一份只做了一半的删除请求,并汇总完成证据。本课程为技术课程,不提供法律建议。

注意: Databricks Academy 正在将 Databricks 环境中的课堂教学过渡为基于 Notebook 的形式,不再使用幻灯片进行讲座。您可以在 Vocareum 实验室环境中访问讲座 Notebook。

Paid & Subscription
3h
Lab
Professional
Data Engineer

Databricks Data Privacy

Databricks Data Privacy teaches data engineers how to find, protect, delete and account for personal data on the Databricks Data Intelligence Platform. It treats privacy as a set of technical decisions and controls built on Unity Catalog and Delta Lake. The course separates privacy from security, governance and compliance, then organizes the work around an identify, protect, manage and verify lifecycle and a short decision model that learners apply to every technique.

Learners start by finding sensitive columns with Data Classification and recording how each should be treated with a governed tag. They protect those columns with tag-driven attribute-based access control (ABAC) policies and use manual column masks and row filters where a central policy doesn't fit. De-identification goes further with salted hashing, tokenization through a separately governed vault and column encryption, with the salt and key held in Databricks Secrets. For the right to erasure, learners track a request, find the subject's data with classification and lineage, delete and propagate the change with DELETE and MERGE, then purge the underlying files with REORG TABLE ... APPLY (PURGE) and retention-aware VACUUM. The course closes by combining lineage, audit logs, query history and Delta table history into an evidence trail that shows whether each control worked.

All data is synthetic. The demos follow one multinational retailer's data estate through the lifecycle and end each control with evidence that it worked, such as a masked query result, an empty erasure check or a table history entry. Two scenario-based labs reinforce the workflow. In the first, learners repair broken protection on a subsidiary's customer table. In the second, they finish a half-completed erasure request and assemble the evidence that it is complete. The course is technical and does not give legal advice.

Note:

1. For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

2. This course is the second in the 'Advanced Data Engineering with Databricks' series.

Languages Available: Languages Available: English | 日本語 | Português BR  | 한국어

Paid & Subscription
3h
Lab
Professional

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.