Databricks Platform Demo

Let’s take a look at the architecture of the Databricks platform. While understanding the details of all the components and how they’re integrated falls under the responsibility of a platform administrator, as a data engineer, it’s good to have a broad understanding of the structure and how it all comes together.

This diagram illustrates the Databricks architecture. The control plane consists of the backend services that Databricks manages in its own cloud account aligned with the cloud service and used by the customer – AWS Azure or GCP. Though the majority of your data does not live here, some elements, such as notebook commands and workspace configurations, are stored in the control plane and encrypted at rest. Through the control plane and the associated UI and APIs it provides, one can launch clusters, start jobs and get results, and interact with table metadata.