Built on open source

Databricks engineers are the original creators of several of the world's most popular open source data technologies shown on this page. Apache Spark is described as a unified engine for executing data engineering, data science and ML workloads, and Databricks engineers also originated projects like Delta Lake and MLflow. Because Databricks built these projects, the platform stays closely aligned with the open source releases available at spark.apache.org, delta.io and mlflow.org.

Databricks runs on the same open source Apache Spark engine rather than a different or proprietary query engine. For teams weighing self-managed open source Spark against running it on Databricks, the page links directly to Comparing Spark and Databricks. Databricks additionally layers in complementary open source projects it created, such as Delta Lake for lakehouse storage and Unity Catalog for governance, which extend beyond what Apache Spark provides on its own.

Delta Lake and Apache Iceberg both let you build a lakehouse architecture on top of storage systems such as AWS S3, ADLS, GCS and HDFS. Unity Catalog is described as the industry's only universal catalog for data and AI, and Delta Sharing is described as the industry's first open protocol for secure data sharing, making it simple to share data with other organizations. You can learn more about these projects at delta.io, apache.iceberg.org, unitycatalog.io and the Delta Sharing site.

Databricks supports MLflow, which manages the ML lifecycle including experimentation, reproducibility, deployment and a central model registry, alongside popular ML libraries such as TensorFlow, PyTorch, Keras, scikit-learn and XGBoost. TensorFlow is supported as a library for deep learning and general computation on clusters, and Keras runs on top of TensorFlow as a deep learning API available in the Databricks Runtime for ML. PyTorch integrations were built through collaboration between Databricks and Facebook, the creator of PyTorch, and XGBoost is supported as a distributed gradient boosting library with bindings in Python, R and C++.

This set of projects is intended for data engineers, data scientists, ML practitioners and infrastructure teams who want to use well-known open source data and AI tools inside a single platform rather than assembling and maintaining them separately. It also serves developers who manage infrastructure as code, since the Databricks Terraform provider lets customers manage their entire Databricks workspaces alongside the rest of their cloud infrastructure using HashiCorp Terraform. Community members can also connect with other users of these projects by joining a meetup.

You can try Databricks for free directly from this page, or explore the individual open source projects on their own sites, including spark.apache.org, delta.io, mlflow.org and unitycatalog.io. The page also links to a meetup for connecting with other users of these open data and AI projects. For a deeper technical introduction, Databricks provides linked tech talks on getting started with Delta Lake and managing the machine learning lifecycle with MLflow.