Skip to main content

Announcing public preview of Databricks Asset Bundles: Apply software development best practices with ease

Saad Ansari
Lennart Kats
Pieter Noordhuis
Jim Allen Wallace
Share this post

We are delighted to announce that Databricks Asset Bundles are now in public preview. Bundles, for short, facilitate the adoption of software engineering best practices, including source control, code review, testing and continuous integration and delivery (CI/CD). Bundles enable data engineers, data scientists and ML engineers to express data, analytics and AI projects as source files. These source files provide an end-to-end definition of a project, including how it should be tested and deployed to the Lakehouse. This definition can easily be edited, tested and deployed.

Automate testing and deployment

CI/CD is essential in modern software development, helping to automate tests and deployments, thus speeding up release cycles and reducing errors. You can configure bundles to describe how to deploy and test projects. Bundles can automatically deploy across multiple workspaces, regions and clouds. You can use your existing CI/CD system to automate testing and deployment of bundles.

Develop, test, and deploy with IDEs

Bundles are typically used with a local IDE and work with the Databricks extension for VS Code - you can configure your IDE for bundle configuration auto-completion so it is easier to author your bundles. You can iteratively develop using a personal copy without affecting collaborators and execute resources such as jobs or DLT pipelines before they are deployed to production.

Standardize development with templates

Use default templates or create your own custom templates to set dev, staging and prod workspaces, permissions, default libraries, service principals, default Apache Spark configs and more. You can use bundles to set these configurations once and reuse them to streamline developing and deploying code.

Get started

Begin using Databricks Asset Bundles in only a few short steps:

  1. Install the latest CLI
    When using Homebrew: brew tap databricks/tap; brew install databricks
    Note: We are adding support for other software package managers soon. See our detailed instructions for installation without Homebrew.
  2. Authenticate to Databricks
    databricks configure
  3. Generate and customize your first bundle
    databricks bundle init
    This generates a bundle with a README that works out of the box. You can customize the default configuration by updating any defaults you like.
  4. Deploy your project to your development workspace
    databricks bundle deploy

How do Databricks Asset Bundles relate to Terraform and dbx?

Bundles are recommended for applying CI/CD to developing data, analytics and AI projects. Bundles and the Databricks Terraform provider work well together: bundles can be used to define lakehouse assets, while Terraform can be used for infrastructure, such as workspaces, service principals, and cloud assets. The Databricks Labs project dbx has been in an "experimental" release state since its launch, and we encourage migrating to bundles instead.

What's next

Databricks Asset Bundles are now in public preview, which means they are ready for production usage. In the coming months, you can look forward to additional features such as support for including dashboards as source files. Bundles already work with Databricks Workflows jobs, Delta Live Tables pipelines, ML Experiments, ML Registered Models and Model Serving endpoints. ML Ops Stack (currently in private preview) also uses bundle templates for productionizing your ML Projects. Learn more about bundles in our docs pages and let us know your feedback.

Try Databricks for free

Related posts

Databricks ❤️ IDEs

February 14, 2023 by Patrick Wendell in
Happy Valentine's Day! Databricks ❤️ Visual Studio Code. On this lovely day, we are thrilled to announce a new and powerful development experience...

Debug your code and notebooks by using Visual Studio Code

Earlier this year we launched the official Databricks extension for Visual Studio Code. Today we are adding support for interactive debugging and local...

Bringing Software Engineering Best Practices to Life Sciences R&D at Exai Bio

October 3, 2023 by Sukh Sekhon and Helen Li in
This blog was written in collaboration with Sukh Sekhon, Software Engineer, Cloud Infrastructure and Helen Li, Sr. Director of Engineering at Exai Bio...
See all Platform Blog posts