Skip to main content
Platform blog

Introducing Databricks Fleet Clusters for AWS

Ankit Batra
Piyush Singh
Quinn Leng
Andrew Bagshaw
Share this post

We're excited to announce the general availability of Databricks Fleet clusters on AWS.

What are Fleet clusters?

Databricks Fleet clusters unlock the potential of Spot pricing without the hassle of manual instance selection by allowing Databricks to intelligently optimize and automate the process for you. You can use AWS Spot instances at the lowest cost possible, all while ensuring availability, thanks to Databricks' ability to flex across multiple instances in a single cluster. Avoid the complexities of managing your cloud infrastructure and spend your time on what truly matters: your data-driven insights.

Databricks Fleet clusters introduce a set of new fleet instance families for our AWS customers: m-fleet, md-fleet, r-fleet and rd-fleet in sizes xlarge, 2xlarge, 4xlarge and 8xlarge. m instances have 4 GB memory/vCPU, r instances have 8 GB memory/vCPU, and instances with d have local NVME SSD attached.

Introducing Databricks Fleet Clusters for AWS

Under the hood, when a cluster uses one of these fleet instance types, Databricks will select the matching physical AWS instance types with the best price and availability to use in your cluster. Databricks has restricted the set of possible instance combinations to ensure that you get maximum stability and performance out of your cluster. We'll continue to update which instances we use behind the scenes as AWS launches additional types so that you no longer have to manually tune your cluster definitions. When your cluster uses spot instances, we'll select the instance types with the lowest price and least likelihood of spot termination.

For example, if a cluster has the following configuration:

  Driver: rd-fleet.xlarge
  Workers: rd-fleet.4xlarge
  Number of Workers: 3

The cluster may end up acquiring the following instance types:

  Driver: r6id.xlarge
  Workers: r6id.4xlarge, r6id.4xlarge, r5d.4xlarge

The specific instances are determined by the price and capacity conditions at the time of cluster creation. Databricks will bill you based on the normal DBU rates for each of the acquired instances.

How can I use it?

Simply create a new cluster or instance pool to use a fleet instance type. We recommend using fleet instance types for both worker and driver.

You will also need to make sure that the AWS IAM role for your databricks workspace includes the most up-to-date permissions listed in our documentation. If any permissions are missing, you'll be prompted to add them when attempting to start a fleet cluster.

NOTE: Older workspaces will be updated to support fleet clusters over the coming weeks. If you have any issues with access, please contact customer support.

Try Databricks for free

Related posts

Platform blog

Best Practices for Cost Management on Databricks

October 18, 2022 by Tomasz Bacewicz and Greg Wood in Platform Blog
This blog is part of our Admin Essentials series, where we'll focus on topics important to those managing and maintaining Databricks environments. Keep...
Platform blog

Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS: Part 1

September 30, 2022 by JD Braun in Platform Blog
As a cloud engineering team, it may seem like a daily occurrence that you get a request from a team to deploy a...
Platform blog

Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS: Part 2

January 27, 2023 by JD Braun and Al Thrussell in Platform Blog
This is part two of a three-part series in Best Practices and Guidance for Cloud Engineers to deploy Databricks on AWS. You can...
See all Platform Blog posts