Announcing the General Availability of Azure Private Link and Azure Storage firewall support for Databricks SQL Serverless

We are excited to announce the general availability of Azure Private Link support for Databricks SQL (DBSQL) Serverless with no data processing charges for use. We are also thrilled to announce that Azure Storage firewall support with stable VNet subnet IDs is generally available for DBSQL Serverless. These features can be enabled by leveraging the Network Connectivity Configuration (NCC). This blog will give an overview of the two features and associated best practices for securely accessing data in your Azure Storage account from Databricks serverless.

Maximize performance and secure workspaces using Azure Databricks serverless network connectivity features

The Databricks Data Intelligence Platform offers robust security through strong multi-layered isolation and built-in best practices, as detailed in our Trust Center, while continuing to leverage data stored in your existing Azure Storage accounts. We build on this foundation and offer two options to connect your DBSQL Serverless workloads to your Azure Storage accounts securely:

1. Configure Azure Storage firewall to allow access based on stable VNet subnet IDs
2. Configure Private Endpoints to use Private Link to your Storage account.

The diagram below shows the high-level connections into and out of your Azure Databricks account for serverless. In this blog, we will focus on securing your connection between DBSQL Serverless workloads and your Azure Storage.

Azure Private Link for serverless is included at no additional cost for data processing

Like many customers, you may have compliance or governance requirements to keep resources accessible on your virtual network traffic via private endpoints. For such scenarios, you can now create and maintain private endpoints for your Storage accounts and grant access to those private endpoints from serverless workloads in specified Workspaces.

As part of the general availability of Private Link on Azure Databricks for serverless, we are excited to announce that Private Link connections from Databricks SQL Serverless workloads are available with no charge for data processing! As a result, your TCO for DBSQL Serverless on Azure Databricks gets a huge boost. It also means that Private Link connections will carry no additional data processing charges as we add support for additional Azure Databricks serverless products and Azure resource types.

"Azure Databricks' advanced networking features offer security and simplicity in managing serverless data transformations and analytics at scale."

— Jonas Kardell, Data Science Lead, SJ AB

Azure Storage firewall support with stable VNet subnet IDs

For those not looking to use Private Link, you likely still have a requirement to lock down access to your data in Azure Storage accounts to only authorized workloads running on authorized networks. Azure Storage firewall enables you to restrict access to only clients that access your Storage account from authorized VNet subnet IDs. With this GA launch, you can configure Databricks to use a stable list of subnets within our Azure VNets to reach out to your Storage. You can obtain this list of subnet IDs directly in the product and manage access by adding them to your Azure Storage firewall rules. Combining this feature with Unity Catalog provides layered protection to ensure that only authorized workloads that also have access to the right Managed Identity can access data in your Storage.

Manage serverless network connectivity easily across a number of Workspaces

With the Network Connectivity Configuration (NCC), you can easily and centrally manage network connectivity. Using NCC enables mapping connectivity configurations to multiple Workspaces, simplifying administration by reducing the number of private endpoints you need to manage. As we continue to broaden our serverless offerings, the NCC will continue to be the single point of managing connectivity across all our serverless products.

Getting Started with Serverless Network Connectivity on Azure Databricks

Azure Storage firewall support and Azure Private Link are available on the Premium Tier version of Azure Databricks. Refer to our documentation for step-by-step instructions on configuring the NCC for Azure Storage firewall support and private connectivity on your Databricks workspaces.

Please visit our Security and Trust Center for more information about Databricks' security best practices and features available to customers.