Skip to main content
Platform blog

Announcing the General Availability of Azure Private Link and Azure Storage firewall support for Databricks SQL Serverless

An overview of Azure Databricks’ new enhanced serverless networking features and security best practices
Share this post

We are excited to announce the upcoming general availability of Azure Private Link support for Databricks SQL (DBSQL) Serverless, planned in April 2024, with no additional charges for use. We are also thrilled to announce that Azure Storage firewall support with stable VNet subnet IDs is now generally available for DBSQL Serverless. This blog will give an overview of the two features and associated best practices for securely accessing data in your Azure Storage account from Databricks serverless.

Maximize performance and secure workspaces using Azure Databricks serverless network connectivity features

The Databricks Data Intelligence Platform offers robust security through strong multi-layered isolation and built-in best practices, as detailed in our Trust Center, while continuing to leverage data stored in your existing Azure Storage accounts. We build on this foundation and offer two options to connect your DBSQL Serverless workloads to your Azure Storage accounts securely:

  1. Configure Azure Storage firewall to allow access based on stable VNet subnet IDs
  2. Configure Private Endpoints to use Private Link to your Storage account.

The diagram below shows the high-level connections into and out of your Azure Databricks account for serverless. In this blog, we will focus on securing your connection between DBSQL Serverless workloads and your Azure Storage.

DBSQL Serverless workloads

Azure Private Link for serverless will soon become GA and is included at no additional cost

Like many customers, you may have compliance or governance requirements to keep resources accessible on your virtual network traffic via private endpoints. For such scenarios, you can now create and maintain private endpoints for your Storage accounts and grant access to those private endpoints from serverless workloads in specified Workspaces.

As part of our upcoming general availability of Private Link on Azure Databricks for serverless, we are excited to announce that Private Link connections from Databricks SQL Serverless workloads will be available at no additional charge to you! As a result, your TCO for DBSQL Serverless on Azure Databricks gets a huge boost. It also means that Private Link connections will carry no additional charge as we add support for additional Azure Databricks serverless products and Azure resource types.

"Azure Databricks' advanced networking features offer security and simplicity in managing serverless data transformations and analytics at scale."
— Jonas Kardell, Data Science Lead, SJ AB

Azure Storage firewall support with stable VNet subnet IDs

For those not looking to use Private Link, you likely still have a requirement to lock down access to your data in Azure Storage accounts to only authorized workloads running on authorized networks. Azure Storage firewall enables you to restrict access to only clients that access your Storage account from authorized VNet subnet IDs. With this GA launch, you can configure Databricks to use a stable list of subnets within our Azure VNets to reach out to your Storage. You can obtain this list of subnet IDs directly in the product and manage access by adding them to your Azure Storage firewall rules. Combining this feature with Unity Catalog provides layered protection to ensure that only authorized workloads that also have access to the right Managed Identity can access data in your Storage.

Manage serverless network connectivity easily across a number of Workspaces

With the Network Connectivity Configuration (NCC), you can easily and centrally manage network connectivity. Using NCC enables mapping connectivity configurations to multiple Workspaces, simplifying administration by reducing the number of private endpoints you need to manage. As we continue to broaden our serverless offerings, the NCC will continue to be the single point of managing connectivity across all our serverless products.

Network Connectivity Configuration

Getting Started with Serverless Network Connectivity on Azure Databricks

Azure Storage firewall support and Azure Private Link are available on the Premium Tier version of Azure Databricks. Refer to our documentation for step-by-step instructions on configuring NCC and Azure Storage firewall support for your Databricks workspaces. While Azure Private Link is in gated public preview, contact your Azure Databricks account team for more information on how to enroll. We are planning to make Azure Private Link support for Azure Databricks serverless generally available in April 2024.

Please visit our Security and Trust Center for more information about Databricks' security best practices and features available to customers.

Try Databricks for free

Related posts

Platform blog

Announcing the General Availability of Azure Databricks support for Azure confidential computing (ACC)

November 16, 2023 by Kelly Albano and Samrat Ray in Platform Blog
Today we are excited to announce the general availability of Azure Databricks support for Azure confidential computing (ACC)! With support for Azure confidential...
Platform blog

Announcing new security controls and compliance certifications for Azure Databricks and AWS Databricks SQL Serverless

We're excited to share a new set of security controls and compliance certifications that can help with regulatory compliance on Azure Databricks and...
Platform blog

Announcing the General Availability of Private Link and Customer Managed Keys for Azure Databricks

We are excited to announce that Private Link and using customer-managed keys (CMK) for encryption are now Generally Available (GA) for Azure Databricks...
See all Platform Blog posts