Announcing OIDC Token Federation for Enhanced Delta Sharing Security

We are excited to introduce the Public Preview of OIDC Token Federation for Enhanced Delta Sharing Security a major security and usability enhancement for when sharing with non-Databricks recipients. With this release, data providers can securely share data with non-Databricks users on any computing platform who prefer to authenticate using a custom Identity Provider (IdP), such as Azure Entra ID or Okta. This eliminates the need for static credentials, enhances security, and enables fine-grained access control—ensuring that only the right users or machines have access to shared data.

5 Benefits of Secure OpenID Connect Token Federation

Delta Sharing is the industry’s first open-source approach to data sharing across data, analytics and AI. This means you’re not locked into any specific vendor or platform. Delta Sharing from Databricks allows for two types of sharing: Sharing with Non-Databricks Recipients and Sharing with Databricks Recipients (also known as D2D Sharing). Check out the blog "How Delta Sharing Enables Secure End-to-End Collaboration" for more details on both these types of sharing. If you’re sharing data with another Databricks customer into their Databricks account, you can already use D2D Sharing, which provides a seamless and integrated experience within the Databricks ecosystem.

On the other hand, when securely sharing with external users who are not on Databricks, Delta Sharing has always provided a simple and fast way to share data using bearer tokens. However, for non-Databricks users who prioritize enhanced security, OIDC Token Federation for Enhanced Security offers a more robust and flexible authentication mechanism. This approach minimizes exposure risks and ensures secure collaboration.

Key Benefits of using OIDC Token Federation when sharing with non-Databricks users on any computing platform:

1. IdP-Managed Identities: Customers can use their current Identity Providers (like Entra ID or Okta) for authentication, avoiding the need to set up new systems or processes.
2. Fine-Grained User Access Control: Customers gain precise control over who can access data, ensuring only the right people or systems have permissions.
3. Multi-Factor Authentication (MFA) Support: If their Identity Provider supports Multi-Factor Authentication (MFA), it adds an extra layer of protection, ensuring only authorized users access shared data.
4. Reduced Security Risks: Short-lived tokens automatically expire, minimizing the chance of unauthorized access without requiring manual intervention.
5. No Shared Secrets: Eliminates the need to distribute static credentials between Databricks, providers, and recipients.

How does the OIDC Token Federation work when sharing with non-Databricks recipients?

With OIDC Token Federation when sharing with non-Databricks recipients, each Delta Sharing recipient is configured with federation policies, ensuring that only authorized users or machines can access shared data. Here's how it works:

1. Setting up Access for Non-Databricks Recipient

• The data provider (Databricks user) configures an OIDC Token Federation policy for the recipient, specifying their external IdP (e.g., Entra ID, Okta).
• The policy defines which users, applications, or systems from the recipient’s identity system are allowed to access the shared data.

2. Secure Authentication with Short-Lived Tokens

1. When the recipient attempts to access shared data, Delta Sharing Connector will authenticate on their behalf against their configure IdP (e.g., Entra ID or Okta).
2. Upon a successful authentication, identity system creates a temporary digital pass, called a JSON Web Token (JWT), which includes information about who they are. This is shared with Delta Sharing Connector.
3. The Delta Sharing Connector will combine the JWT token issued by the IdP along the Delta Sharing request and send it to Databricks Delta Sharing Server.
4. Databricks Delta Sharing will validates the JWT against the recipient's policy, and matches the rules set by the data provider, such as verifying who issued it, who it’s for, and who is requesting access.
5. Upon a successful authentication, Databricks Delta Sharing server shares the requested data.
6. Delta Sharing Client in turn returns it to recipient workflow.

This approach eliminates the need for shared secrets. Instead, it uses temporary authentication tokens that expire quickly and it allows precise control over who or what (a specific user or machine) can access the data.

Three Authentication Scenarios Supported

OIDC Token Federation approach supports both User-to-Machine (U2M) and Machine-to-Machine (M2M) authentication flows, enabling a broad range of use cases.

1. User-to-Machine (U2M) Authentication

• A human user from the recipient organization authenticates via their IdP.
• If Multi-Factor Authentication (MFA) is enabled in the recipient’s or provider’s IdP, it will be enforced.
• The user can then use tools like Power BI or Tableau to access and analyze the shared data easily.
• The data provider can set rules to allow access only to specific people or groups, ensuring tight control over who gets access.

This demo shows how to securely share data from Databricks to Power BI with EntraID authentication

2. Machine-to-Machine (M2M) Authentication

Delta Sharing now supports two secure ways for non-Databricks recipient machines to authenticate automatically:

Scenario 1: OAuth Client Credentials Grant Flow

• The recipient or provider organization registers a Service Principal in their IdP. A service principal is like a “user identity” for applications or automated systems, allowing them to securely access resources without needing a human to log in.
• No credentials are shared externally between Databricks, Provider or Recipient, and secret management is local—everything stays secure within each organization.
• Support for Python Delta Sharing Client and Spark Delta Sharing Client ensures that recipients can access shared data through scripts/automation.

This demo shows how to securely share data from Databricks to Python Delta Sharing Client using OAuth Client Credentials Grant

Scenario 2: Managed Identity Authentication (Coming Soon)

• For workloads running in cloud environments (e.g., Azure VMs), authentication occurs automatically using managed identities.
• No secrets or manual credential management is required.
• Initial support will focus on Azure Compute, with potential expansion to other cloud providers.

This demo shows how to securely share data from Databricks to Python Delta Sharing Client using Cloud provider Managed Identity

Choosing the Identity Provider:

Customers can choose to authenticate using an External Identity Provider (recipient-managed) or an Internal Identity Provider (provider-managed).

• External Identity Provider (Recipient-Managed): The recipient’s identity system (like Entra ID or Okta) is used. The provider sets it up in the sharing policy, so the recipient controls who from their organization can access the data.
• Internal Identity Provider (Provider-Managed): The provider’s identity system is used. The provider manages authentication by adding external recipients as guests in their own identity system. This allows the provider’s system to handle access on behalf of the recipient.

What’s Next?

We will make it easier to set up secure data sharing with pre-built templates for common OIDC Federation Policies tailored for popular identity providers like Entra ID and Okta. Additionally, upcoming support for managed identity authentication will enable cloud-based workloads (e.g., Azure VMs) to authenticate without needing passwords or secrets, ensuring a seamless and secure connection to Databricks Delta Sharing endpoints.

Get Started

OIDC Token Federation for Enhanced Security when sharing with non-Databricks recipients is available in Public Preview today to AWS, GCP and Azure customers. Learn how Delta Sharing makes it easy for organizations to securely share data with non-Databricks users on any computing platform—without compromising on security.