Introducing English as the New Programming Language for Apache Spark
Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™...
Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering
We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...
Project Lightspeed Update - Advancing Apache Spark Structured Streaming
In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...
Introducing Lakehouse Federation Capabilities in Unity Catalog
Lakehouse Federation is now in public preview! Data teams face many challenges to quickly access the right data primarily due to data fragmentation...
Introducing Materialized Views and Streaming Tables for Databricks SQL
We are thrilled to announce that materialized views and streaming tables are now publicly available in Databricks SQL on AWS and Azure. Streaming...
A guide to Databricks SQL and Data Warehousing talks at Data + AI Summit 2023
It's been only 18 months since we announced Databricks SQL general availability - the serverless data warehouse on the Lakehouse - and we...
Seamlessly Migrate Your Apache Parquet Data Lake to Delta Lake
Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows...
Adaptive Query Execution in Structured Streaming
In Databricks Runtime, Adaptive Query Execution (AQE) is a performance feature that continuously re-optimizes batch queries using runtime statistics during query execution. Starting...
Announcing the General Availability of Databricks SQL Serverless !
Today, we are thrilled to announce that serverless compute for Databricks SQL is Generally Available on AWS and Azure! Databricks SQL (DB SQL)...
Latency goes subsecond in Apache Spark Structured Streaming
Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the...