We recently announced our AI-generated documentation feature , which uses large language models (LLMs) to automatically generate documentation for tables and columns in...

The observation that " software is eating the world " has shaped the modern tech industry. Today, software is ubiquitous in our lives...

We are excited to announce that we have completed our acquisition of Arcion , a leading provider for real-time data replication technologies. Arcion’s...

Predictive Optimization intelligently optimizes your Lakehouse table data layouts for peak performance and cost-efficiency - without you needing to lift a finger.

We are excited to announce the public preview of the next generation of Databricks SQL dashboards, dubbed Lakeview dashboards . Available today, this...

Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our...

We are thrilled to announce the opening of Databricks’ latest development center in Belgrade, Serbia. This addition joins our existing R&D centers in...

Today, we’re excited to share that we’ve completed our acquisition of MosaicML, a leading platform for creating and customizing generative AI models for...

Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™...

We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in...

In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...

We are incredibly excited to announce that the team behind Rubicon is joining Databricks. Founded by large scale infrastructure builders, Akhil Gupta and...

For a decade, Databricks has focused on democratizing data and AI for organizations around the world. And since the debut of ChatGPT last...

Last year Spark Connect was introduced at the Data and AI Summit. As part of the recently released Apache SparkTM 3.4, Spark Connect...

Today, we are happy to announce the availability of Apache Spark™ 3.4 on Databricks as part of Databricks Runtime 13.0 . We extend...

Two weeks ago, we released Dolly , a large language model (LLM) trained for less than $30 to exhibit ChatGPT-like human interactivity (aka...

We are excited to announce that Datajoy will be joining Databricks to accelerate our mission to unlock the value of data for everyone...

At last week's Data and AI Summit, we highlighted a new project called Spark Connect in the opening keynote. This blog post walks...

The theme of this year's Data + AI Summit is that we are building the modern data stack with the lakehouse. A fundamental...

Today we are thrilled to announce a full lineup of open source connectors for Go , Node.js , Python , as well as...

Streaming data is a critical area of computing today. It is the basis for making quick decisions on the enormous amounts of incoming...

Databricks SQL Serverless is now generally available. Read our blog to learn more. We are excited to announce the availability of serverless compute...

This week, many of the most influential engineers and researchers in the data management community are convening in-person in Philadelphia for the ACM...

Today we are happy to announce the availability of Apache Spark™ 3.3 on Databricks as part of Databricks Runtime 11.0 . We want...

Today we’re pleased to announce the availability of Databricks SQL in public preview on Google Cloud . With this announcement, customers can further...

Today, we are thrilled to announce that Databricks SQL is Generally Available (GA)! This follows our earlier announcements about Databricks SQL’s world record-setting...

Today, we are excited to announce the opening of our Seattle R&D site and our plan to hire hundreds of engineers in Seattle...

Databricks SQL is now generally available on AWS and Azure. Today, we are excited to announce that Databricks SQL will use the ANSI...

On Nov 2, 2021, we announced that we set the official world record for the fastest data warehouse with our Databricks SQL lakehouse...

We are excited to announce a new program called Databricks Engineering Fellowship to recognize new graduates with exceptional academic achievements or extracurricular impact...

At Databricks, we often use the phrase “the future is open” to refer to technology; it reflects our belief that open data architecture...

Today, we are proud to announce that Databricks SQL has set a new world record in 100TB TPC-DS , the gold standard performance...

Today, Databricks is known for our backend engineering, building and operating cloud systems that span millions of virtual machines processing exabytes of data...

We are excited to announce the availability of Apache Spark™ 3.2 on Databricks as part of Databricks Runtime 10.0 . We want to...

Databricks SQL is now generally available on AWS and Azure. Originally announced at Data + AI Summit 2020 Europe, Databricks SQL lets you...

Question Index What is a Data Lakehouse? What is a Data Lake? What is a Data Warehouse? How is a Data Lakehouse different...

Databricks SQL is now generally available on AWS and Azure. Business Intelligence (BI) tools such as Tableau and Microsoft Power BI are notoriously...

We are excited to announce the availability of Apache Spark 3.1 on Databricks as part of Databricks Runtime 8.0 . We want to...

Back in 2013, we held the first Spark Summit — a gathering of the Apache Spark™ community with leading contributors and production users...

This morning at Spark and AI Summit, we announced that Databricks has acquired Redash, the company behind the popular open source project of...

We’re excited to announce that the Apache Spark TM 3.0.0 release is available on Databricks as part of our new Databricks Runtime 7.0...

Some brands start out as, well, brands. A lot of work goes into the concept and painting the picture before the business is...

Read Building the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data warehouse...

We started Databricks in 2013 in a tiny little office in Berkeley with the belief that data has the potential to solve the...

Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. At today’s Spark +...

Today we are excited to announce Brickchain , the next generation technology for zettabyte-scale analytics, by harnessing all the compute power on the...

UPDATED: 11/19/2018 We are excited to announce the availability of Apache Spark 2.4 on Databricks as part of the Databricks Runtime 5.0...

Apache Spark has become the de facto unified analytics engine for big data processing in a distributed environment. Yet we are seeing more...

Today we are happy to announce the availability of Apache Spark 2.3.0 on Databricks as part of its Databricks Runtime 4.0. We want...

In an earlier blog post , we analyzed the performance impact of Meltdown and Spectre on big data workloads in the cloud. In...

Last week, the details of two industry-wide security vulnerabilities, known as Meltdown and Spectre , were released. These exploits enable cross-VM and cross-process...

We are excited to announce the general availability of Databricks Cache, a Databricks Runtime feature as part of the Unified Analytics Platform that...

For a deeper dive on these benchmarks, watch the webinar featuring Reynold Xin. Performance is often a key factor in choosing big data...

Try this notebook on Databricks When MapReduce was introduced 15 years ago, it showed the world a glimpse into the future. For the...

At Databricks, our engineers guide thousands of organizations to define their big data and cloud strategies. When migrating big data workloads to the...

A major value Databricks provides is the automatic provisioning, configuration, and tuning of clusters of machines that process data. Running on these machines...

This blog post describes our experience debugging a failing test case caused by a cross join query running “too fast.” Because the root...

Spark Summit will be held in Boston on Feb 7-9, 2017. Check out the full agenda and get your ticket before it sells...

Spark Summit will be held in Boston on Feb 7-9, 2017. Check out the full agenda and get your ticket before it sells...

We are excited to share with you that a joint effort by Nanjing University, Alibaba Group, and Databricks set a new world record...

Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications . The main goal is...

Today, we're excited to announce the general availability of Apache Spark 2.0 on Databricks. This release builds on what the community has learned...

When our team at Databricks planned our contributions to the upcoming Apache Spark 2.0 release, we set out with an ambitious goal by...

For the past few months, we have been busy contributing to the next major release of the big data open source software we...

Update: this post is an April Fools joke. It is not an actual project we're working on. For the past three years, our...

Last week, Stack Overflow released the result of their 2016 developer survey . This is one of the most significant surveys in the...

To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . 2015 has been a year of...

To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . Today we are happy to announce...

Developers have always loved Apache Spark for providing APIs that are simple yet powerful, a combination of traits that makes complex analysis possible...

Today we are happy to announce the availability of an Apache Spark 1.6 preview package in Databricks. The Apache Spark 1.6.0 release is...

The inaugural Spark Summit Europe will be held in Amsterdam on October 27 - 29. Check out the full agenda and get your...

To try new features highlighted in this blog post, download Spark 1.5 or sign up Databricks for a 14-day free trial today...

The inaugural Spark Summit Europe will be held in Amsterdam this October. Check out the full agenda and get your ticket before it...

We are excited to announce that starting today, Apache Spark 1.5.0 is available as a preview in Databricks. Our users can now choose...

We introduced DataFrames in Apache Spark 1.3 to make Apache Spark much easier to use. Inspired by data frames in R and Python...

In a previous blog post , we looked back and surveyed performance improvements made to Apache Spark in the past year. In this...

Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...

Check out the Why the Data Lakehouse is Your Next Data Warehouse ebook to discover the inner workings of the Databricks Lakehouse Platform...

Yesterday, to celebrate Apache Spark’s 5 year old birthday, we looked back at the history of the project. Today, we are happy to...

Today, we are excited to announce a new DataFrame API designed to make big data processing even easier for a wider audience. When...

A month ago, we shared with you our entry to the 2014 Gray Sort competition, a 3rd-party benchmark measuring how fast a system...

Update November 5, 2014 : Our benchmark entry has been reviewed by the benchmark committee and Apache Spark has won the Daytona GraySort...

Recommendation systems are among the most popular applications of machine learning. The idea is to predict whether a customer would like a certain item: a product, a movie, or a song. Scale is a key concern for recommendation systems, since computational complexity increases with the size of a company's customer base. In this blog post, we discuss how Apache Spark MLlib enables building recommendation models from billions of records in just a few lines of Pyt

With the introduction of Spark SQL and the new Hive on Apache Spark effort ( HIVE-7292 ), we get asked a lot about...

Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...

The AMPLab at UC Berkeley, with help from Databricks, recently released an update to the Big Data Benchmark . This benchmark uses Amazon...