Large language models (LLMs) have generated interest in effective human-AI interaction through optimizing prompting techniques. “Prompt engineering” is a growing methodology for tailoring...

Databricks’ mission is to deliver data intelligence to every enterprise by allowing organizations to understand and use their unique data to build their...

Today, we are thrilled to announce that Lilac is joining Databricks. Lilac is a scalable, user-friendly tool for data scientists to search, cluster...

In today's interconnected digital landscape, data sharing and collaboration across organizations and platforms are crucial for modern business operations. Delta Sharing, an innovative...

The observation that " software is eating the world " has shaped the modern tech industry. Today, software is ubiquitous in our lives...

Predictive Optimization intelligently optimizes your Lakehouse table data layouts for peak performance and cost-efficiency - without you needing to lift a finger.

Talk to any data science organization and they will almost unanimously tell you that the biggest challenge to building high quality AI models...

Today, we’re excited to share that we’ve completed our acquisition of MosaicML, a leading platform for creating and customizing generative AI models for...

The promise of artificial intelligence (AI) is undeniable, but its enormous potential also comes with enormous responsibilities. Companies and organizations around the world...

In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance...

The fundamental principles of governance – accountability, compliance, quality, and transparency – that are essential for data management have now become equally imperative...

Lakehouse Federation is now in public preview! Data teams face many challenges to quickly access the right data primarily due to data fragmentation...

Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural...

Generative AI will have a transformative impact on every business. Databricks has been pioneering AI innovations for a decade, actively collaborating with thousands...

Lakehouse Apps is a new way to build native applications for Databricks. Lakehouse Apps will offer the most secure way to build, distribute...

Today, we are excited to announce the preview of a Hive Metastore (HMS) interface for Databricks Unity Catalog , which allows any software...

Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the...

For a decade, Databricks has focused on democratizing data and AI for organizations around the world. And since the debut of ChatGPT last...

New Large Language Model Courses with edX As Large Language Model (LLM) applications disrupt countless industries, generative AI is becoming an important foundational...

Two weeks ago, we released Dolly , a large language model (LLM) trained for less than $30 to exhibit ChatGPT-like human interactivity (aka...

Update Apr 12, 2023: We have released Dolly 2.0, licensed for both research and commercial use. See the new blog post here...

Today we are excited to announce that Delta Sharing is generally available (GA) on AWS and Azure. With the GA release, you can...

The theme of this year's Data + AI Summit is that we are building the modern data stack with the lakehouse. A fundamental...

We're pleased to announce Databricks Marketplace, an open marketplace for exchanging data products such as datasets, notebooks, dashboards, and machine learning models. To...

We are excited to announce data clean rooms for the Lakehouse, allowing businesses to easily collaborate with their customers and partners on any...

Streaming data is a critical area of computing today. It is the basis for making quick decisions on the enormous amounts of incoming...

This week, many of the most influential engineers and researchers in the data management community are convening in-person in Philadelphia for the ACM...

Today, we are thrilled to announce that Databricks Feature Store is generally available (GA)! In this blog post, we explore how Databricks Feature...

Update: Unity Catalog is now generally available on AWS and Azure. At the Data and AI Summit 2021, we announced Unity Catalog...

Update: Delta Sharing is now generally available on AWS and Azure. Data sharing has become an essential component to drive business value as...

On Nov 2, 2021, we announced that we set the official world record for the fastest data warehouse with our Databricks SQL lakehouse...

Today, Databricks is known for our backend engineering, building and operating cloud systems that span millions of virtual machines processing exabytes of data...

Question Index What is a Data Lakehouse? What is a Data Lake? What is a Data Warehouse? How is a Data Lakehouse different...

This is a guest post from the Stanford University Computer Science Department. We thank Daniel Kang, Deepti Raghavan and Peter Bailis of Stanford...

Update: Delta Sharing is now generally available on AWS and Azure. Get an early preview of O'Reilly's new ebook for the step-by-step guidance...

Apache Spark™ has reached its 10th anniversary with Apache Spark 3.0 which has many significant improvements and new features including but not limited...

Back in 2013, we held the first Spark Summit — a gathering of the Apache Spark™ community with leading contributors and production users...

At today’s Spark + AI Summit 2020, we unveiled the next generation of the Databricks Data Science Workspace: An open and unified experience...

Watch Spark + AI Summit Keynotes here At today's Spark + AI Summit 2020, we announced that MLflow is becoming a Linux Foundation...

We’re excited to announce that the Apache Spark TM 3.0.0 release is available on Databricks as part of our new Databricks Runtime 7.0...

Some brands start out as, well, brands. A lot of work goes into the concept and painting the picture before the business is...

Read Building the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data warehouse...

Watch the announcement and demo At today’s Spark + AI Summit in Amsterdam , we announced the availability of the MLflow Model Registry...

We’re excited to announce today the release of MLflow 1.1. In this release, we’ve focused on fleshing out the tracking component of MLflow...

MLflow is an open source platform to help manage the complete machine learning lifecycle. With MLflow, data scientists can track and share experiments...

At Spark+AI Summit 2019, we announced the GA of Managed MLflow on Databricks in which we take the latest and greatest of open...

Try this tutorial in Databricks MLflow is an open source platform to help manage the complete machine learning lifecycle. With MLflow, data scientists...

Try this tutorial in Databricks Building production machine learning applications is challenging because there is no standard way to record experiments, ensure reproducible...

It’s been six months since we launched MLflow , an open source platform to manage the machine learning lifecycle, and the project has...

On August 30th, our team hosted a live webinar— Introducing MLflow: Infrastructure for a complete Machine Learning lifecycle —with Matei Zaharia, Co-Founder and...

At this year’s Spark+AI Summit , we introduced MLflow , an open source platform to simplify the machine learning lifecycle. In the 3...

Learn more about Managed MLflow on Databricks Everyone who has tried to do machine learning development knows that it is complex. Beyond the...

Over the past few years, the demand for artificial intelligence (AI) and machine learning capabilities has surged with innovations in natural language processing...

We’re excited to announce that Spark Summit is expanding its coverage in 2018 to include in-depth content on artificial intelligence. We are also...

This is a joint blog post from Matei Zaharia, Chief Technologist at Databricks and Peter Carlin, Distinguished Engineer at Microsoft. Today at Microsoft...

Combining the best of data warehouses, data lakes and streaming For an in-depth look and demo, join the webinar . Today we are...

Apache Spark has seen immense growth over the past several years. The size and scale of this Spark Summit is a true reflection...

Spark Summit will be held in Boston on Feb 7-9, 2017. Check out the full agenda and get your ticket before it sells...

Apache Spark 2.0 adds the first version of a new higher-level API, Structured Streaming, for building continuous applications . The main goal is...

Since its release, Spark Streaming has become one of the most widely used distributed streaming engines, thanks to its high-level API and exactly-once...

Today, we're excited to announce the general availability of Apache Spark 2.0 on Databricks. This release builds on what the community has learned...

As developers at heart, we at Databricks are committed to the development of Apache Spark and the continued growth of the community. Today...

To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . 2015 has been a phenomenal year...

To learn more about Apache Spark, attend Spark Summit East in New York in Feb 2016 . 2015 has been a year of...

Developers have always loved Apache Spark for providing APIs that are simple yet powerful, a combination of traits that makes complex analysis possible...

We ran the Spark Survey 2015 this summer to gain insights on how organizations are using Apache Spark. The results of this year’s...

With so many distributed stream processing engines available, people often ask us about the unique benefits of Apache Spark Streaming . From early...

We are excited to announce today, at Spark Summit 2015 , the general availability of the Databricks – a hosted data platform from...

Check out the Why the Data Lakehouse is Your Next Data Warehouse ebook to discover the inner workings of the Databricks Lakehouse Platform...

Today, we’re celebrating an important milestone for the Apache Spark project -- it’s now been five years since Spark was first open sourced...

2014 has been a year of tremendous growth for Apache Spark. It became the most active open source project in the Big Data...

Today we are happy to announce that the complete Learning Spark book is available from O’Reilly in e-book form with the print copy...

This post originally appeared in insideBIGDATA and is reposted here with permission. With the second Spark Summit behind us, we wanted to take...

One of Apache Spark’s main goals is to make big data applications easier to write. Spark has always had concise APIs in Scala...

This article was cross-posted in the Cloudera developer blog . Apache Spark is well known today for its performance benefits over MapReduce...

This year has seen unprecedented growth in both the user and contributor communities around Apache Spark . This rapid growth validates the tremendous...

When we announced that the original team behind Apache Spark is starting a company around the project, we got a lot of excited...