Skip to main content
Company Blog

The 2023 State of Data + AI: How Businesses Are Preparing for the New Age of AI

Share this post

The historic surge of interest in large language models (LLMs) since ChatGPT launched to the public late last year has made the topic inescapable. Not only is the technology improving at an unparalleled cadence, but companies are also building their own models like never before. Now, predictive models are underpinning mission-critical tasks, giving organizations a window into the future instead of just a review of the past, and helping them operate quicker and leaner.

On the cusp of this new computing revolution, we were eager to learn exactly where enterprises are at in this transformation, as well as the platforms and tools they’re using to take advantage of it. By analyzing anonymized usage data from more than 9,000 global Databricks customers, we’ve compiled the 2023 State of Data + AI, a comprehensive look at organizations’ data and AI initiatives.

Here’s a glimpse at what we discovered:

Use of LLMs in 2023

  • The hype around LLMs is real: From the end of November 2022 to the beginning of May 2023, the usage of SaaS LLMs, which are used to access models like OpenAI, grew exponentially with Lakehouse customers at 1310%. Transformer-related libraries like HuggingFace (an NLP toolkit and model hub), which are used to train homegrown LLMs and were in demand even before the launch of ChatGPT, grew 82% within the same time frame.


  • Data transformation and integration is more vital than ever: The fastest growing tools on Databricks are dbt (206% YoY) and FiveTran (181%). But of the 10 most popular data and AI products, six are data integration tools, including Informatica and Qlik, making it the fastest growing market on the Databricks Lakehouse.

The ratio of logged vs. registered ML models

  • Enterprises are doing more AI projects than ever before – and getting better at it: The number of models that are candidates for production (used in operations) grew 411% year-over-year, while the number of experimental projects grew 54%. Our data also shows that, on average, one in three experimental models are a candidate for the real-world, compared to one in five last year, suggesting organizations are getting better at building and scaling these projects.
  • AI is growing, but don’t forget traditional data analytics: The Lakehouse is increasingly being used for data warehousing, including serverless data warehousing with Databricks SQL, which grew 144% YoY.

While it’s still early days, these emerging trends are bound to define the future of AI. And business leaders need to pay attention. It's never been more clear: the companies that harness the power of DS/ML will lead the next generation of data.

Download the full report here to learn more!

Try Databricks for free

Related posts

Company blog

Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM

Two weeks ago, we released Dolly , a large language model (LLM) trained for less than $30 to exhibit ChatGPT-like human interactivity (aka...
Company blog

Enroll in our New Expert-Led Large Language Models (LLMs) Courses on edX

New Large Language Model Courses with edX As Large Language Model (LLM) applications disrupt countless industries, generative AI is becoming an important foundational...
Platform blog

Announcing the General Availability of Databricks SQL Serverless !

Today, we are thrilled to announce that serverless compute for Databricks SQL is Generally Available on AWS and Azure! Databricks SQL (DB SQL)...
See all Company Blog posts