Skip to main content
Login
      • Discover
        • For Executives
          • For Startups
            • Lakehouse Architecture
              • Mosaic Research
              • Customers
                • Featured Stories
                  • See All Customers
                  • Partners
                    • Cloud Providers
                      Databricks on AWS, Azure, GCP, and SAP
                      • Consulting & System Integrators
                        Experts to build, deploy and migrate to Databricks
                        • Technology Partners
                          Connect your existing tools to your Lakehouse
                          • C&SI Partner Program
                            Build, deploy or migrate to the Lakehouse
                            • Data Partners
                              Access the ecosystem of data consumers
                              • Partner Solutions
                                Find custom industry and migration solutions
                                • Built on Databricks
                                  Build, market and grow your business
                                • Databricks Platform
                                  • Platform Overview
                                    A unified platform for data, analytics and AI
                                    • Data Management
                                      Data reliability, security and performance
                                      • Sharing
                                        An open, secure, zero-copy sharing for all data
                                        • Data Warehousing
                                          Serverless data warehouse for SQL analytics
                                          • Governance
                                            Unified governance for all data, analytics and AI assets
                                            • Real-Time Analytics
                                              Real-time analytics, AI and applications made simple
                                              • Artificial Intelligence
                                                Build and deploy ML and GenAI applications
                                                • Data Engineering
                                                  ETL and orchestration for batch and streaming data
                                                  • Business Intelligence
                                                    Intelligent analytics for real-world data
                                                    • Data Science
                                                      Collaborative data science at scale
                                                    • Integrations and Data
                                                      • Marketplace
                                                        Open marketplace for data, analytics and AI
                                                        • IDE Integrations
                                                          Build on the Lakehouse in your favorite IDE
                                                          • Partner Connect
                                                            Discover and integrate with the Databricks ecosystem
                                                          • Pricing
                                                            • Databricks Pricing
                                                              Explore product pricing, DBUs and more
                                                              • Cost Calculator
                                                                Estimate your compute costs on any cloud
                                                              • Open Source
                                                                • Open Source Technologies
                                                                  Learn more about the innovations behind the platform
                                                                • Databricks for Industries
                                                                  • Communications
                                                                    • Media and Entertainment
                                                                      • Financial Services
                                                                        • Public Sector
                                                                          • Healthcare & Life Sciences
                                                                            • Retail
                                                                              • Manufacturing
                                                                                • See All Industries
                                                                                • Cross Industry Solutions
                                                                                  • Cybersecurity
                                                                                    • Marketing
                                                                                    • Migration & Deployment
                                                                                      • Data Migration
                                                                                        • Professional Services
                                                                                        • Solution Accelerators
                                                                                          • Explore Accelerators
                                                                                            Move faster toward outcomes that matter
                                                                                          • Training and Certification
                                                                                            • Learning Overview
                                                                                              Hub for training, certification, events and more
                                                                                              • Training Overview
                                                                                                Discover curriculum tailored to your needs
                                                                                                • Databricks Academy
                                                                                                  Sign in to the Databricks learning platform
                                                                                                  • Certification
                                                                                                    Gain recognition and differentiation
                                                                                                    • University Alliance
                                                                                                      Want to teach Databricks? See how.
                                                                                                    • Events
                                                                                                      • Data + AI Summit
                                                                                                        • Data + AI World Tour
                                                                                                          • Data Intelligence Days
                                                                                                            • Event Calendar
                                                                                                            • Blog and Podcasts
                                                                                                              • Databricks Blog
                                                                                                                Explore news, product announcements, and more
                                                                                                                • Databricks Mosaic Research Blog
                                                                                                                  Discover the latest in our Gen AI research
                                                                                                                  • Data Brew Podcast
                                                                                                                    Let’s talk data!
                                                                                                                    • Champions of Data + AI Podcast
                                                                                                                      Insights from data leaders powering innovation
                                                                                                                    • Get Help
                                                                                                                      • Customer Support
                                                                                                                        • Documentation
                                                                                                                          • Community
                                                                                                                          • Dive Deep
                                                                                                                            • Resource Center
                                                                                                                              • Demo Center
                                                                                                                              • Company
                                                                                                                                • Who We Are
                                                                                                                                  • Our Team
                                                                                                                                    • Databricks Ventures
                                                                                                                                      • Contact Us
                                                                                                                                      • Careers
                                                                                                                                        • Working at Databricks
                                                                                                                                          • Open Jobs
                                                                                                                                          • Press
                                                                                                                                            • Awards and Recognition
                                                                                                                                              • Newsroom
                                                                                                                                              • Security and Trust
                                                                                                                                                • Security and Trust
                                                                                                                                            • Data and AI summit

                                                                                                                                              JUNE 9–12 | SAN FRANCISCO

                                                                                                                                              Data + AI Summit is almost here — don’t miss the chance to join us in San Francisco!

                                                                                                                                              REGISTER
                                                                                                                                            • Ready to get started?
                                                                                                                                            • Get a Demo
                                                                                                                                            Data and AI summit

                                                                                                                                            JUNE 9–12 | SAN FRANCISCO

                                                                                                                                            Data + AI Summit is almost here — don’t miss the chance to join us in San Francisco!

                                                                                                                                            REGISTER
                                                                                                                                            • Login
                                                                                                                                            • Try Databricks
                                                                                                                                            1. Blog
                                                                                                                                            2. /
                                                                                                                                              Data Leader
                                                                                                                                            3. /
                                                                                                                                              Article

                                                                                                                                            Data Intelligence and AI Trends: Top products, RAG and more

                                                                                                                                            The playbook for your data and AI strategy

                                                                                                                                            State of Data and AI

                                                                                                                                            Published: June 11, 2024

                                                                                                                                            Data Leader3 min read

                                                                                                                                            by Kunal Marwah, Siddesh Pore, Amir Lyall and Avi Sooriyarachchi

                                                                                                                                            Share this post

                                                                                                                                            Keep up with us

                                                                                                                                            Generative AI fever shows no signs of cooling off. As pressure and excitement build to execute strong GenAI strategies, data leaders and practitioners are searching for the best platform, tools and use cases to help them get there.

                                                                                                                                            How is this playing out in the real world? We just launched the 2024 State of Data + AI, which leverages data from our 10,000 global customers, to understand how organizations across industries are approaching AI. While our report covers a broad range of themes relevant to any data-driven company, clear trends emerged on the GenAI journey.

                                                                                                                                            Here’s a snapshot of what we discovered:

                                                                                                                                            Top 10 data and AI products: the GenAI Stack is forming

                                                                                                                                            With any new technology, developers will experiment with lots of different tools to figure out what works best for them.

                                                                                                                                            Our Top 10 Data and AI Products showcase the most widely-adopted integrations on the Databricks Data Intelligence Platform. From data integration to model development, this list shows how companies are investing in their stack to support new GenAI priorities:

                                                                                                                                            Hugging Face transformers jump to No. 2

                                                                                                                                            In just 12 months, Hugging Face jumps from spot #4 to spot #2. Many companies use the open source platform’s transformer models together with their enterprise data to build and fine-tune foundation models. 

                                                                                                                                            LangChain becomes top product months after integration

                                                                                                                                            LangChain, an open source toolchain for working with and building proprietary LLMs, rose to spot #4 in less than one year of integration. When companies build their own modern LLM applications and work with specialized transformer-related Python libraries to train the models, LangChain enables them to develop prompt interfaces or integrations to other systems.

                                                                                                                                            Enterprise GenAI is all about customizing LLMs

                                                                                                                                            Last year, our data showed SaaS LLMs as the “it” tool of 2023, when analyzing the most popular LLM Python libraries. This year, our data is showing the use of general- purpose LLMs continue but with slowed year-over-year growth.

                                                                                                                                            This year’s strategy has taken a major shift. Our data shows that companies are hyper-focused on augmenting LLMs with their custom data vs just using standalone off-the-shelf LLMs.

                                                                                                                                            Businesses want to harness the power of SaaS LLMs, but also improve the accuracy and mold the underlying models to better work for them. With RAG, companies can use something like an employee handbook or their own financial statements so the model can start to generate outputs that are specific to the business. And there’s huge demand across our customers to build these customized systems. The use of vector databases, a vital component of RAG models, grew 377% in the last year – including a 186% jump after Databricks Vector Search went into Public Preview.

                                                                                                                                            The enterprise AI strategy and open LLMs

                                                                                                                                            As companies build their technology stacks, open source is making its mark. In fact, 9 of the 10 top products are open source, including our two big GenAI players: Hugging Face and LangChain.

                                                                                                                                            Open-source LLMs also offer many enterprise benefits, such as the ability to customize them to your organization's unique needs and use cases. We analyzed the open source model usage of Meta Llama and Mistral, the two biggest players, to understand which models companies gravitated toward.

                                                                                                                                            Use of Open Source LLMs

                                                                                                                                            With each model, there is a trade-off between cost, latency and performance. Together, usage of the two smallest Meta Llama 2 models (7B and 13B) is significantly higher than the largest, Meta Llama 2 70B.

                                                                                                                                            Across both Llama and Mistral users, 77% choose models with 13B parameters or fewer. This suggests that companies care significantly about cost and latency.

                                                                                                                                            Dive deeper into these and other trends in the 2024 State of Data + AI. Consider it your playbook for an effective data and AI strategy. Download the full report here.

                                                                                                                                            Keep up with us

                                                                                                                                            Recommended for you

                                                                                                                                            Share this post

                                                                                                                                            Never miss a Databricks post

                                                                                                                                            Subscribe to the categories you care about and get the latest posts delivered to your inbox

                                                                                                                                            Sign up

                                                                                                                                            What's next?

                                                                                                                                            Winning at GenAI: Building the right processes for the data intelligence future

                                                                                                                                            Product

                                                                                                                                            August 30, 2024/6 min read

                                                                                                                                            Winning at GenAI: Building the right processes for the data intelligence future

                                                                                                                                            The role of AI in changing company structures and dynamics

                                                                                                                                            Data Strategy

                                                                                                                                            November 12, 2024/9 min read

                                                                                                                                            The role of AI in changing company structures and dynamics

                                                                                                                                            databricks logo
                                                                                                                                            Why Databricks
                                                                                                                                            Discover
                                                                                                                                            • For Executives
                                                                                                                                            • For Startups
                                                                                                                                            • Lakehouse Architecture
                                                                                                                                            • Mosaic Research
                                                                                                                                            Customers
                                                                                                                                            • Featured
                                                                                                                                            • See All
                                                                                                                                            Partners
                                                                                                                                            • Cloud Providers
                                                                                                                                            • Technology Partners
                                                                                                                                            • Data Partners
                                                                                                                                            • Built on Databricks
                                                                                                                                            • Consulting & System Integrators
                                                                                                                                            • C&SI Partner Program
                                                                                                                                            • Partner Solutions
                                                                                                                                            Discover
                                                                                                                                            • For Executives
                                                                                                                                            • For Startups
                                                                                                                                            • Lakehouse Architecture
                                                                                                                                            • Mosaic Research
                                                                                                                                            Customers
                                                                                                                                            • Featured
                                                                                                                                            • See All
                                                                                                                                            Partners
                                                                                                                                            • Cloud Providers
                                                                                                                                            • Technology Partners
                                                                                                                                            • Data Partners
                                                                                                                                            • Built on Databricks
                                                                                                                                            • Consulting & System Integrators
                                                                                                                                            • C&SI Partner Program
                                                                                                                                            • Partner Solutions
                                                                                                                                            Product
                                                                                                                                            Databricks Platform
                                                                                                                                            • Platform Overview
                                                                                                                                            • Sharing
                                                                                                                                            • Governance
                                                                                                                                            • Artificial Intelligence
                                                                                                                                            • Business Intelligence
                                                                                                                                            • Data Management
                                                                                                                                            • Data Warehousing
                                                                                                                                            • Real-Time Analytics
                                                                                                                                            • Data Engineering
                                                                                                                                            • Data Science
                                                                                                                                            Pricing
                                                                                                                                            • Pricing Overview
                                                                                                                                            • Pricing Calculator
                                                                                                                                            Open Source
                                                                                                                                            Integrations and Data
                                                                                                                                            • Marketplace
                                                                                                                                            • IDE Integrations
                                                                                                                                            • Partner Connect
                                                                                                                                            Databricks Platform
                                                                                                                                            • Platform Overview
                                                                                                                                            • Sharing
                                                                                                                                            • Governance
                                                                                                                                            • Artificial Intelligence
                                                                                                                                            • Business Intelligence
                                                                                                                                            • Data Management
                                                                                                                                            • Data Warehousing
                                                                                                                                            • Real-Time Analytics
                                                                                                                                            • Data Engineering
                                                                                                                                            • Data Science
                                                                                                                                            Pricing
                                                                                                                                            • Pricing Overview
                                                                                                                                            • Pricing Calculator
                                                                                                                                            Integrations and Data
                                                                                                                                            • Marketplace
                                                                                                                                            • IDE Integrations
                                                                                                                                            • Partner Connect
                                                                                                                                            Solutions
                                                                                                                                            Databricks For Industries
                                                                                                                                            • Communications
                                                                                                                                            • Financial Services
                                                                                                                                            • Healthcare and Life Sciences
                                                                                                                                            • Manufacturing
                                                                                                                                            • Media and Entertainment
                                                                                                                                            • Public Sector
                                                                                                                                            • Retail
                                                                                                                                            • View All
                                                                                                                                            Cross Industry Solutions
                                                                                                                                            • Cybersecurity
                                                                                                                                            • Marketing
                                                                                                                                            Data Migration
                                                                                                                                            Professional Services
                                                                                                                                            Solution Accelerators
                                                                                                                                            Databricks For Industries
                                                                                                                                            • Communications
                                                                                                                                            • Financial Services
                                                                                                                                            • Healthcare and Life Sciences
                                                                                                                                            • Manufacturing
                                                                                                                                            • Media and Entertainment
                                                                                                                                            • Public Sector
                                                                                                                                            • Retail
                                                                                                                                            • View All
                                                                                                                                            Cross Industry Solutions
                                                                                                                                            • Cybersecurity
                                                                                                                                            • Marketing
                                                                                                                                            Resources
                                                                                                                                            Documentation
                                                                                                                                            Customer Support
                                                                                                                                            Community
                                                                                                                                            Training and Certification
                                                                                                                                            • Learning Overview
                                                                                                                                            • Training Overview
                                                                                                                                            • Certification
                                                                                                                                            • University Alliance
                                                                                                                                            • Databricks Academy Login
                                                                                                                                            Events
                                                                                                                                            • Data + AI Summit
                                                                                                                                            • Data + AI World Tour
                                                                                                                                            • Data Intelligence Days
                                                                                                                                            • Full Calendar
                                                                                                                                            Blog and Podcasts
                                                                                                                                            • Databricks Blog
                                                                                                                                            • Databricks Mosaic Research Blog
                                                                                                                                            • Data Brew Podcast
                                                                                                                                            • Champions of Data & AI Podcast
                                                                                                                                            Training and Certification
                                                                                                                                            • Learning Overview
                                                                                                                                            • Training Overview
                                                                                                                                            • Certification
                                                                                                                                            • University Alliance
                                                                                                                                            • Databricks Academy Login
                                                                                                                                            Events
                                                                                                                                            • Data + AI Summit
                                                                                                                                            • Data + AI World Tour
                                                                                                                                            • Data Intelligence Days
                                                                                                                                            • Full Calendar
                                                                                                                                            Blog and Podcasts
                                                                                                                                            • Databricks Blog
                                                                                                                                            • Databricks Mosaic Research Blog
                                                                                                                                            • Data Brew Podcast
                                                                                                                                            • Champions of Data & AI Podcast
                                                                                                                                            About
                                                                                                                                            Company
                                                                                                                                            • Who We Are
                                                                                                                                            • Our Team
                                                                                                                                            • Databricks Ventures
                                                                                                                                            • Contact Us
                                                                                                                                            Careers
                                                                                                                                            • Open Jobs
                                                                                                                                            • Working at Databricks
                                                                                                                                            Press
                                                                                                                                            • Awards and Recognition
                                                                                                                                            • Newsroom
                                                                                                                                            Security and Trust
                                                                                                                                            Company
                                                                                                                                            • Who We Are
                                                                                                                                            • Our Team
                                                                                                                                            • Databricks Ventures
                                                                                                                                            • Contact Us
                                                                                                                                            Careers
                                                                                                                                            • Open Jobs
                                                                                                                                            • Working at Databricks
                                                                                                                                            Press
                                                                                                                                            • Awards and Recognition
                                                                                                                                            • Newsroom
                                                                                                                                            databricks logo

                                                                                                                                            Databricks Inc.
                                                                                                                                            160 Spear Street, 15th Floor
                                                                                                                                            San Francisco, CA 94105
                                                                                                                                            1-866-330-0121

                                                                                                                                            See Careers
                                                                                                                                            at Databricks

                                                                                                                                            © Databricks 2025. All rights reserved. Apache, Apache Spark, Spark, the Spark Logo, Apache Iceberg, Iceberg, and the Apache Iceberg logo are trademarks of the Apache Software Foundation.

                                                                                                                                            • Privacy Notice
                                                                                                                                            • |Terms of Use
                                                                                                                                            • |Modern Slavery Statement
                                                                                                                                            • |California Privacy
                                                                                                                                            • |Your Privacy Choices