Skip to main content
Login
      • Discover
        • For Executives
          • For Startups
            • Lakehouse Architecture
              • Mosaic Research
              • Customers
                • Customer Stories
                • Partners
                  • Cloud Providers
                    Databricks on AWS, Azure, GCP, and SAP
                    • Consulting & System Integrators
                      Experts to build, deploy and migrate to Databricks
                      • Technology Partners
                        Connect your existing tools to your Lakehouse
                        • C&SI Partner Program
                          Build, deploy or migrate to the Lakehouse
                          • Data Partners
                            Access the ecosystem of data consumers
                            • Partner Solutions
                              Find custom industry and migration solutions
                              • Built on Databricks
                                Build, market and grow your business
                              • Databricks Platform
                                • Platform Overview
                                  A unified platform for data, analytics and AI
                                  • Data Management
                                    Data reliability, security and performance
                                    • Sharing
                                      An open, secure, zero-copy sharing for all data
                                      • Data Warehousing
                                        Serverless data warehouse for SQL analytics
                                        • Governance
                                          Unified governance for all data, analytics and AI assets
                                          • Real-Time Analytics
                                            Real-time analytics, AI and applications made simple
                                            • Artificial Intelligence
                                              Build and deploy ML and GenAI applications
                                              • Data Engineering
                                                ETL and orchestration for batch and streaming data
                                                • Business Intelligence
                                                  Intelligent analytics for real-world data
                                                  • Data Science
                                                    Collaborative data science at scale
                                                  • Integrations and Data
                                                    • Marketplace
                                                      Open marketplace for data, analytics and AI
                                                      • IDE Integrations
                                                        Build on the Lakehouse in your favorite IDE
                                                        • Partner Connect
                                                          Discover and integrate with the Databricks ecosystem
                                                        • Pricing
                                                          • Databricks Pricing
                                                            Explore product pricing, DBUs and more
                                                            • Cost Calculator
                                                              Estimate your compute costs on any cloud
                                                            • Open Source
                                                              • Open Source Technologies
                                                                Learn more about the innovations behind the platform
                                                              • Databricks for Industries
                                                                • Communications
                                                                  • Media and Entertainment
                                                                    • Financial Services
                                                                      • Public Sector
                                                                        • Healthcare & Life Sciences
                                                                          • Retail
                                                                            • Manufacturing
                                                                              • See All Industries
                                                                              • Cross Industry Solutions
                                                                                • Cybersecurity
                                                                                  • Marketing
                                                                                  • Migration & Deployment
                                                                                    • Data Migration
                                                                                      • Professional Services
                                                                                      • Solution Accelerators
                                                                                        • Explore Accelerators
                                                                                          Move faster toward outcomes that matter
                                                                                        • Training and Certification
                                                                                          • Learning Overview
                                                                                            Hub for training, certification, events and more
                                                                                            • Training Overview
                                                                                              Discover curriculum tailored to your needs
                                                                                              • Databricks Academy
                                                                                                Sign in to the Databricks learning platform
                                                                                                • Certification
                                                                                                  Gain recognition and differentiation
                                                                                                  • University Alliance
                                                                                                    Want to teach Databricks? See how.
                                                                                                  • Events
                                                                                                    • Data + AI Summit
                                                                                                      • Data + AI World Tour
                                                                                                        • Data Intelligence Days
                                                                                                          • Event Calendar
                                                                                                          • Blog and Podcasts
                                                                                                            • Databricks Blog
                                                                                                              Explore news, product announcements, and more
                                                                                                              • Databricks Mosaic Research Blog
                                                                                                                Discover the latest in our Gen AI research
                                                                                                                • Data Brew Podcast
                                                                                                                  Let’s talk data!
                                                                                                                  • Champions of Data + AI Podcast
                                                                                                                    Insights from data leaders powering innovation
                                                                                                                  • Get Help
                                                                                                                    • Customer Support
                                                                                                                      • Documentation
                                                                                                                        • Community
                                                                                                                        • Dive Deep
                                                                                                                          • Resource Center
                                                                                                                            • Demo Center
                                                                                                                            • Company
                                                                                                                              • Who We Are
                                                                                                                                • Our Team
                                                                                                                                  • Databricks Ventures
                                                                                                                                    • Contact Us
                                                                                                                                    • Careers
                                                                                                                                      • Working at Databricks
                                                                                                                                        • Open Jobs
                                                                                                                                        • Press
                                                                                                                                          • Awards and Recognition
                                                                                                                                            • Newsroom
                                                                                                                                            • Security and Trust
                                                                                                                                              • Security and Trust
                                                                                                                                          • Data and AI summit

                                                                                                                                            JUNE 9–12 | SAN FRANCISCO

                                                                                                                                            Data + AI Summit is almost here — don’t miss the chance to join us in San Francisco!

                                                                                                                                            REGISTER
                                                                                                                                          • Ready to get started?
                                                                                                                                          • Get a Demo
                                                                                                                                          Data and AI summit

                                                                                                                                          JUNE 9–12 | SAN FRANCISCO

                                                                                                                                          Data + AI Summit is almost here — don’t miss the chance to join us in San Francisco!

                                                                                                                                          REGISTER
                                                                                                                                          • Login
                                                                                                                                          • Try Databricks
                                                                                                                                          1. Blog
                                                                                                                                          2. /
                                                                                                                                            Product
                                                                                                                                          3. /
                                                                                                                                            Article

                                                                                                                                          Announcing Storage-Optimized Endpoints for Vector Search

                                                                                                                                          Multi-billion Vector Scale, 7x lower cost

                                                                                                                                          Announcing Storage-Optimized Mosaic AI Vector Search

                                                                                                                                          Published: June 6, 2025

                                                                                                                                          Product4 min read

                                                                                                                                          by Adam Gurary and Dima Kotlyarov

                                                                                                                                          Share this post

                                                                                                                                          Keep up with us

                                                                                                                                          Summary

                                                                                                                                          • Introducing storage-optimized Vector Search: Billion vector scale, up to 7x lower cost, 20x faster indexing, familiar SQL-like filtering.
                                                                                                                                          • Unlock more value from unstructured data for AI: Build high-performance RAG, entity resolution, and semantic search systems on documents, images, and more.
                                                                                                                                          • Enterprise-ready and easy to adopt: Backed by Unity Catalog governance and integrated with tools like AI Playground for rapid RAG prototyping and budget policies for cost management.

                                                                                                                                          Most enterprises sit on a massive amount of unstructured data—documents, images, audio, video—yet only a fraction ever turns into actionable insight. AI-powered apps such as retrieval‑augmented generation (RAG), entity resolution, recommendation engines, and intent‑aware search can change that, but they quickly run into familiar barriers: hard capacity limits, ballooning costs, and sluggish indexing.

                                                                                                                                          Today, we are announcing the Public Preview of storage-optimized endpoints for Mosaic AI Vector Search—our new Vector Search engine, purpose‑built for petabyte‑scale data. By decoupling storage from compute and leveraging Spark's massive scale and parallelism inside the Databricks Data Intelligence Platform, it delivers:

                                                                                                                                          • Multi-billion vector capacity
                                                                                                                                          • Up to 7x lower cost
                                                                                                                                          • 20x faster indexing
                                                                                                                                          • SQL‑style filtering

                                                                                                                                          Best of all, it’s a true drop‑in replacement for the same APIs your teams already use, now super‑charged for RAG, semantic search, and entity resolution in real‑world production. Additionally, to further support enterprise teams, we’re also introducing new features designed to streamline development and improve cost visibility.

                                                                                                                                          What’s new in storage-optimized Vector Search

                                                                                                                                          Storage-optimized endpoints were built in direct response to what enterprise teams told us they need most: the ability to index and search across entire unstructured data lakes, infrastructure that scales without ballooning costs, and faster development cycles.

                                                                                                                                          Multi-billion Vector Scale, 7x lower cost

                                                                                                                                          Scale is no longer a limitation. Where our Standard offering supported a few hundred million vectors, storage optimized is built for billions of vectors at a reasonable cost, allowing organizations to run full-data-lake workloads without the need to sample or filter down. Customers running large workloads are seeing up to 7x lower infrastructure costs, making it finally feasible to run GenAI in production across massive unstructured datasets.

                                                                                                                                          For comparison, storage optimized pricing would be ~$900/month for 45M vectors and ~$7K/month for 1.3B vectors. The latter represents significant savings compared to ~$47K/month on our standard offering.

                                                                                                                                          Up to 20x Faster Indexing

                                                                                                                                          Unlock rapid iteration cycles that were previously impossible. Our re-architecture powers one of the most requested improvements—dramatically faster indexing. You can now build a 1 billion vector index in under 8 hours, and smaller indices of 100M vectors or smaller are built in minutes. 

                                                                                                                                          “The indexing speed improvement with storage-optimized is huge for us. What previously took about 7 hours now takes just one hour, a 7-8x improvement." 
                                                                                                                                          — Ritabrata Moitra, Sr. Lead ML Engineer, CommercelIQ

                                                                                                                                          SQL-like Filtering

                                                                                                                                          Easily filter records without learning unfamiliar syntax. Beyond performance and scale, we’ve also focused on usability. Metadata filtering is now done using intuitive, SQL-style syntax, making it simple to narrow down search results using criteria you’re already familiar with.

                                                                                                                                          Same APIs, Brand New Backend

                                                                                                                                          Migrating to storage-optimized endpoints is easy—just select it when creating a new endpoint, and create a new index on your table. The similarity search API remains the same, so there is no need for major code changes. 

                                                                                                                                          “We see storage-optimized Vector Search as essentially a drop-in replacement for the standard offering. It unlocks the scale we need to support hundreds of internal investors querying tens of millions of documents daily, without compromising on latency or quality.”
                                                                                                                                          — Alexandre Poulain, Director, Data Science & AI Team, PSP Investments

                                                                                                                                          Because this capability is part of the Mosaic AI platform, it comes with full governance powered by Unity Catalog. That means proper access controls, audit trails, and lineage tracking across all your Vector Search assets—ensuring compliance with enterprise data and security policies from day one.

                                                                                                                                          Enhanced Features to Streamline Your Workflow

                                                                                                                                          To further support enterprise teams, we’re introducing new capabilities that make it easier to experiment, deploy, and manage Vector Search workloads at scale.

                                                                                                                                          Teams can now test and deploy a chat agent backed by a Vector Search index as a knowledge base in two clicks – a process that used to require significant custom code. With direct integration in the Agent Playground now in Public Preview, select your Vector Search index as a tool, test your RAG agent, and export, deploy, and evaluate agents without writing a single line of code. This dramatically shortens the path from prototype to production.

                                                                                                                                          Our improved cost visibility with endpoint budget policy tagging allows platform owners and FinOps teams to easily track and understand spend across multiple teams and use cases, allocate budgets, and manage costs as usage grows. More support for tagging indices and compute resources is coming soon.

                                                                                                                                          This Is Just the Beginning

                                                                                                                                          The release of storage-optimized endpoints is a major milestone, but we're already working on future enhancements:

                                                                                                                                          • Scale-to-Zero: Automatically scale compute resources down when not in use to further reduce costs
                                                                                                                                          • High QPS Support: Infrastructure to handle thousands of queries per second for demanding real-time applications
                                                                                                                                          • Beyond Semantic Search: Efficient non-semantic retrieval capabilities for keyword-only workloads.

                                                                                                                                          Our goal is simple: build the best vector search technology available, fully integrated with the Databricks Data Intelligence Platform you already rely on.

                                                                                                                                          Start Building Today

                                                                                                                                          Storage-optimized endpoints transform how you work with unstructured data at scale. With massive capacity, better economics, faster indexing, and familiar filtering, you can confidently build more powerful AI applications.

                                                                                                                                          Ready to get started?

                                                                                                                                          • Try Mosaic AI Vector Search for free: express setup gives you instant access and free serverless credits.
                                                                                                                                          • Check out our documentation to see how it’s done!

                                                                                                                                          Keep up with us

                                                                                                                                          Recommended for you

                                                                                                                                          Share this post

                                                                                                                                          Never miss a Databricks post

                                                                                                                                          Subscribe to the categories you care about and get the latest posts delivered to your inbox

                                                                                                                                          Sign up

                                                                                                                                          What's next?

                                                                                                                                          Introducing Predictive Optimization for Statistics

                                                                                                                                          Product

                                                                                                                                          November 20, 2024/4 min read

                                                                                                                                          Introducing Predictive Optimization for Statistics

                                                                                                                                          How to present and share your Notebook insights in AI/BI Dashboards

                                                                                                                                          Product

                                                                                                                                          November 21, 2024/3 min read

                                                                                                                                          How to present and share your Notebook insights in AI/BI Dashboards

                                                                                                                                          databricks logo
                                                                                                                                          Why Databricks
                                                                                                                                          Discover
                                                                                                                                          • For Executives
                                                                                                                                          • For Startups
                                                                                                                                          • Lakehouse Architecture
                                                                                                                                          • Mosaic Research
                                                                                                                                          Customers
                                                                                                                                          • Customer Stories
                                                                                                                                          Partners
                                                                                                                                          • Cloud Providers
                                                                                                                                          • Technology Partners
                                                                                                                                          • Data Partners
                                                                                                                                          • Built on Databricks
                                                                                                                                          • Consulting & System Integrators
                                                                                                                                          • C&SI Partner Program
                                                                                                                                          • Partner Solutions
                                                                                                                                          Discover
                                                                                                                                          • For Executives
                                                                                                                                          • For Startups
                                                                                                                                          • Lakehouse Architecture
                                                                                                                                          • Mosaic Research
                                                                                                                                          Customers
                                                                                                                                          • Customer Stories
                                                                                                                                          Partners
                                                                                                                                          • Cloud Providers
                                                                                                                                          • Technology Partners
                                                                                                                                          • Data Partners
                                                                                                                                          • Built on Databricks
                                                                                                                                          • Consulting & System Integrators
                                                                                                                                          • C&SI Partner Program
                                                                                                                                          • Partner Solutions
                                                                                                                                          Product
                                                                                                                                          Databricks Platform
                                                                                                                                          • Platform Overview
                                                                                                                                          • Sharing
                                                                                                                                          • Governance
                                                                                                                                          • Artificial Intelligence
                                                                                                                                          • Business Intelligence
                                                                                                                                          • Data Management
                                                                                                                                          • Data Warehousing
                                                                                                                                          • Real-Time Analytics
                                                                                                                                          • Data Engineering
                                                                                                                                          • Data Science
                                                                                                                                          Pricing
                                                                                                                                          • Pricing Overview
                                                                                                                                          • Pricing Calculator
                                                                                                                                          Open Source
                                                                                                                                          Integrations and Data
                                                                                                                                          • Marketplace
                                                                                                                                          • IDE Integrations
                                                                                                                                          • Partner Connect
                                                                                                                                          Databricks Platform
                                                                                                                                          • Platform Overview
                                                                                                                                          • Sharing
                                                                                                                                          • Governance
                                                                                                                                          • Artificial Intelligence
                                                                                                                                          • Business Intelligence
                                                                                                                                          • Data Management
                                                                                                                                          • Data Warehousing
                                                                                                                                          • Real-Time Analytics
                                                                                                                                          • Data Engineering
                                                                                                                                          • Data Science
                                                                                                                                          Pricing
                                                                                                                                          • Pricing Overview
                                                                                                                                          • Pricing Calculator
                                                                                                                                          Integrations and Data
                                                                                                                                          • Marketplace
                                                                                                                                          • IDE Integrations
                                                                                                                                          • Partner Connect
                                                                                                                                          Solutions
                                                                                                                                          Databricks For Industries
                                                                                                                                          • Communications
                                                                                                                                          • Financial Services
                                                                                                                                          • Healthcare and Life Sciences
                                                                                                                                          • Manufacturing
                                                                                                                                          • Media and Entertainment
                                                                                                                                          • Public Sector
                                                                                                                                          • Retail
                                                                                                                                          • View All
                                                                                                                                          Cross Industry Solutions
                                                                                                                                          • Cybersecurity
                                                                                                                                          • Marketing
                                                                                                                                          Data Migration
                                                                                                                                          Professional Services
                                                                                                                                          Solution Accelerators
                                                                                                                                          Databricks For Industries
                                                                                                                                          • Communications
                                                                                                                                          • Financial Services
                                                                                                                                          • Healthcare and Life Sciences
                                                                                                                                          • Manufacturing
                                                                                                                                          • Media and Entertainment
                                                                                                                                          • Public Sector
                                                                                                                                          • Retail
                                                                                                                                          • View All
                                                                                                                                          Cross Industry Solutions
                                                                                                                                          • Cybersecurity
                                                                                                                                          • Marketing
                                                                                                                                          Resources
                                                                                                                                          Documentation
                                                                                                                                          Customer Support
                                                                                                                                          Community
                                                                                                                                          Training and Certification
                                                                                                                                          • Learning Overview
                                                                                                                                          • Training Overview
                                                                                                                                          • Certification
                                                                                                                                          • University Alliance
                                                                                                                                          • Databricks Academy Login
                                                                                                                                          Events
                                                                                                                                          • Data + AI Summit
                                                                                                                                          • Data + AI World Tour
                                                                                                                                          • Data Intelligence Days
                                                                                                                                          • Event Calendar
                                                                                                                                          Blog and Podcasts
                                                                                                                                          • Databricks Blog
                                                                                                                                          • Databricks Mosaic Research Blog
                                                                                                                                          • Data Brew Podcast
                                                                                                                                          • Champions of Data & AI Podcast
                                                                                                                                          Training and Certification
                                                                                                                                          • Learning Overview
                                                                                                                                          • Training Overview
                                                                                                                                          • Certification
                                                                                                                                          • University Alliance
                                                                                                                                          • Databricks Academy Login
                                                                                                                                          Events
                                                                                                                                          • Data + AI Summit
                                                                                                                                          • Data + AI World Tour
                                                                                                                                          • Data Intelligence Days
                                                                                                                                          • Event Calendar
                                                                                                                                          Blog and Podcasts
                                                                                                                                          • Databricks Blog
                                                                                                                                          • Databricks Mosaic Research Blog
                                                                                                                                          • Data Brew Podcast
                                                                                                                                          • Champions of Data & AI Podcast
                                                                                                                                          About
                                                                                                                                          Company
                                                                                                                                          • Who We Are
                                                                                                                                          • Our Team
                                                                                                                                          • Databricks Ventures
                                                                                                                                          • Contact Us
                                                                                                                                          Careers
                                                                                                                                          • Open Jobs
                                                                                                                                          • Working at Databricks
                                                                                                                                          Press
                                                                                                                                          • Awards and Recognition
                                                                                                                                          • Newsroom
                                                                                                                                          Security and Trust
                                                                                                                                          Company
                                                                                                                                          • Who We Are
                                                                                                                                          • Our Team
                                                                                                                                          • Databricks Ventures
                                                                                                                                          • Contact Us
                                                                                                                                          Careers
                                                                                                                                          • Open Jobs
                                                                                                                                          • Working at Databricks
                                                                                                                                          Press
                                                                                                                                          • Awards and Recognition
                                                                                                                                          • Newsroom
                                                                                                                                          databricks logo

                                                                                                                                          Databricks Inc.
                                                                                                                                          160 Spear Street, 15th Floor
                                                                                                                                          San Francisco, CA 94105
                                                                                                                                          1-866-330-0121

                                                                                                                                          See Careers
                                                                                                                                          at Databricks

                                                                                                                                          © Databricks 2025. All rights reserved. Apache, Apache Spark, Spark, the Spark Logo, Apache Iceberg, Iceberg, and the Apache Iceberg logo are trademarks of the Apache Software Foundation.

                                                                                                                                          • Privacy Notice
                                                                                                                                          • |Terms of Use
                                                                                                                                          • |Modern Slavery Statement
                                                                                                                                          • |California Privacy
                                                                                                                                          • |Your Privacy Choices