Skip to main content
Platform blog

Introducing LakehouseIQ: The AI-Powered Engine that Uniquely Understands Your Business

Share this post

Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural language access to it for a wide range of use cases. Any employee in your organization can use LakehouseIQ to search, understand, and query data in natural language. LakehouseIQ uses information about your data, usage patterns, and org chart to understand your business’s jargon and unique data environment, and give significantly better answers than naive use of Large Language Models (LLMs).

Large Language Models have, of course, promised to bring language interfaces to data, and every data company is adding an AI assistant, but in reality, many of these solutions fall short on enterprise data. Every enterprise has unique datasets, jargon, and internal knowledge that is required to answer its business questions, and simply calling an LLM trained on the Internet to answer questions gives wrong results. Even something as simple as the definition of a “customer” or the fiscal year varies across companies.

LakehouseIQ is a first-of-its-kind knowledge engine that directly solves this problem by automatically learning about business and data concepts in your enterprise. It uses signals from across the Databricks Lakehouse platform, including Unity Catalog, dashboards, notebooks, data pipelines, and docs, leveraging the unique end-to-end nature of the Databricks platform to see how data is used in practice. This lets LakehouseIQ build highly accurate specialized models for your enterprise.

LakehouseIQ

We are using LakehouseIQ to power a spectrum of new natural language interfaces throughout Databricks, from queries to troubleshooting. And even more importantly, we are exposing its functionality through APIs to let customers build their own AI apps that use this automatically trained knowledge. We believe that this kind of knowledge engine for the enterprise will become a vital component of the next-generation software stack.

Natural Language Queries

The first AI surface most Databricks users will see is the new Assistant in our SQL Editor and Notebooks that can write queries, explain them, and answer questions. It is already saving our users hundreds of hours of time. The Assistant relies heavily on LakehouseIQ to find and understand the right data for each activity and give accurate answers. Without a knowledge engine like LakehouseIQ, LLMs often cannot know how data is used in your enterprise – for example, in the query below, our Assistant with LakehouseIQ turned off searches for a sales territory called “Europe” and finds no results, because it does not know that the company actually has two European territories, North and South. The LakehouseIQ version not only knows this information, but automatically adds a filter to exclude internal usage, learned from other queries, dashboards and notebooks that used this dataset.

Assistant without LakehouseIQAssistant with LakehouseIQ

Search with LakehouseIQ

LakehouseIQ also significantly enhances Databricks’ in-product Search. Our new search engine doesn't just find data, it interprets, aligns and presents it in an actionable, contextual format, helping all users get started with their data faster. In this example on some of our internal data, LakehouseIQ understands that at Databricks, the codename for serverless is “Nephos”, and that “DBUs” are a measure of usage, thus finding the right result. It also exposes signals on popularity, freshness, and frequent users for each table.

Search without LakehouseIQSearch with LakehouseIQ

Management and Troubleshooting

We are also integrating LakehouseIQ into many of the management workflows in the Lakehouse. For example, providing meaningful comments on datasets gets easier with automatic suggestions – and the more documentation you add, the better LakehouseIQ will be able to use that data. LakehouseIQ can also understand and debug jobs, data pipelines, and Spark and SQL queries (e.g., tell you that a dataset may be incomplete because an upstream job is failing), helping users figure out when something is wrong.

Metadata suggestions with LakehouseIQ

Governance and Security

LakehouseIQ is built on and governed by Unity Catalog, Databricks’ flagship solution for security and governance across data and AI. When using LakehouseIQ, your users will only see results for datasets they have access to in Unity Catalog, so you can open data analysis to more users without worrying about new security headaches. Coupled with other functionality we are announcing today, including AI-based automatic data classification, monitoring, and Lakehouse Federation to external systems, LakehouseIQ helps democratize all data in your enterprise.

Next Steps

We believe that LakehouseIQ is the dawn of an unprecedented era of data democratization. By harnessing LakehouseIQ's sophisticated language capabilities and deep contextual comprehension, Databricks offers substantial insights over any source of data in an engaging conversational format, revolutionizing the way we interact with data. We're not just making data accessible; we're making it intelligible, actionable, and much more valuable. We will be rolling out various LakehouseIQ features throughout the year, and are excited to get your feedback.

 

Don't miss the opportunity to witness LakehouseIQ in action at our Data + AI Summit.

Try Databricks for free

Related posts

Engineering blog

Introducing Databricks Vector Search Public Preview

Following the announcement we made yesterday around Retrieval Augmented Generation (RAG), today, we’re excited to announce the public preview of Databricks Vector Search. W...
Company blog

Enterprise Cloud Service Public Preview on AWS

June 12, 2020 by Vinay Wagh and Abhinav Garg in Company Blog
At Databricks, we have had the opportunity to collaborate with companies that have transformed the way people live. Some of our customers have...
See all Platform Blog posts