Databricks SQL Year in Review (Part III): User Experience

This blog continues our series looking at advancements from 2023 to the serverless data warehouse Databricks SQL. The best data warehouse is a lakehouse, which is why Databricks SQL (DBSQL) is built on the lakehouse architecture as part of our Data Intelligence Platform. In part 3 of our series, let's take a closer look at the many enhancements to the user experience for DBSQL.

We know that a simple and streamlined user experience is critical to user productivity and customer success. This past year marked the introduction of the AI-powered Databricks Assistant, simplifying the user experience for Databricks SQL with the power of natural language. We also launched a brand new dashboarding tool called Lakeview. We'll explore 5 of the areas that showcase the latest enhancements to the DBSQL user experience in more detail below.

• AI Assistant and SQL Editor for writing SQL
• AI Assistant for the new Lakeview dashboards
• AI-Generated documentation for your data
• ERDs for understanding data and relationships
• Workflows and monitoring for managing workloads

AI Assistant and SQL Editor for Writing SQL

We introduced the Databricks Assistant, a context-aware AI assistant that improves analyst and developer productivity, to the SQL Editor. The Databricks Assistant leverages Unity Catalog metadata and generative AI to provide personalized responses to user prompts and helps users write, explain, and debug queries in English rather than SQL. Simply enter a question or statement as a prompt and watch the SQL generate!

For those writing SQL themselves, autocomplete suggestions in the SQL Editor saw a number of improvements. To enable faster query writing, SQL developers now receive suggestions for column completions in CTEs and join recommendations on commonly joined tables. The autocomplete is also more thoughtful and better tuned about when it provides recommendations.

In addition to improving the query authoring experience, we shipped a totally new results grid. This new grid showcases improved data type icons, search and filtering over displayed results, and overhauled selection mechanics that make it even easier to work with tabular outputs.

Finally, for users that prefer the Notebooks coding paradigm instead of the SQL Editor, Notebooks can now connect to SQL Warehouse endpoints. Query execution statistics and query profiles are available directly within Notebooks, allowing users who write SQL to diagnose query performance issues in the context of their interactive development experience.

AI Assistant for the New Lakeview Dashboards

In September 2023, we announced the Public Preview of Lakeview dashboards. Built on a brand new visualization engine, Lakeview dashboards offer beautiful, interactive charts that render up to 10x faster. These dashboards are easy to manage and can be securely shared with anyone in your organization.

We then introduced the Databricks Assistant for Lakeview with AI-assisted authoring capabilities to easily create Lakeview visualizations using natural language. Simply enter a question or command as the prompt and wait for Databricks Assistant to generate a response with a new chart!

Dashboards can be securely shared by simply clicking the "Publish" link in the Share dropdown. Under the hood, a streamlined information architecture now bundles the underlying datasets and widgets together, so there's no need to worry about permissions getting out of sync across queries.

Share the dashboard with users or groups, or anyone in your organization who may not have their own Databricks account. Use the permission settings to control their level of access such as view, run or edit the dashboard. If you edit the dashboard, the published dashboard remains in place until you decide to publish a new version. This makes it flexible for you to work on changes while others are using the existing version.

Finally, Lakeview dashboards showcase built-in lineage from Unity Catalog. This makes it easy to trace data back to source systems and helps your consumers trust where the dashboard insights originate.

Read more about Lakeview dashboards here and get started today!

AI-Generated Documentation for Your Data

Documenting your data is hard. In 2023, we introduced AI-generated documentation to help data owners reduce the burden of manually creating descriptions for their data. When a user navigates to a Unity Catalog table they own (or have the MODIFY permission), this AI-driven feature will suggest table- and column-level comments based on the table metadata. Users then have the option to accept, reject, or edit these comments. Learn more about how this feature was built.

ERDs for Understanding Data and Relationships

Entity Relationship Diagrams are a new addition that make it easier than ever to find and understand relationships between tables. These diagrams visualize primary key and foreign key relationships between tables in a graph, providing a clear and intuitive representation of how data entities connect. Easily understand what tables you might want to join, or what other tables exist that might have relevant data to your analytical questions.

We also improved users' ability to browse changes and track down updates to tables by improving the History tab experience for tables in Catalog Explorer. The History tab now offers filtering and column sorting, allowing you to easily identify important related content. If users wish to dive in further, they can click on new links to relevant notebooks and jobs to easily trace changes. These additions will make it easier to efficiently track down changes to your data.

Workflows and Monitoring for Managing Workloads

Databricks SQL is part of the Databricks Intelligence Platform with capabilities for all your data and AI workloads. In 2023, we delivered integrations across the platform that made managing workloads for your data warehouse even easier.

Databricks SQL is now integrated with Databricks Workflows, a fully-managed orchestration service that lets you easily define, manage and monitor multi-task workflows. This integration allows you to add your DBSQL queries, dashboards, and alerts to your Workflow jobs for scheduling! You can also view details of past job runs in Workflows and use Unity Catalog's lineage feature to identify upstream pipelines that your SQL assets depend on.

We also introduced a warehouse monitoring page that gives administrators real-time access to critical metrics such as active, running, and queued queries, along with active clusters. Intuitive charts now showcase peak times for query queuing and execution in an easy-to-understand visual, enabling more precise warehouse administration. Query history has also been integrated into this page so that you can easily see the list of queries executed on just that warehouse. For each query, you get an overview of the main execution statistics, such as bytes read or rows returned. You can also access the query profile for details about the various operators involved in the execution of the query, which can help you diagnose and improve query performance.

Additionally, we introduced a system table for Warehouse Events that provides detailed insights into warehouse operations. This table acts as a comprehensive log of that, detailing starts/stops and scale up/down events, which is critical for understanding uptime and associated costs. Users can set up custom alerting capabilities on top of these warehouses to proactively identify issues such as warehouses running outside of normal business hours.

Conclusion

2023 was a big year for Databricks SQL with a focus on elevating the user experience for our data warehouse with AI-powered assistants, brand new dashboarding visualizations and a host of new features that make it easy for users to understand data, author queries and inspect performance.

To learn more about AI-powered data warehousing and see demos of the AI assistants in action with Databricks SQL, check out our recent webinar.

If you want to migrate your SQL workloads to a high-performance, serverless data warehouse with a great user experience, then Databricks SQL is the solution -- try it for free.