Building the Future of AI Agents and Intelligence Apps: Celebrating 4 years of Databricks Seattle R&D

In November 2021, we announced the opening of our Seattle R&D site and our plan to hire engineers to build out the Databricks Intelligence Platform. Today, we have hundreds of engineers in Bellevue and Seattle, working on mission-critical initiatives, ranging from infrastructure optimization to GenAI use cases to features that help our customers generate insights faster. 

All these efforts align with our goal at Databricks to simplify and democratize data and AI in service of enabling our customers to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform. 

On our four-year anniversary, we’re excited to share examples of the innovative work underway in our offices!  

AI-Powered Data Science and Analytics

Databricks continued to advance its AI-native data science and analytics experience, streamlining how users explore data, write code, and build data pipelines.

This year, the team released several major features, including:

• Lakeflow Designer: a new product experience to enable self-service business analytics through a low-code drag-and-drop interface. Designer is built from the ground up to be an AI-native experience, leveraging the complete data intelligence platform to provide accurate AI-generated responses. Everything in the visual workflow is represented with an underlying SQL file that can be stored in Git for CI/CD, version control, and collaboration across data teams.
• Data Science Agent in Databricks Assistant: a new autonomous workflow mode that transforms the Assistant from a conversational helper into a hands-on partner for data science. Users can ask the Agent to explore data, generate and run code, train and evaluate ML models and resolve errors.
• General Availability of the new SQL Editor: delivers a unified, modern authoring experience for SQL analysts with faster execution, real-time collaboration, split-screen editing, improved results visualization, and deep integration with the Databricks Assistant for writing SQL. 

Supported by the work of Seattle engineers Michael Piatek, Tomas Isdal, Weston Hutchins and Zhong Chen.

Democratized Intelligent Analytics

Databricks AI/BI provides a complete AI-powered BI experience. It combines rich dashboarding and reporting capabilities with Genie, a conversational interface that turns natural-language questions into insights.

Recent Major features:

• General Availability of Genie + Genie Research Agent: New ad hoc analysis via file upload, support for evaluation and benchmarks, and significant accuracy upgrades for high-quality responses. The team also released the Genie Research Agent, which provides deeper data insights and answers for complex business questions using multi-step reasoning and hypothesis investigation.
• Embedded Analytics: In many organizations, the most valuable analytics are the ones that need to be shared with customers, suppliers, or partners. Databricks customers can now take a dashboard that already exists in Databricks and place it directly inside a customer or partner-facing application. The experience is fully interactive and live, and consumption-based pricing means customers can scale analytics to thousands of viewers without incurring unpredictable fees.
•  You can read more about the team’s latest innovations in AI/BI here. 

Built with the help of Seattle engineers Kanit Wongsuphasawat, Justin Talbot, Miranda Luna, Amir Hormati, Yi Liu,  Alnur Ali, and Clark Wildenradt.

Data warehousing in the age of AI 

The Serverless Apache Spark Team, supports all of our Serverless Spark-based applications at Databricks. 

The team is focused on building a highly reliable platform capable of running millions of VMs a day, while also ensuring the workloads perform efficiently. Key projects include: 

• Leveraging historical usage to improve session binpacking on Serverless Spark clusters
• Providing best-in-class price-perf by deeply integrating with Spark to horizontally and vertically scale our Spark clusters based on users’ workloads
• Enabling low-latency provisioning, O(seconds), by analyzing demand and pre-warming compute accordingly
• Unblocking Serverless usage by removing feature divergence between Serverless and Classic (i.e. Budget Policies, Cost Controls, Instance Profiles, etc.)

Led by, engineers Mitchell Webster, Lev Novik, Akshay Singla, Swapandeep Singh, and Anwell Wang.

Open data sharing and collaboration 

The fundamental ingredient to AI is data. And increasingly, companies need to look externally to enrich and expand their data. 

Our Bellevue team has worked on Databricks’ core data sharing products, including Delta Sharing, Databricks Marketplace (built from the ground-up by Seattle-based engineers), and Databricks Cleanrooms. 

The impact is already visible in the real world. Listen to Mastercard talk about how Databricks Clean Rooms help them collaborate on sensitive data safely and at scale. 

Recent advancements have made open collaboration even more powerful:

• Delta Sharing innovations: Full iceberg interoperability, a new delta sharing network gateway that simplifies cross-organization connectivity, and fine-grained governance for shares using Attribute-Based Access Controls (ABAC). These improvements make it easier for providers to share governed data with a large number of recipients at scale.  
• Model and Agent Sharing - Providers can now publish MCP to the Databricks Marketplace, making it easy to discover and connect to MCP tools to accelerate AI development.
• Clean Rooms enhancements: Multi-party collaborations are now GA with advanced privacy approvals. Clean Rooms also integrate with leading identity partners to enable privacy-centric Identity Resolution. These new capabilities make clean rooms even more powerful for privacy-preserving collaboration

Seattle engineers  Mengxi Chen, Moe Derakhshani, Qihua Wang and Tao Tao have played a central role in building out these data sharing and collaboration capabilities. 

Optimizing performance and efficiency

Databricks can’t provide best-in-class products if they aren’t running on the world’s most performative and reliable infrastructure.  Here are some of the biggest infrastructure advancements built by our Seattle engineers:

• A highly customized, light-weight operating system that can boot VMs super fast
• A specialized container runtime that can warm up Spark in seconds
• A specialized container snapshotter that can fast fetch LLM weights
• A highly scalable container registry that distributes binaries at 10Tbps

Read more about our work:

• Booting Databricks VMs 7x Faster for Serverless Compute
• Downloading tens of millions of container images daily from the Serverless optimized Artifact Registry

Anders Liu, Max Wolffe, Shuo Chen, Shuai Chang led the Node Platform team that built container infrastructure for every Databricks product, helping to keep our machines secure, reliable, and improve efficiency across the fleet.

The Money Team: The Intersection of Business & Tech 

The Money Team is responsible for the systems that keep the financial heartbeat of Databricks pumping. This team is responsible for the end-to-end journey of turning Data + AI products into a sustainable business by evolving the supported business models, accelerating launch velocity, integrating acquisition,s and protecting against fraud and abuse. 

This team built the world’s only cross-cloud integrated rating engine, capable of processing trillions of usage events from both first-party and third-party partners — including AWS, Azure, GCP, and SAP — all unified under a single platform that operates in 85+ regions and works an order of magnitude faster than industry peers. This feat was made possible because they've developed their systems on top of Databricks' innovative Data + AI products, working closely with the R&D teams to drive new requirements that push the products forward.

In addition, the Money team has been central to the Databricks mission to democratize data + AI by allowing us to offer the industry’s only truly free trial - no credit card required. This is a powerful tool for students and developers who want to learn the latest technology but don’t have large budgets. We enable this with our cutting-edge admission control systems that safeguard Databricks products from abuse and unintended use. 

Read more about our work and team:

• How we drove down ops burden to allow for more investment in innovation
• Customer-facing cost management tools we’ve launched
• Our product management lead shares why he believes in Databricks 

The Money Team was built with the leadership of Seattle engineers Kazi Al-Rashid, Li Xiong, and Mahesh Venkataramani, with their Product Lead Greg Kroleski.

We’re growing!

We’re thrilled with the progress our Bellevue and Seattle engineering teams have made over the last four years! From AI/BI to the Money team, our teams here are building some of the most complex systems in the platform and driving several of our most strategic product initiatives. If you’re excited about solving hard problems at massive scale, we’re hiring here in Bellevue/Seattle and across our R&D locations. To learn more about available openings, visit our Careers page.