Washington, DC agencies frequently need each other's data to deliver resident services, but legacy silos made sharing slow, copy-heavy, and governance-intensive. As the District’s centralized IT provider, the Office of the Chief Technology Officer (OCTO) sits at the intersection of more than 80 DC agencies — making secure, scalable data access essential. By deploying on the Databricks Platform with Delta Sharing as an enterprise service, OCTO enabled secure, purpose-based cross-agency access without duplicating data, accelerating collaboration and cutting platform costs in half.
Challenge: District data silos slowed resident services
Delivering services in Washington, DC rarely stops at one agency. A single resident request can touch multiple teams — from public safety and transportation to education, housing, and health and human services — and those agencies need to share data to act quickly. OCTO serves as the District’s centralized IT provider, supporting around 80 DC agencies and everyone from business analysts to data scientists. In 2017, OCTO built what it called the DC Data Lake using Hadoop on bare-metal hardware and the Apache stack to bring together growing volumes of Internet of Things (IoT) and streaming data.
As sensors multiplied and data became more unstructured, that legacy platform became harder to maintain, harder to update, and harder for everyday users to work with. The pain peaked when agencies needed to share data. Even basic uploads required engineering effort, creating a bottleneck for Districtwide innovation. Cross-agency requests turned into one-off integrations and custom APIs; the more OCTO copied data around, the more governance slowed collaboration.
This lack of collaboration across and within District entities created inefficiencies, increasing the time for valuable analysis. For example, the Department of Buildings gathers data for inspection, permits and property status to provide insight into operations. Similarly, the Deputy Mayor of Education lacked a secure, unified mechanism to link longitudinal student records with workforce data. This gap made it nearly impossible to measure the long-term ROI of educational programs.
"There was no real way to just click and upload a document," said Matt Sokol, Chief Data Officer at OCTO. "We had to have someone script that process into place. And we had agencies that didn't want their data taken and kept somewhere else."
Solution: Delta Sharing turns data exchange into a handshake
OCTO deployed the Databricks Platform within the city's Microsoft Azure environment to reduce platform overhead and enable self-service across agencies. The shift from on-premises infrastructure to a managed cloud service eliminated the maintenance burden that had consumed the team's capacity. Unity Catalog added consistent governance and visibility so agencies could discover what data exists, understand it and request access with clear guardrails. OCTO established medallion architecture as a best practice, creating bronze, silver, and gold data layers that help analysts know exactly what stage of the curation process a dataset is at and the quality of data they are accessing.
With that foundation, Delta Sharing became the District’s data-sharing handshake. Delta Sharing addresses the fear of losing control as data owners can grant access for a specific purpose without relinquishing a copy, then revoke access when the work is done. Native connectors to other enterprise tools the District uses (e.g., Oracle, SQL Server, ServiceNow and Box) will eliminate the custom integration work that has created bottlenecks in the past.
Delta Sharing allows agencies to share data exactly where it lives instead of moving or duplicating it. The technical labor of manual transfers is no longer needed and the egress costs typically triggered when data leaves an environment are removed. Agencies can now scale collaboration with any number of partners without ever hitting "surprise" budget penalties.
"The technology is now there to click a button and share the data rather than spending hours building custom integrations and APIs," Matt said. "Delta Sharing has given agencies a different view. They're providing data for a specific purpose and then turning it off when they are done."
Results: Faster cross-agency decisions at half the cost
By shifting from bespoke integrations to governed sharing, OCTO is helping agencies move faster on work that directly impacts residents. For the Deputy Mayor for Education and its Office of Education through Employment Pathways (ETEP), Databricks is powering a records linkage system, using education records with workforce data to understand how residents progress from school into jobs and to evaluate which programs improve economic mobility. The project has brought together data from six different DC agencies and utilized Databricks to perform secure data ingestion. For the first time DC is now able to understand wage outcomes for more than 70,000 residents served by District education and workforce programs. OCTO is also centralizing its Amazon Connect call center data in Databricks, which gives agencies better visibility into KPIs and SLAs. District leadership can also view performance across agencies in a unified way.
Even when the outcome is not a publicly visible dashboard, the value is faster service delivery because agencies can access needed data sooner to support applications, processes and policy decisions. Just over a year into rollout, OCTO has onboarded 10 agencies – a rate that exceeds the adoption achieved over 7 years on the legacy system. Platform costs dropped by around 50 percent compared to previous licensing, even as usage grows. Deployment time fell from weeks to days; the Databricks installation only takes approximately one to two days once the preliminary Azure setup completes.
OCTO can now deliver curated datasets to any agency or external stakeholder, whether they are on the Databricks Platform or using another platform. H. This enhanced dataset delivery still maintains strict governance and zero data replication.
Looking ahead, OCTO is building on this modernization work to explore new, AI-driven capabilities for District staff. OCTO has begun exploring Agent Bricks to make District regulations and mayoral orders searchable through natural language. The proof of concept vectorizes regulatory documents and enables staff to ask questions and receive relevant, linked results rather than navigating rigid search tools. "Other states and cities have reached out about what we're doing with Delta Sharing," Matt said. "We’ve laid the foundation for Districtwide collaboration, and now we’re building on it with sharing, modernization, and AI to make government work faster."