HomepageData + AI Summit 2023 Logo
JUNE 26-29, 2023
Attend Live

Enabling Advanced Analytics at The Department of State using Databricks

Wednesday, June 29 @4:45 PM


The Center for Analytics (CFA) is the State Department's first enterprise-wide capability to transform data into valuable insights to inform foreign policy and management decisions essential to the Department’s diplomatic mission. The Department leverages Azure Databricks to enable the processing and enrichment of high-volume datasets, sourced from operational applications, that feed downstream analytic products.

What makes Databricks so beneficial to CFA is the high-performance compute power in conjunction with the ease of integrating with existing Azure infrastructure and services. This includes the seamless integration with Azure Data Factory to build and automate robust data pipelines, the high bandwidth connection to double encrypted Azure Data Lake containers, and the elastic nature of cluster management to address variable-sized ETL jobs.

An example where Databricks plays a key role is the integration between the Department’s Electronic Records Systems that host over 2 billion documents and the analytic environment. Databricks serves as the ETL platform, enriches the data sets using open-source ML models, and then provides the derived data to various downstream applications.

Databricks also played a critical role in helping the Department respond to and monitor the global COVID-19 pandemic. Databricks’ scalable architecture enabled the CFA team to create and automate data pipelines across dozens of underlying data sources and environments at the Department. The output of these Databricks pipelines equipped senior Department leadership with regularly updated information on COVID-19 trends around the world.

Our lightning round discussion will focus on how Databricks addresses the needs of CFA and enables the Department to meet its diplomatic mission.


  • Session


  • Hybrid


  • Industry and Business Use Cases


  • Public Sector


  • Intermediate


  • Moscone South | Upper Mezzanine | 159


  • 35 min

Session Speakers

Headshot of Mark Lopez

Mark Lopez

Specialist Master


Headshot of Alan Gersch

Alan Gersch

Specialist Master


See the best of Data+AI Summit

Watch on demand