Enabling Advanced Analytics at The Department of State using Databricks
What makes Databricks so beneficial to CFA is the high-performance compute power in conjunction with the ease of integrating with existing Azure infrastructure and services. This includes the seamless integration with Azure Data Factory to build and automate robust data pipelines, the high bandwidth connection to double encrypted Azure Data Lake containers, and the elastic nature of cluster management to address variable-sized ETL jobs.
An example where Databricks plays a key role is the integration between the Department’s Electronic Records Systems that host over 2 billion documents and the analytic environment. Databricks serves as the ETL platform, enriches the data sets using open-source ML models, and then provides the derived data to various downstream applications.
Databricks also played a critical role in helping the Department respond to and monitor the global COVID-19 pandemic. Databricks’ scalable architecture enabled the CFA team to create and automate data pipelines across dozens of underlying data sources and environments at the Department. The output of these Databricks pipelines equipped senior Department leadership with regularly updated information on COVID-19 trends around the world.
Our lightning round discussion will focus on how Databricks addresses the needs of CFA and enables the Department to meet its diplomatic mission.
- Industry and Business Use Cases
- Public Sector
- Moscone South | Upper Mezzanine | 159
- 35 min