In the past, stream processing over data lakes required a lot of development efforts from data engineering teams, as Itai has shown in his talk at Spark+AI Summit 2019 (https://tinyurl.com/2s3az5td).
Today, with Delta Lake and Databricks Auto Loader, this becomes a few minutes' work!
Not only that, it unlocks a new set of ways to efficiently leverage your data.

Nexar, a leadi...

Every industry must comply with some form of compliance or data security in order to operate. As data becomes more mission critical to the organization, so does the need to protect and secure it.

Public Sector organizations are responsible for securing sensitive data sets and complying with regulatory programs such as HIPAA, FedRAMP, and StateRAMP.

This does not come as a su...

The California Healthcare Eligibility, Enrollment, and Retention System (CalHEERS)—one of the largest State-based health exchanges in the country—was looking to modernize their data warehouse (DWH) environment to support the vision that every decision to design, implement and evaluate their state-based health exchange portal was informed by timely and rigorous evidence about its consumers’ experience. Th...

Join our capstone event for Public Sector attendees at Summit featuring keynotes and panel discussions with the Centers for Disease Control (CDC), U.S. Department of Veterans Affairs (VA), United States Special Operations Command (SOCOM), DoD Advana and more. Following the program, we will host a dessert and coffee networking reception with Public Sector product demonstrations by Databricks and our leadi...

This talk is a case study in migrating on-premises data pipelines to a cloud environment. In our instance, we wished to rebuild a pipeline which included both streaming and batch components. The streaming component performed field extractions and stored the resulting raw data to HDFS, where a Spark job picked it up and applied aggregations for later use by analysts.

We were able to replica...

Many federal agencies use SAS software for critical operational data processes (ETL). While SAS has historically been a leader in analytics, it has often been used by data analysts for ETL purposes as well. However, modern data science demands on ever-increasing volumes and types of data require a shift to modern, cloud architectures and data management tools and paradigms for ETL/ELT. In this presentati...

Unity Catalog unifies governance and security for Databricks in one place. It can store data classifications and privileges and enforce them.



This talk will go into the details of Unity Catalog and explains the core building blocks in Unity Catalog for Security and Governance. I will also explain how Privacera translates Apache Ranger policies into native policies of Unity ...

The Department of Veterans Affairs (VA) is home to over 420,000 employees, provides health care for 9.16 million enrollees and manages the benefits of 5.75 million recipients. The VA also hosts an array of financial management, professional, and administrative services at their Financial Service Center (FSC), located in Austin, Texas. The FSC is divided into various service groups organized around revenu...

Data security and data quality are significant challenges facing federal agencies as they apply AI, ML, and data analytics to achieve their mission. These large agencies all have multiple data-driven initiatives being implemented or considered across functional domains. They range from personnel operations, acquisition efforts, and agency-specific missions for both national security to humanitarian aid. ...

Adversarial AI/ML is an emerging research area focused on the vulnerabilities of Artificial Intelligence (AI)/Machine Learning (ML) models to adversarial exploitation such as data poisoning, adversarial perturbations, inference, and extraction attacks. This research area is of particular interest to domains where AI/ML models play an essential role in the mission-critical decision-making processes. In th...

Los Angeles County’s Department of Human Resources (DHR) is responsible for attracting a diverse workforce for the 37 departments it supports. Each year, DHR processes upwards of 400,000 applications for job opportunities making it one of the largest employers in the nation. Managing a hiring process of this scale is complex with many complicated factors such as background checks and skills examination. ...

The US Air Force VAULT platform is a cloud-native enterprise data platform designed to provide the Department of the Air Force (DAF) with a robust, interoperable, and secure data environment. The strategic goals of VAULT include:

- Leading Data Culture - Increase data use and literacy to improve efficiency and effectiveness of decisions, readiness, mission operations, and cybersecurity.