Skip to main content

Strengthening Cyber Resilience through Efficient Data Management: A Response to M-21-31

Securely store and share data across public sector agencies and maintain regulatory compliance with Deloitte and Databricks
Brendan Barsness
Sujit Mohanty
Eric Popowich
Share this post

In today's environment, proactive cybersecurity is crucial to any public sector agency. For many organizations, log data that security professionals need for effective threat monitoring and incident response is not readily accessible in one place, or it lives in siloed departments. In some instances, the data may also be stored only for short-term operational purposes. This severely limits the ability to effectively manage security, and underscores the need for effective log retention as well as secure access to critical cyber information.

In 2021, the White House issued the OMB M-21-31 memorandum, mandating that federal agencies retain information systems logs over a multi-year period to support the detection, investigation, and remediation of cyber incidents. This creates multiple challenges for agencies to navigate. First, storing massive volumes of data for the extended duration required by M-21-31 can be costly, particularly if done in relatively high-cost on-premises or proprietary storage. Furthermore, transferring large volumes of data to a single monolithic repository to provide centralized access can also be expensive and result in data duplication across multiple environments. In short, the memorandum significantly increases data management and cybersecurity demands on federal organizations.

Lakehouse Hub

Deloitte's M-21-31 Cybersecurity solution looks to address these challenges by employing a hub-and-spoke model on the Databricks Data Intelligence Platform. A central analytics "Lakehouse Hub" coordinates with enterprise clouds and source systems, the "Nodes", to establish a centralized analytics layer for log data. Data is retained in low-cost cloud storage at the nodes and accessible by centralized queries from the hub, avoiding transfer of raw data across cloud boundaries. This multi-node, federated model allows data to be securely shared from individual nodes to the central hub, enabling comprehensive log access to address potential cyber threats more efficiently. This approach allows organizations to navigate the changing cyber landscape more effectively while avoiding costly data storage and egress.

M-21-31 Compliance

M-21-31 compliance requires that organizations not only collect an extensive list of system logs for an extended retention period, but also ensure comprehensive data visibility in order to support cybersecurity operations. The scale of M-21-31 log data volumes can make it technically and financially unsupportable for many organizations within their current toolbox.

Deloitte's M-21-31 Cybersecurity solution addresses these cost and scale challenges by leveraging low-cost cloud storage, reducing the need for expensive data indexing in proprietary systems. This is particularly impactful for high-volume telemetry data that is growing to petabyte scale.

The federated model provides centralized access and visibility to remote data distributed across the organization. Security operations center (SOC) analysts then have the opportunity to compile, search and perform advanced analytics on M-21-31 logs, enabling rapid response to cyber investigations that require significant historical data.

Efficient Data Management Across Clouds

The hub-and-spoke architecture manages large volume log data across multi-cloud environments by eliminating data duplication and reducing data egress transfer. The framework is a federation of Databricks workspaces that take advantage of a distributed medallion data pattern, incrementally increasing data quality at each node as data flows from raw to consumption-ready. Nodes are deployed at or near source systems as much as possible. Raw log data is ingested at the node, processed, and made available to be queried by the central hub. This eliminates costly data egress across clouds and regions by keeping the source log data at a single node. Only curated responses to federated queries by the hub are transferred from node to hub.

Efficient Data Management Across Clouds

Strong Central Governance

Ensuring the right users have the right access to log data is vital. By leveraging the Databricks governance framework, the hub defines and enforces access control rules that associate role-based user pools with collections of log datasets. In cases where more granular access management is needed, dynamic view functions can be constructed for row/column-level permissions or data masking.

Integration, Augmentation and Adoption

The Cyber Lakehouse integrates with common systems familiar to the organization's workforce, augmenting the existing toolset while maintaining continuity and accelerating adoption. This eliminates the need for additional training while leveraging the benefits of the Databricks Data Intelligence Platform. With the M-21-31 Cybersecurity solution, several use cases have been exercised such as:

  • BI tool dashboards populated with aggregated log data distributed across the enterprise and centrally accessible from the lakehouse hub.
  • SIEM tool queries pushed down to the lakehouse and returned results without requiring SIEM data ingestion and indexing.
  • Alerts detected while continuously monitoring at the nodes are pushed up to the BI or SIEM tool interface.

Why Deloitte and Databricks

The M-21-31 Cybersecurity Brickbuilder Solutions pairs the deep industry expertise of Deloitte with the Databricks Data Intelligence Platform. With Brickbuilder Solutions, you are guaranteed to get:

  • A Trusted Partner: Databricks is partnering with Deloitte to help you solve critical analytics challenges, reduce costs, and enhance productivity with as little friction as possible.
  • Credible Frameworks: The Deloitte team is certified on the Databricks Data Intelligence Platform to implement cybersecurity for your organization and provide the expertise needed to address your biggest data, analytics and AI needs.
  • Accelerated Value: Deloitte helps you to quickly unlock the full potential of the Databricks Data Intelligence Platform to boost productivity and extract value from data.

M 21-31 Cybersecurity by Deloitte is available now

Deloitte will be at the Databricks Government Forum on February 29. Come meet the team in person and see our M 21-31 Cybersecurity solution in action by registering here.

Try Databricks for free

Related posts

See all Platform Blog posts