Data Mesh and Compliance in a Multi-Regional Data Lake at Atlassian
OVERVIEW
EXPERIENCE | In Person |
---|---|
TYPE | Breakout |
TRACK | Data Governance |
INDUSTRY | Enterprise Technology |
TECHNOLOGIES | Data Sharing, Apache Spark, Delta Lake |
SKILL LEVEL | Intermediate |
DURATION | 40 min |
DOWNLOAD SESSION SLIDES |
This session will explore how we are in the journey of implementing the Data Mesh concept at Atlassian. We'll examine the contrasts between logical and physical replication, articulating their implications and trade-offs. We will also discuss a product-agnostic replication protocol used for logical replication and how we apply schema enforcement using Data Contracts. Next, we'll explore the ramifications of empowering our Cloud customers to select the AWS Region for storing their product data, leading to a deployment of a multi-region data lake. We'll also address our strategies for regulatory compliance, including GDPR and FedRAMP, as well as sophisticated enterprise functionalities such as BYOK (Bring Your Own Key). Lastly, we'll showcase how the data lake serves as the backbone for various use cases, including Atlassian Analytics, in-product data experiences, ML training, and data exports leveraging Delta Sharing, ultimately unlocking the potential of every team.
SESSION SPEAKERS
Mike Dias
/Principal Data Platform Engineer
Atlassian
Paul Ashley
/Principal Software Engineer
Atlassian