SESSION

Architectural Overview of Atlassian's Next-Generation Data Lakehouse

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Lakehouse Architecture
INDUSTRYEnterprise Technology
TECHNOLOGIESDeveloper Experience, Governance, Orchestration
SKILL LEVELBeginner
DURATION40 min

We are rebuilding our data lakehouse from the ground up - a greenfield re-design based on everything we have learned from the last 5 years of working with Databricks. In this talk, we’ll give an architectural overview of how we’re setting up this new lake, covering topics like:

 

  • How we are laying out our Databricks accounts/workspaces and AWS accounts to create the concept of “environments”
  • Our “workbench environment” concept that separates insights work from production pipelines
  • The benefit we get from doubling down on Unity Catalog, Delta Lake, and Managed Tables
  • Creation and enforcement of a consistent information architecture
  • Making our data lakehouse declarative, and keeping human users out of the production environment
  • Supporting and governing machine learning workloads

SESSION SPEAKERS

Perry Stephenson

/Principal Data Platform Engineer
Atlassian

Chen Zhou

/Senior Data Platform Engineer
Atlassian