Ensuring GDPR Compliance for Real-Time Data Pipelines at Conde Nast

This session details Conde Nast's use of Databricks tools in real-time data pipelines for privacy compliance. Our journey started with a real-time customer subscription data pipeline, and we aim to extend these practices. We'll share strategies for immediate and long-term data privacy and compliance. In our short-term approach, we utilize Delta Lake's Change Data Feed (CDF) for efficient data modification tracking, ensuring GDPR compliance in real-time processes, and addressing regulations like 'right to be forgotten'. Our long-term plan involves separating Personally Identifiable Information (PII) and implementing a secure data vault, which is crucial for minimizing GDPR risks and enhancing security. The session offers practical solutions for privacy compliance in data processing, targeting data engineers, architects, and compliance officers looking to align data practices with privacy regulations using advanced technology.