SQL-Based ETL: Options for SQL-Only Databricks Development

Using SQL for data transformation is a powerful way for an analytics team to create their own data pipelines. However, relying on SQL often comes with tradeoffs such as limited functionality, hard-to-maintain stored procedures or skipping best practices like version control and data tests. Databricks supports building high-performing SQL ETL workloads. Attend this session to hear how Databricks supports SQL for data transformation jobs as a core part of your Data Intelligence Platform.

In this session we will cover 4 options to use Databricks with SQL syntax to create Delta tables:

• Lakeflow Declarative Pipelines: A declarative ETL option to simplify batch and streaming pipelines
• dbt: An open-source framework to apply engineering best practices to SQL based data transformations
• SQLMesh: an open-core product to easily build high-quality and high-performance data pipelines
• SQL notebooks jobs: a combination of Databricks Workflows and parameterized SQL notebooks