Announcing Databricks Runtime 5.3

Published: April 4, 2019

We are proud to announce the release of Databricks Runtime 5.3, which includes several new features and improvements, including:

GA of Delta Time Travel
Public Preview of MySQL table replication to Delta
Optimized DBFS FUSE folder for deep learning workloads (Azure-only)

Databricks Delta Time Travel (Generally Available)

Delta Time Travel has graduated to general availability. It adds the ability to query a snapshot of a table using a timestamp string or a version, using SQL syntax as well as DataFrameReader options for timestamp expressions.

Time Travel has many use cases, including:

Re-creating analyses, reports, or outputs (for example, the output of a machine learning model), which is useful for debugging or auditing, especially in regulated industries.
Writing complex temporal queries.
Fixing mistakes in your data.
Providing snapshot isolation for a set of queries for fast changing tables.

https://www.youtube.com/watch?v=TrBDuaF7g5A

For more details, see Query an older snapshot of a table (time travel), and Merge Into (Databricks Delta).

MySQL table replication to Delta (Public Preview)

Databricks Runtime 5.3 lets you stream data from a MySQL table directly into Delta for downstream consumption in Spark analytics or data science workflows. Leveraging the same strategy that MySQL uses for replication to other instances, the binlog is used to identify updates that are then processed and streamed to Databricks as follows:

Reads change events from the database log.
Streams the events to Databricks.
Writes in the same order to a Delta table.
Maintains state in case of disconnects from the source.

Optimized DBFS FUSE folder for deep learning workloads (Azure only)

A new FUSE mount optimized for data loading, model checkpointing, and logging from each worker to a shared storage location, file:/dbfs/ml provides high-performance I/O for deep learning workloads.

For details, see Prepare Storage for Data Loading and Model Checkpointing.