The recent Databricks Data+AI Summit 2024 buzzed with exciting announcements, and the one that grabbed our attention at Konverge AI is LakeFlow! This unified solution for data ingestion, transformation, and orchestration, built on Databricks’ Delta Live Tables (DLT) framework will be a game-changer. As a Databricks partner, we are excited about LakeFlow, enabling us to help clients enhance data management.

Here are the key benefits we’re looking forward to:

1. Simplified Data Connectivity

LakeFlow is going to be equipped with a range of connectors. This would help teams avoid the initial plumbing required to fetch data from external sources, push/convert it into object storage like ABFSS (Azure) and S3 (AWS), from which DLT (Delta Live Tables) pipelines can extract. While Partner Connect and Lakehouse Federation can achieve this, LakeFlow aims to make the process even smoother.

2. Built-In Change Data Capture (CDC)

For implementing CDC, we have to depend on tools like Debezium, which often requires lengthy discussions with enterprise clients due to their open-source nature. LakeFlow’s built-in CDC support will remove this dependency, making the process seamless and more efficient.

3. User-Friendly Pipeline UI

One of the most exciting features of LakeFlow is its intuitive pipeline user interface. Even non-coders will be able to define an end-to-end pipeline really fast. It will be interesting to see how LakeFlow fares compared to the existing Workflow jobs.

4. Integrated Orchestration

Presently, job orchestration relies on external tools such as Apache Airflow. We anticipate that LakeFlow will integrate this capability, simplifying the orchestration process within a single platform.

5. Enhanced Data Quality Checks

In DLT pipelines, we can achieve data quality checks using the “expect” function. We are curious to see how LakeFlow will present these checks within its UI, potentially offering a more streamlined and user-friendly experience.

6. Comprehensive Monitoring

Monitoring the DLT pipelines is achieved by utilizing table valued functions (TVF) over event logs. But for the actual Monitoring process, we still require the likes of Grafana and Promethus. LakeFlow promises to integrate these monitoring capabilities, providing a unified and comprehensive monitoring solution within the platform.

With LakeFlow, Databricks is making significant strides towards user-friendliness and accessibility for non-coders. We at Konverge.AI are confident that LakeFlow will revolutionize data management strategies, making them more efficient and user-friendly.