Microsoft Fabric Updates Blog

Announcing: Automatic Log Checkpointing for Fabric Warehouse

We are excited to announce automatic log checkpointing for Data Warehouses!

One of our goals with the Data Warehouse is automate as much as possible to make it easier and cheaper for you to build and use them. This means you will be spending your time on adding and gaining insights from your data instead of spending it on tasks like maintenance. As a user, you should also expect great performance which is where log checkpointing comes in!

What is Log Checkpointing and why is it important?

To understand what log checkpointing is and why it is important, we need to first talk about how tables are stored and how they are queried.

When you create a table and add data to it, the data is stored in parquet files on OneLake. Internally, there is also a log file that keeps track of which parquet files, when combined, make up the data that is in the table. These log files are internal and cannot be used directly by other engines. Instead, we automatically publish Delta Lake Logs so that other engines can directly access the right parquet files.

Now, imagine that you load data into your table every 5 minutes. That means over the course of a year, you would have loaded data to your table 105,120 times. Each time, a new log file would be created that tells the system that when reading the table, the new parquet files need to be read as well. That means when reading the table, the system first needs to read all 105,120 log files which is not very performant.

This is where log checkpointing comes in! As of the time of this blog, after every 10 transactions, we automatically and asynchronously create a new log file that is called a checkpoint. This file is basically a summary of all the previous log files. Now when you query the table, the system needs to read the latest checkpoint and any log files that were created after. Instead of having to read 105,120 log files, we would typically need to read 10 or less files!

Conclusion

Log Checkpointing is one of the ways that we help your Data Warehouse to provide you with great performance and best of all, it involves no additional work from you! This helps give you more time to work on leveraging your Data Warehouse to gain more value and insights!

Please look forward to more announcements about more automated performance enhancements!

Related blog posts

Announcing: Automatic Log Checkpointing for Fabric Warehouse

April 9, 2024 by Idris Motiwala

In the era of digital transformation, advanced analytics and an AI driven world, data has emerged as the new oil, powering businesses, and driving decision-making. But what good is this oil if it’s not refined and ready for use when needed? Moreover, managing, and ingesting data into a central platform for analytics and AI is … Continue reading “Announcing Mirroring Azure SQL Database in Fabric for Public Preview”

March 26, 2024 by Charles Webb

Mirroring is now available to customers in Public Preview. You can now seamlessly bring your databases into OneLake in Microsoft Fabric, enabling seamless zero-ETL, near real-time insights on your data – and unlocking warehousing, BI, AI, and more.