Microsoft Fabric Updates Blog

Announcing: Automatic Log Checkpointing for Fabric Warehouse

We are excited to announce automatic log checkpointing for Data Warehouses!

One of our goals with the Data Warehouse is automate as much as possible to make it easier and cheaper for you to build and use them. This means you will be spending your time on adding and gaining insights from your data instead of spending it on tasks like maintenance. As a user, you should also expect great performance which is where log checkpointing comes in!

What is Log Checkpointing and why is it important?

To understand what log checkpointing is and why it is important, we need to first talk about how tables are stored and how they are queried.

When you create a table and add data to it, the data is stored in parquet files on OneLake. Internally, there is also a log file that keeps track of which parquet files, when combined, make up the data that is in the table. These log files are internal and cannot be used directly by other engines. Instead, we automatically publish Delta Lake Logs so that other engines can directly access the right parquet files.

Now, imagine that you load data into your table every 5 minutes. That means over the course of a year, you would have loaded data to your table 105,120 times. Each time, a new log file would be created that tells the system that when reading the table, the new parquet files need to be read as well. That means when reading the table, the system first needs to read all 105,120 log files which is not very performant.

This is where log checkpointing comes in! As of the time of this blog, after every 10 transactions, we automatically and asynchronously create a new log file that is called a checkpoint. This file is basically a summary of all the previous log files. Now when you query the table, the system needs to read the latest checkpoint and any log files that were created after. Instead of having to read 105,120 log files, we would typically need to read 10 or less files!

Conclusion

Log Checkpointing is one of the ways that we help your Data Warehouse to provide you with great performance and best of all, it involves no additional work from you! This helps give you more time to work on leveraging your Data Warehouse to gain more value and insights!

Please look forward to more announcements about more automated performance enhancements!

関連するブログ記事

Announcing: Automatic Log Checkpointing for Fabric Warehouse

7月 17, 2024 作成者: Ambika Jagadish

In the rapidly evolving world of generative artificial intelligence, historical data plays a significant role in influencing the decision-making process and shaping organizational strategies. Data retention within data warehouse refers to the practice of preserving and managing previous iterations of the data encompassing any inserts, updates or deletes made to the warehouse for a specified … Continue reading “Announcing the General availability of time travel and 30 days of data retention in Fabric Warehouse”

7月 17, 2024 作成者: Sowmya Sivaraman

In today’s rapidly evolving data management landscape, maintaining the resilience and continuity of your data infrastructure is essential. Unplanned system failures and scheduled maintenance alike demand the ability to restore data warehouses swiftly and seamlessly. This capability is no longer just a feature – it’s a critical necessity in modern analytics environments. A quick and … Continue reading “Seamless Data Recovery through Warehouse Restoration within Fabric Query Editor”