Eventhouse Accelerated OneLake Table Shortcuts (Generally Available)
Turbo charge queries over Delta Lake and Iceberg tables in OneLake
Eventhouse accelerated OneLake Table shortcuts aka. Query Acceleration is now Generally Available!
OneLake shortcuts are references from an Eventhouse that point to internal Fabric or external sources. Previously, queries run over OneLake shortcuts were less performant than on data that is ingested directly to Eventhouses due to various factors such as network calls to fetch data from storage, the absence of indexes, and more – Query Acceleration changes that.
During the preview, hundreds of customers adopted the feature, and the feedback has been overwhelmingly positive. From faster insights to smoother cross-platform collaboration, Query Acceleration is already making a difference.
“Eventhouse(“Kusto” engine) is one of the most effective tools I’ve worked with. The performance of Query Acceleration is impressive, and the new interface is a game changer for us at Siemens Healthineers. It’s helping bridge the gap between our Databricks and Fabric Real-Time Intelligence (RTI) teams—thank you for the continued innovation and commitment!”
— Dr. Werner Zirkel, Senior Key Expert Data Analytics, Siemens Healthineers
Query Acceleration
Query acceleration indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark.
When creating a shortcut from an Eventhouse to a OneLake delta table, users can choose if they want to accelerate the shortcut. Accelerating the shortcut means equivalent ingestion into the Eventhouse: indexing, caching, other optimizations that deliver the same level of performance for accelerated shortcuts as native Eventhouse tables.
By using this feature, you can accelerate data landing in OneLake, including existing data and any new updates. This eliminates the need to manage ingestion pipelines, maintain duplicate copies of data, while ensuring that data remains in sync without additional effort.
When using accelerated OneLake shortcuts, all data management is done by the data writer and in the Eventhouse the accelerated table shortcut, like any other shortcut is read only.
Customers can expect significant performance improvements by enabling this capability, in some cases up to 50x and beyond.
Supports Delta and Iceberg Tables
Eventhouse accelerated OneLake table shortcuts supports accelerating Delta Lake Tables natively, but it also supports accelerating Apache Iceberg Tables using a OneLake feature called metadata virtualization, which allows Iceberg tables to be interpreted as Delta Lake tables from the shortcut’s perspective. When you create a shortcut to an Iceberg table folder in Lakehouse, OneLake automatically generates the corresponding Delta Lake metadata (the Delta log) for that table. Eventhouse can then accelerate this Lakehouse shortcut (which appears as delta table) by creating a shortcut to it.
When to use accelerated table shortcuts
Here are some scenarios where this feature proves invaluable:
1. Query data in OneLake at blazing fast speed – When you have existing workloads that are uploading data and managing it in storage (in a different cloud or region), and you would like to query some or all of it at blazing fast speed.
2. Mesh historical data with real-time streams – When you want to seamlessly combine data landing in OneLake directly with real-time streams coming into Eventhouse without compromising on query speeds.
3. Leverage dimension data managed by other items – Often high value and small volume data is hosted in SQL servers, Cosmos DB, Snowflake or other systems that can be mirrored into OneLake. Accelerated OneLake shortcuts are excellent in making this data easily consumable for joins and enrichment in the Evenhouse query. As dimension data is often significantly smaller than activity data, the additional cost associated with that usage is typically minimal, but the performance gain is huge.
How to enable Query Acceleration
To enable query acceleration on a new shortcut, follow these steps:
- Browse to an existing KQL database.
- Select New > OneLake shortcut.
3. Select a source.
4. Under Internal sources, select Microsoft OneLake (or any other source where your data resides).
5. Select the item you want to connect to, and then select Next.
6.Expand Tables, and select a specific table to connect to.
7. Select Next.
8. Toggle the Accelerate button to On.
9. Select Create.
10. Query the data using external_table(‘Shortcut name‘) syntax.
It might take few minutes to notice the performance gain on existing data as Eventhouse will take some time to create initial indexes.
Cost and billing
Enabling Query Acceleration does come with some additional costs. The accelerated data will be charged under OneLake Premium cache meter, similar to native Eventhouse tables. You can control the amount of data to accelerate by configuring number of days to cache. Indexing activity may also count towards CU consumption.
These charges will appear in the Fabric metrics app under the Eventhouse where the accelerated shortcut is created.
Get started now: Query Acceleration for OneLake Shortcuts in Eventhouse
What’s next?
There will be many more exciting developments as we continue to innovate and expand the capabilities of Real-Time Intelligence. Learn more about all the features and follow the step-by-step tutorial. Join the conversation and vote for your favorite features.
