Microsoft Fabric Updates Blog

From raw data to insights: How to ingest data from Azure Event Hubs into KQL database for powerful analytics

Synapse Real-Time Analytics in Microsoft Fabric is a fully managed big data analytics experience optimized for streaming and analyzing real-time data. In this blog post, we’ll learn about the integration between Azure Event Hubs and your KQL database.
 
One of the components of Real-Time Analytics is the Kusto Query Language (KQL) Database, a fast, powerful, scalable, and highly available tool that is designed to handle large volumes of data in real time. With its simple yet powerful syntax, KQL allows anyone to query, analyze, and visualize data to get insights in real-time. This allows, for example, businesses to quickly respond to changes in their operations and make informed decisions based on the latest data.
 
One of the key benefits of KQL Databases is its ability to ingest data from a wide range of sources, including Azure Event Hubs, making it easy to get data into the service and start analyzing it.
 
Azure Event Hubs is a big data streaming platform and event ingestion service that can process and direct millions of events per second with low latency. Event Hubs represents the “front door” for an event pipeline, often called an event ingestor in solution architectures. An event ingestor is a component or service that sits between event publishers and event consumers to decouple the production of an eventstream from the consumption of those events. Event Hubs provides a unified streaming platform with a time retention buffer, decoupling event producers from event consumers.


Screenshot of a KQL database showing the Get data from Event Hubs option
 
With the new integration, you can ingest data directly from Azure Event Hubs into your KQL database, which is useful when you already have an Azure event hub streaming data, and you want to analyze the data in real-time. Using Azure Event Hubs, you can easily bring data from endless sources including devices and sensors, applications, logs and metrics, and Azure cloud resources.
 
The integration between Azure Event Hubs and KQL Database is straightforward, utilizing a friendly wizard to ingest data.
 

Screenshot of the Event Hubs ingestion wizard showing a preview of event data, with the option to modify its format, and adjust nested levels (in case of JSON data)
 

Screenshot of a KQL query example that analyzes IoT sensor data received from Azure Event Hubs
 
The video below provides an end-to-end example, which includes several KQL queries for analyzing IoT sensor data coming from Event Hubs.

  

Get started with Microsoft Fabric

Microsoft Fabric is currently in preview. Try out everything Fabric has to offer by signing up for the free trial—no credit card information required. Everyone who signs up gets a fixed Fabric trial capacity, which may be used for any feature or capability from integrating data to creating machine learning models. Existing Power BI Premium customers can simply turn on Fabric through the Power BI admin portal. After July 1, 2023, Fabric will be enabled for all Power BI tenants.
 
Sign up for the free trial. For more information read the Fabric trial docs.
 
To read more about the integration of Azure Event Hubs with KQL Database, check out Get Data from Azure Event Hubs.

Gerelateerde blogberichten

From raw data to insights: How to ingest data from Azure Event Hubs into KQL database for powerful analytics

oktober 21, 2024 door Amir Jafari

The Share capability for Fabric AI Skill is now in public preview. This feature enables AI Skill creators to share their AI Skill with colleagues, allowing them to consume the AI Skill and gain valuable insights from the data. How Sharing WorksOnce you have created and validated the effectiveness of your AI Skill, you can … Continue reading “Announcing Public Preview of the “Share” Feature for Fabric AI Skill”

oktober 18, 2024 door Santhosh Kumar Ravindran

Guidance to Maximizing Productivity and Efficiency for your Data Engineering Workloads Data engineering teams often grapple with the complexities of planning and configuring compute resources for their data platforms. This is especially true when working with large-scale, complex datasets and demanding downstream SLAs. A one-size-fits-all approach is rarely effective, as different data layers and datasets … Continue reading “Optimizing Spark Compute for Medallion Architectures in Microsoft Fabric”