Ingest, transform, and route real-time events with Microsoft Fabric event streams

• Announcements
• Synapse

• May 23, 2023 by Ajeta Singhal

See Arun Ulagaratchagan’s blog post to read the full Microsoft Fabric preview announcement.
 
You can now ingest, capture, transform and route real-time events to various destinations in Microsoft Fabric with a no-code experience using Microsoft Fabric event streams. It enables customers to ingest real-time event data from external event sources into the data stores in Fabric. The events could be transformed into the native formats required for target destination. For example, Eventstream could transform the events into Delta Lake format for the Lakehouse, into SQL columns based on the table schema, or filter events so that homogenous data can be sent to a KQL table.
 
To create a new instance of Microsoft Fabric event streams, navigate to Real-time Analytics in your Fabric workspace where you will find Eventstream.

After creating a new Eventstream item, you will land into a no code & easy to use experience where you can configure your eventstream from source to destination.
  

What is Fabric event streams

1. Centralized place for real-time events: It provides the capability to capture, modify, and direct your streaming data in real-time using a fully managed and scalable infrastructure.
2. Multiple source connectors: It enables you to ingest your real-time streaming data from 3 source types today: Azure Event Hubs, Sample data or Custom application.
3. Multiple destinations: It enables you to transform/capture/route real-time streaming data to 3 destination types today: KQL database, Lakehouse, or Custom application.
4. No code experience: It provides an intuitive and easy to use drag & drop experience with end-to-end data visibility and monitoring.

 
Let’s dive into each point stated above in detail:

1. Centralized place for real-time events 

Microsoft Fabric event streams utilize a scalable infrastructure that efficiently manages scaling and resourcing automatically. This enables you to effortlessly capture your real-time events, facilitating streaming ETL (Extract, Transform, Load) operations, as well as transforming and routing the events to your desired destination.
 

2. Multiple source connectors

Microsoft Fabric event streams enable you to ingest your real-time event data from 3 source types today:

• • Azure Event Hubs: You can seamlessly ingest events from an Azure event hub configured using Fabric Data connections.
  • Sample data: You can easily configure your eventstream to continuously ingest sample events from either Yellow Taxi data or Stock Market data.
  • Custom application: With this source, a streaming endpoint is created that allows customers to have their custom streaming application communicate directly with Fabric with either a Kafka client or an AMQP client.

3. Multiple destinations

Microsoft Fabric event streams enable you to transform and/or route your real-time event data to 3 destination types today: 

• • Fabric KQL Database: This destination helps with direct ingestion of your real-time events into KQL database. Once the data lands in KQL database, you can then perform additional queries/analysis to gain deeper insights or build Power BI reports.
  • Fabric Lakehouse: With this destination, you can transform (manipulate, filter, aggregate etc.) your real-time events before routing into a Lakehouse table. Real-time events are converted into Delta Lake format and then stored in the designated Lakehouse tables.
  • Custom application: With this destination, you can configure your custom applications to use their Kafka or AMQP client application to pull the events directly from Fabric.

4. No code experience

There is a super simple & easy to use drag & drop experience with end-to-end data visibility. You can view the shape of your data at every step from source to transformation to destination. You can also monitor each step by looking at meaningful insights.

 
To see detailed documentation, please see link.  If you have any questions or feedback, please reach out to us at askeventstreams@microsoft.com  
  

Get started with Microsoft Fabric

Microsoft Fabric is currently in preview. Try out everything Fabric has to offer by signing up for the free trial—no credit card information required. Everyone who signs up gets a fixed Fabric trial capacity, which may be used for any feature or capability from integrating data to creating machine learning models. Existing Power BI Premium customers can simply turn on Fabric through the Power BI admin portal. After July 1, 2023, Fabric will be enabled for all Power BI tenants.
 
Sign up for the free trial. For more information read the Fabric trial docs.
  

Other resources

If you want to learn more about Microsoft Fabric, consider:

• • Signing up for the Microsoft Fabric free trial
  • Visiting the Microsoft Fabric website
  • Reading the more in-depth Fabric experience announcement blogs:
    • Data Factory experience in Fabric blog
    • Synapse Data Engineering experience in Fabric blog
    • Synapse Data Science experience in Fabric blog
    • Synapse Data Warehousing experience in Fabric blog
    • Synapse Real-Time Analytics experience in Fabric blog
    • Power BI announcement blog
    • Data Activator experience in Fabric blog
    • Administration and governance in Fabric blog
    • OneLake in Fabric blog
    • Microsoft 365 data integration in Fabric blog
    • Dataverse and Microsoft Fabric integration blog
  • Exploring the Fabric technical documentation
  • Reading the free e-book on getting started with Fabric
  • Exploring Fabric through the Guided Tour
  • Watching the free Fabric webinar series
  • Joining the Fabric community to post your questions, share your feedback, and learn from others
  • Visiting Microsoft Fabric Ideas to submit suggestions for improvements and vote on your peers’ ideas