Microsoft Fabric Updates Blog

Decoding Data with Confluent Schema Registry Support in Eventstream (Preview)

We are pleased to announce that Eventstream’s Confluent Cloud for Apache Kafka streaming connector now supports decoding data from Confluent Cloud for Apache Kafka topics that are associated with a data contract in Confluent Schema Registry. The Challenge with Schema Registry Encoded Data The Confluent Schema Registry serves as a centralized service for managing and …

Continue reading “Decoding Data with Confluent Schema Registry Support in Eventstream (Preview)”

Accelerating Insights from Unstructured Text with AI‑Powered OneLake Shortcut Transformations in Microsoft Fabric (Public Preview)

Shortcut‑based AI transformations in Microsoft Fabric convert raw text files into governed Delta Lake tables within minutes, removing the need for complex data‑integration pipelines and significantly reducing time to insight. Why adopt AI transformations? How to set up an AI-powered shortcut transformations 3. Monitor progressStatus can be observed under Manage shortcut Practical implementation patterns All these …

Continue reading “Accelerating Insights from Unstructured Text with AI‑Powered OneLake Shortcut Transformations in Microsoft Fabric (Public Preview)”

Enhancing Data Transformation Flexibility with Multiple-Schema Inferencing in Eventstream (Preview)

Introducing multiple-schema inferencing in Eventstream! This feature empowers you to work seamlessly with data sources that emit varying schemas by inferring and managing multiple schemas simultaneously. It eliminates the limitations of single-schema inferencing by enabling more accurate and flexible transformations, preventing field mismatches when switching between Live and Edit modes, and allowing you to view …

Continue reading “Enhancing Data Transformation Flexibility with Multiple-Schema Inferencing in Eventstream (Preview)”

User Data Functions now support async functions and pandas DataFrame, Series types

Microsoft Fabric has introduced new features for its User Data Functions (UDFs), enhancing Python-based data processing capabilities within the platform. These updates include support for asynchronous functions and the use of pandas DataFrame and Series types for input and output, enabling more efficient handling of large-scale data.
• Async function support: Developers can now write async functions in Fabric UDFs to improve responsiveness and efficiency, especially for managing high volumes of I/O-bound operations, such as reading files asynchronously from a Lakehouse.
• Pandas DataFrame and Series integration: UDFs can accept and return pandas DataFrames and Series, allowing batch processing of rows with improved speed and performance in data analysis tasks. An example function calculates total revenue by driver using pandas groupby operations.
• Usage in notebooks: These functions can be invoked directly from notebooks using pandas objects, facilitating efficient aggregation and analysis of large datasets interactively within Microsoft Fabric.
• Getting started and benefits: Users can enable these features by updating the fabric-user-data-functions library to version 1.0.0. The enhancements reduce I/O operations, enable concurrent task handling, and improve performance on datasets with millions of rows.

Secure Data Streaming with Private Endpoints in Eventstream (Generally Available)

We’re excited to announce the General Availability of Managed Private Endpoints (MPE) in Fabric Eventstream. This network security feature allows you to stream data from Azure resources to Fabric over a private and secure network without the complexity of manual network configurations. Why Network Security Matters for Streaming As organizations increasingly adopt real-time data streaming …

Continue reading “Secure Data Streaming with Private Endpoints in Eventstream (Generally Available)”