Microsoft Fabric Updates Blog

Introducing Mirroring in Microsoft Fabric

As mentioned in Arun’s blog post, we are thrilled to announce that Mirroring in Microsoft Fabric is coming soon.

In today’s AI driven world, analytics platforms are only as good as their data. With the ever-increasing amount of data being collected in various applications, databases, and data warehouses in an enterprise, managing and ingesting data into a central platform for purposes of analytics and AI is a cumbersome and costly process. Databases and data warehouses use proprietary storage formats making the ability to create shortcuts to their data impossible. Data needs to be extracted, transformed, normalized, and made available in a central place for analytics. Even when this is implemented, data is not real-time making any insights stale pretty quicky resulting in users having to query the data in the source.

Mirroring provides a modern way of accessing and ingesting data continuously and seamlessly from any database or data warehouse into the Data Warehousing experience in Microsoft Fabric. This is all in near real-time thus giving users immediate access to changes in the source!

Let’s dive deeper into its capabilities.

Access and manage any database or data warehouse

Any database can be accessed and managed centrally from within Fabric without having to switch database clients. By just providing connection details, your database is instantly available in Fabric as a Mirrored database. Familiar database editors provide the ability to manage the database. Here is an example of the Azure Cosmos DB editor available within the Mirrored database in Fabric. Any edits go directly against the Azure Cosmos DB source.

A screenshot of a computer

Description automatically generated

Real-time data replication

There is no complex setup or ETL for data replication. With the same connection details, data is replicated in a reliable way in real-time. An initial snapshot is created after which data is kept in sync in near real-time with every transaction whether a new table is created, or new data is inserted/updated/deleted. Replication uses the database’s Change Data Capture (CDC ) technology, transforms it into appropriate Delta tables and lands it in OneLake. Intelligent logic determines when the source has changed, ensuring compute isn’t used unnecessarily before replicating the data. Granular controls enable configuring what is mirrored into Fabric. Detailed monitoring is also available to gain insights into mirroring operations and when the replica in Fabric OneLake was last refreshed. From here on, the data is ready for consumption immediately in any Fabric workload.

Data warehousing experiences simplified!

Every Mirrored database comes with default data warehousing experiences via a SQL Analytics Endpoint which houses the metadata of the Delta tables and points to the data in OneLake. Whether a SQL developer or citizen developer, one can query using the T-SQL editor which comes with full Intellisense or the visual query editor.

A screenshot of a computer

Description automatically generated

Cross-joining Mirrored databases, Warehouses, Lakehouses

As all data in Fabric, whether added as a shortcut to a lake in a Lakehouse or is data ingested through Data Factory, Warehouse or Spark is already in Delta format, data can be cross joined. Now with Mirroring, data across any database can be cross joined as well enabling querying across any database, warehouse or lakehouse whether that be data in Azure Cosmos DB, Azure SQL DB, Snowflake, MongoDB etc.

A screenshot of a computer

Description automatically generated

Data Science experiences unlocked!

Any data scientist can create a Lakehouse with a shortcut to the Mirrored database and use Notebooks to analyze and create models with the data.

A screenshot of a computer

Description automatically generated

Power BI Direct Lake mode

Power BI reports and semantic models can be built in Direct Lake mode getting the blazing fast performance of import mode but without duplicating the data. As Direct Lake mode also supports reading Delta tables right from OneLake, the Mirrored database is Power BI ready.

A screenshot of a computer

Description automatically generated

Mirroring Availability

Azure Cosmos DB, Azure SQL DB and Snowflake customers will be able to use Mirroring to mirror their data in OneLake and unlock all the capabilities of Fabric Warehouse, Direct Lake Mode, Notebooks and much more. SQL Server, Azure PostgreSQL, Azure MySQL, MongoDB and other databases and data warehouses will be coming in CY24. For participation in our early adopter program, submit your application here.

In the meantime, to try out Fabric, sign up for a free trial.

Related blog posts

Introducing Mirroring in Microsoft Fabric

April 11, 2024 by Matthew Hicks

Microsoft OneLake is a unified data lake for all of your organization’s data. With OneLake shortcuts, you can reference data in different locations and have that data logically represented within OneLake, with no data movement or duplication. With the recent announcement of shortcuts to Google Cloud Storage, you can use shortcuts to seamlessly bring in … Continue reading “Public Preview of OneLake shortcuts to S3-compatible data sources”

April 11, 2024 by Trevor Olson

We are excited to announce that you can now create OneLake shortcuts to your Google Cloud Storage (GCS) buckets. With the addition of GCS, you can now utilize cross-cloud shortcuts to analyze your data across all three major cloud platforms. Shortcuts in OneLake allow you to connect to your existing data through a single unified … Continue reading “Shortcuts to Google Cloud Storage, now available in Public Preview”