Announcing the Public Preview of Mirroring in Microsoft Fabric
We are excited to announce that Mirroring, previously announced at Ignite in November 2023, is now available to customers in Public Preview. You can now seamlessly bring your databases into OneLake in Microsoft Fabric, enabling seamless zero-ETL, near real-time insights on your data – and unlocking warehousing, BI, AI, and more.
Data driven insights are important for every business. With the critical need to make smart decisions, create new things, improve your products or services – time to value is everything. Yet, this can be difficult when you have data in different places, like apps, databases, and data warehouses. These places typically store data differently, so you can’t easily analyze and cross reference them – you have to laboriously move their data to a place where you can analyze and harmonize at scale. Doing this takes time, money, and typically, costly expertise to build complex, connected solutions. By the time you do this your data is old, and your insights are out of date. Decision makers need to be able to ask questions about their data, without time consuming complexity that adds risk and can impact mission critical workloads.
Mirroring simplifies this process into clicks and seconds, not complex processes and hours, days, or weeks. You get a modern, fast, and safe way of accessing and ingesting data continuously and seamlessly from databases or data warehouses into Fabric’s OneLake, without the need for cumbersome pipelines – in near real time. Combined with the rest of your organization’s data in OneLake, you can quickly unify and govern your data estate, removing data silos. Sound like a win-win?
Let’s jump in.
Easily Mirror your database or data warehouse’s data
Mirroring enables your database to be securely accessed and managed centrally from within Fabric without having to switch database clients or install any software. By just providing connection details and securely logging in, your database is instantly available in Fabric as a Mirrored database.
Frictionless, real-time data replication
There is no complex setup or ETL for data replication. With the same connection details, data is updated in a reliable way continuously, no intervention or code needed. An initial snapshot is created after which data is kept in sync in near real-time with every transaction whenever a new table is created, or new data is inserted/updated/deleted.
Replication uses the source database’s Change Data Capture (CDC ) technology, transforms it into appropriate Delta tables and lands it in OneLake. Intelligent logic determines when the source has changed, ensuring compute isn’t used unnecessarily before replicating the data. Granular controls enable configuring what is mirrored into Fabric. Detailed monitoring is also available to gain insights into mirroring operations and when the replica in Fabric OneLake was last refreshed. From here on, the data is ready for consumption immediately in any Fabric workload.
Data warehousing experiences simplified
Every Mirrored database comes with default data warehousing experiences (and the industry leading security capabilities) via a SQL Analytics Endpoint which houses the metadata of the Delta tables and points to the data in OneLake. Whether a SQL developer or citizen developer, one can query using the T-SQL editor which comes with full Intellisense or the visual query editor.
Pro code:
No code:
Easily join data across Mirrored databases, Warehouses, Lakehouses
As you may know, all data in Fabric is already in Delta format in OneLake. This includes Mirrored data. Any Mirrored database can be cross joined with other databases, warehouses or lakehouses whether that be data in Azure Cosmos DB, Azure SQL DB, Snowflake, etc.
Data Science experiences unlocked
Because a single copy of the data lives in OneLake, any data scientist can create a Lakehouse with a shortcut to the Mirrored database and use Notebooks to analyze and create models with the data.
Power BI Direct Lake – enabling clicky-clicky, draggy-droppy, analysis
With the same data, Power BI reports and semantic models can be built in Direct Lake mode – leveraging the blazing fast performance of import mode but without duplicating the data. As Direct Lake mode also supports reading Delta tables right from OneLake, the Mirrored database is Power BI ready.
What early adopters are saying
Since late last year’s announcement, we’ve seen many customers in the private preview analyzing their data, finding new insights, and providing much appreciated feedback. Below we’ve captured a few snippets:
- “Cosmos DB Mirroring is revolutionizing No-ETL design patterns, providing near-real-time data synchronization into the Delta format, where the mirrored data can easily be accessed by a variety of Fabric tools optimized to work with Delta. The technology is enabling us to reduce our code footprint, eliminate costs associated with data ingestion, and speed time to analytics without impacting the performance of operational data stores.” Ann Wallinger, Engineering Specialist, R&D, Baxter
- “The ability to cross-query mirrored Cosmos and SQL databases from Fabric will change how quickly we are able to innovate in our ever-changing data world. Very excited” Jacob Corey, VP of Data Solutions, Netrush, LLC
- “At Unite Digital, our primary data storage solution is Cosmos DB. To facilitate reporting, we conduct daily ETL processes that transfer data from Cosmos DB to Fabric. The introduction of Cosmos DB Mirroring for Fabric presents an opportunity to streamline this process significantly, providing near real-time data availability for reporting purposes within Microsoft Fabric. This integration has proven to be straightforward to implement and has functioned flawlessly. The capability to offer near real-time reporting in a domain where the norm is a 24-48 hour turnaround time for reports serves as a distinct competitive edge. We have plans to phase out our traditional ETL processes in favor of transitioning entirely to Cosmos DB Mirroring for Fabric upon its general availability.” Shahid Syed, Director of Technology and Security Compliance, Unite Digital
- “By streamlining transfer of data from Azure SQL DB to Fabric, Mirroring not only saves us time and money, but also allows us to conveniently support near real time analytics” Dhyan, Product Manager, Data Platform, Ramsoft
- “After having tested Mirroring under private preview, we are convinced, that it will play a central part in our Enterprise Fabric. Being able to seamlessly mirror any CosmosDB or Azure SQL Database to OneLake with just a few clicks will dramatically speed up development of new data products, minimize risk and reduce technical debt.” Jacob Rønnow Jensen, Head of Department for Data Platform, AP Pension
Mirroring Availability
Today, Azure Cosmos DB, Azure SQL DB, and Snowflake customers on any cloud will be able to mirror their data in OneLake and unlock all the capabilities of Fabric’s Data Warehouse, Direct Lake Mode in Power BI, Notebooks and much more. Besides the support for Azure Cosmos DB, Azure SQL Database, Snowflake in Mirroring, many more data sources will be added based on your feedback. You can help us prioritize what’s next by adding an idea here: https://aka.ms/fabricideas. In the meanwhile, stay tuned to the roadmap for updates: https://aka.ms/FabricRoadmap
Pricing:
We’re excited to announce that Mirroring storage and compute cost is free, providing generous amounts (terabytes) of storage based on the capacity size. This enables your organization to focus on finding valuable analytics scenarios unlocked by Mirroring.
For example, if you purchase an F2 capacity, compute for mirroring is not billed and you get 2 free terabytes worth of storage exclusively for mirroring. OneLake storage is billed only when the free Mirroring storage limit is exceeded, or the capacity is paused.
Learn more about Mirroring
- Get Started with a Fabric free trial
- Overview of Mirroring in Microsoft Fabric
- Mirrored Azure SQL Database in Fabric
- Mirrored Cosmos DB in Fabric
- Mirrored Snowflake in Fabric
- Turn it on for your organization, specific groups, or specific users
- OneLake Storage allotment by SKU
- Vote for new data sources or features for Mirroring
