Microsoft Fabric Updates Blog

Announcing the General Availability of Fast Copy in Dataflows Gen2

Fast Copy in Dataflow Gen2 is now General Available! This powerful feature enables rapid and efficient ingestion of large data volumes, leveraging the same robust backend as the Copy Activity in Data pipelines.

With Fast Copy, you can experience significantly shorter data processing times and improved cost efficiency for your Dataflow Gen2. Additionally, it boosts end-to-end performance by allowing you to use Fast Copy to ingest data into staging, and then seamlessly transform it at scale using SQL DW compute.

Fast Copy: Enhance Performance and Reduce Costs in Dataflow Gen2

Fast Copy leverages the same powerful backend as the copy activity in Data pipelines, utilizing a serverless architecture that enables parallelism to maximize data movement throughout. This approach fully utilizes network bandwidth and data store IOPS for optimal performance.

As detailed in this blog, using Fast Copy to load a 6 GB CSV file into a Lakehouse table in Microsoft Fabric resulted in an 8x increase in performance and a 3x reduction in cost in Dataflow Gen2.

Unlock the Power of Fast Copy: How to Get Started

When you create a new Dataflow Gen2, Fast Copy is automatically enabled. For data sizes over 100 MB or 5 million rows, Dataflow Gen2 seamlessly switches to the Fast Copy backend, no extra steps are needed during dataflow authoring.

After running a refresh, simply check the Refresh History to see if Fast Copy was used by reviewing the entity status.

Supported data sources

Fast Copy supports numerous source connectors, including ADLS Gen2, Blob storage, Azure SQL DB, On-Premises SQL Server, Oracle, Fabric Lakehouse, Fabric Warehouse, PostgreSQL, and Snowflake.

An on-premises gateway is also supported, allowing for high-performance data transfer from on-premises sources. For details, see Fast Copy with On-premises Data Gateway Support in Dataflow Gen2.

Start optimizing your data processing with Fast Copy in Dataflow Gen2 today!

Resources:

Docs: Fast copy in Dataflows Gen2 – Microsoft Fabric | Microsoft Learn

Related blog posts

Announcing the General Availability of Fast Copy in Dataflows Gen2

October 7, 2024 by Alex Lin

Introducing Managed VNet Support for Fabric Eventstream! By creating a Fabric’s Managed Private Endpoint, you can now securely connect Eventstream to your Azure services, such as Azure Event Hubs or IoT Hub, within a private network or behind a firewall. This integration ensures your data is securely transmitted over a private network, enabling you to … Continue reading “Secure Data Streaming with Managed Private Endpoints in Eventstream (Preview)”

October 4, 2024 by Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”