Microsoft Fabric Updates Blog

Create Metadata Driven Data Pipelines in Microsoft Fabric

Metadata-driven pipelines in Azure Data Factory and Synapse Pipelines, and now, Microsoft Fabric, give you the capability to ingest and transform data with less code, reduced maintenance and greater scalability than writing code or pipelines for every data source that needs to be ingested and transformed. The key lies in identifying the data loading and transformation pattern(s) for your data sources and destinations and then building the framework to support each pattern.

I recently posted 2 blogs about a Metadata driven pipeline solution I created in Fabric.

A screenshot of a computer

Description automatically generated

Features include:

  • Metadata driven pipelines
  • Star schema design for Gold layer tables
  • Source data loaded into Fabric Lakehouse with Copy Data
  • Incremental loads and watermarking for large transaction tables and fact tables
  • 2 patterns for Gold layer
    • Fabric Lakehouse loaded with Copy Data activities and Spark notebooks
    • Fabric Data Warehouse loaded with Copy Data activities and SQL Stored Procedures

Why two options for the Gold layer? If you want to use T-SQL Stored Procedures for transformations, or have existing Stored Procedures to migrate to Fabric, Fabric Data Warehouse may be your best option, since it supports multi-table transactions and INSERT/UPDATE/DELETE statements. Comfortable with Spark notebooks? Then consider Fabric Lakehouse, which has the added bonus of Direct Lake connection from Power BI.

Check out the posts below to learn about building Metadata Driven Pipelines in Microsoft Fabric!

Part 1 – Metadata Driven Pipelines for Fabric with Lakehouse as Gold Layer

Part 2 – Metadata Driven Pipelines for Fabric with Data Warehouse as Gold Layer

Kapcsolódó blogbejegyzések

Create Metadata Driven Data Pipelines in Microsoft Fabric

június 10, 2024 Készítette Sean Mirabile

Authors: Prashant Atri & Prabhjot Kaur Welcome to Version 1 of the ultimate Data Factory Pipeline Mind Map! This one-page view is packed with guidelines to help you navigate Data Factory pipelines on your Data Factory journey to build a successful Data Integration project. Starting from the left-hand side, you will find broad feature areas … Continue reading “A Data Factory Pipeline Navigator Map to a Successful POC”

június 6, 2024 Készítette Jianlei Shen

We are excited to announce the release of several powerful new connectors in Fabric Data Factory this May 2024! These connectors will significantly enhance your data integration and transformation capabilities, enabling seamless workflows and expanding your data connectivity options. Here’s a detailed overview of the newly released connectors and a sneak peek at what’s coming … Continue reading “Announcing New Data Factory Connectors Released in May 2024”