Microsoft Fabric Updates Blog

Introducing Job Queueing for Notebook in Microsoft Fabric

Users orchestrate their data engineering or data science processes using notebooks and in most of the enterprise scenarios pipelines and job schedulers are used as a primary option to schedule and trigger these Spark jobs. We are thrilled to announce a new feature Job Queueing for Notebook Jobs in Microsoft Fabric. This feature aims to eliminate manual retries and improve the user experience for our customers who run notebook jobs on Microsoft Fabric.

Notebook jobs are a popular way to run data analysis and machine learning workflows on Fabric. They can be triggered by pipelines or a job scheduler, depending on the user’s needs. However, in the current system, notebook jobs are not queued when the Fabric capacity is at its max utilization. They are rejected with a Capacity Limit Exceeded error, which forces the user to retry the job later when the resources are available. This can be time-consuming expensive in operations, especially for enterprise users who run many notebook jobs to resubmit the jobs that have been throttled.

With Job Queueing for Notebook Jobs, this problem is solved. Notebook jobs that are triggered by pipelines or job scheduler will be added to a queue and will be retried automatically when the capacity frees up.

The user does not need to do anything to resubmit the job when its added to the queue as its automatically retried and starts execution when the capacity is freed up. The status of these notebook jobs will be Not Started when in queued state and will be changed to In Progress when they start the execution.

Fabric Spark enforces queue sizes based on the capacity SKU size attached to a workspace, providing a queueing mechanism based on the purchased Fabric capacity SKUs.

The following section lists various queue sizes for Spark workloads based on Microsoft Fabric based on the capacity SKUs:

Fabric capacity SKUEquivalent Power BI SKUQueue limit
F24
F44
F88
F1616
F3232
F64P164
F128P2128
F256P3256
F512P4512
F10241024
F20482048
Trial CapacityP1NA
Spark queue limits based on Fabric Capacity SKUs

To learn more about the job queueing experience in Fabric Spark, please refer to our documentation Job queueing for Fabric Spark – Microsoft Fabric | Microsoft Learn

To learn more about the throttling experience on Fabric Spark based on the Fabric capacity SKU, please refer to our documentation Concurrency limits and queueing in Microsoft Fabric Spark

相關部落格文章

Introducing Job Queueing for Notebook in Microsoft Fabric

10月 7, 2024 作者: Alex Lin

Introducing Managed VNet Support for Fabric Eventstream! By creating a Fabric’s Managed Private Endpoint, you can now securely connect Eventstream to your Azure services, such as Azure Event Hubs or IoT Hub, within a private network or behind a firewall. This integration ensures your data is securely transmitted over a private network, enabling you to … Continue reading “Secure Data Streaming with Managed Private Endpoints in Eventstream (Preview)”

10月 4, 2024 作者: Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”