Microsoft Fabric December 2023 Update
Welcome to the December 2023 update.
We have lots of features this month including More styling options for column and bar charts, calculating distinct counts in Power BI running reports on KQL Databases, Changes to workspace retention settings in Fabric and Power BI, and many more.
Skill up on Fabric: Take the Microsoft Fabric Challenge
Join the thousands of professionals that have completed the Microsoft Fabric Challenge as part of the Microsoft Ignite Cloud Skills Challenge. In less than 8 hours, you will learn how to connect, ingest, store, analyze, and report on data with Power BI and Microsoft Fabric. The Microsoft Fabric challenge prepares you for the Microsoft Certified: Fabric Analytics Engineer Associate certification that is coming in the next months. The challenge is on until January 15, 2024.
The Microsoft Learn Cloud Skills Challenge enables you to skill up for in-demand technologies like Power BI and Microsoft Fabric, track your progress in a leaderboard and get rewards for your work once you complete the challenge. As part of the Ignite edition, you have a chance to win a VIP pass to the next Microsoft Ignite.
The Microsoft Fabric Community Conference
Join us at the first annual Microsoft Fabric Community Conference and see firsthand how Microsoft Fabric and the rest of the data and AI products at Microsoft can help your organization prepare for the era of AI. You’ll hear from leading Microsoft and community experts from around the world and get hands on experiences with the latest features from Microsoft Fabric, Power BI, Databases, Azure AI, Microsoft Purview, and more. We have just released more sessions to the website today!
Register today to immerse yourself in the future of data and AI and connect with thousands of data innovators like yourself eager to share their insights.
About our January 2024 Feature Summary blog
Due to the holidays, we will not publish the monthly feature summary blog in January. Instead, to begin the new year we will post a special blog looking back at 2023 and call out some key highlights!
Also, we will be making a very important and exciting announcement about certifications and learning for Microsoft Fabric! So, make sure you read the January blog.
Contents
- Reporting
- Data Connectivity
- Service
- Developers
- Visualizations
- New visuals in AppSource
- Activity Gauge by Powerviz
- Navigate Financial Risks and Rewards with Profitbase’s Power BI Visualization
- Intuitive multi-level donut charts for your reports
- Inforiver Analytics+ 3.2
- The Microsoft Purview hub in Fabricfor creators
- VideoPlayer Visual
- Ridgeline Plot by Powerviz
- Core
- Synapse
Reporting
New on-object default setup is here!
Each month we’ve been working toward giving users more flexibility in how they prefer to work with the PBI editor. This month we’re introducing a new default setup that feels closer to the previous classic look. With this new default, we’ll help you preset the following options with just one click!
- Move the “build a visual” menu to the pane switcher by default
- Show both build and data panes expanded by default
- Format pane in the pane switcher by default
- “Always open in new pane” set to ON, thereby stacking panes by default
- Collapse panes to the pane switcher instead of close “X” by default
Upon updating to the new December build, with the new on-object preview switch ON, you’ll be greeted with a dialog to update to the new default setup. If you like your existing on-object setup and do not wish to have a more classic look, choose Keep setup and no changes will be made.
If you change your mind, you can always go back to the Options menu to customize the experience to your needs. Here’s a walkthrough of what each of these options do:
- Suggest a type by default – If you prefer option in the Build menu to be off by default, expanding all the field wells always, make sure this is not checked.
- In the Build a visual menu, always show all the visualization types – If you prefer to see all the visualization types shown by default in the Build menu, be sure to check this option.
- Always show pane switcher – When checked, this provides the right-hand pane switcher shown even when working with zero panes. Provides access to the “+” menu and setting gear always.
- Always show the Build a visual menu in the pane switcher – This moves the build menu from showing next to the visual, to the pane switcher on the right side instead. Check this option if you prefer to work with the Build menu as a pane.
- Always open a new pane – Instead of switching between panes, check this option if you prefer the stacking behavior as before. With this option on, opening a new pane will open side-by-side by default.
To summarize, here’s the state of these settings for the new default setup:
Multi-select in data flyout
To cut down on the extra clicks when first adding data using the new “add data” button, the data flyout now allows you to multi-select fields to add fields in bulk to your visual, just like the data pane! Simply check all the fields you’d like to add to your visual in one go.
This works best when “suggest a type” is ON so you can add as many fields as you like and PBI will place the fields in the appropriate field wells for you.
In-product user education
With the many changes the on-object interaction introduced, it can hard to know where certain features may have moved. This month we’ve added four teaching bubbles to help users along the way.
- Where’s drillthrough and tooltip page set up?
- A teaching bubble will now point to the new location within Format > Page information.
- Where did the analytics pane go?
- Analytics options like error bars, max, min or percentile lines are now consolidated in the Format pane at the bottom. You can set a max, min or other reference line using the new “Reference line” card in the format pane.
- What happened to the right click options for a field like rename or ‘show value as’?
- They are still available at the right click! Simply right click on the field name to see the same options as before.
- How do I bring back my data pane if the pane switcher is gone?
- You shouldn’t run into this as frequently now that we’ve updated the behavior to collapse panes by default instead of closing. But in case you do, you can always re-open a pane from the View tab in the ribbon.
More styling options for column and bar charts
We’re thrilled to announce that this month brings many new formatting enhancements for column and bar charts in Power BI. These additions are designed to improve your data visualization, enabling more impactful analysis and storytelling capabilities. Here’s a quick look at the new features available now!
Key Enhancements:
- Apply Settings to All or Series: Enjoy the versatility of applying formatting options globally across all categories or tailor them for each unique individual series, effortlessly and efficiently!
- Transparency Control: You can now adjust the fill color transparency to achieve the perfect balance of visibility and design.
- Border Customization: Gaining superior control, you can now introduce borders with unmatched precision, choosing a distinct color, or match it to the column’s fill, and define both the border’s width and transparency for a polished look.
- Layout: We’ve renamed the Spacing card to be the new Layout card, relocated the familiar Reverse Order toggle that allows you to invert series, and introduced Sort by Value, Space between categories, and Space between series. And, with clustered columns and bar charts, you now have the option to Erode or Explode the stacked series.
- Advanced ribbon settings: Highlight a series or create visual continuity with an array of options including matching series color or choosing a color selection, transparency, border color matching or color selection, border width, and border spacing.
These features aim to provide you with greater flexibility and creative freedom in presenting your data, ensuring that your insights stand out with clarity and style. Enjoy exploring these new capabilities and crafting compelling narratives with your data in Power BI.
If you’re looking to learn more comprehensive information about these new features we’ve rolled out for column and bar charts, look for our dedicated blog post.
Extended customization for data labels
Data labels are also getting more options this month! With this month’s update, you’ll discover some new enhanced capabilities for Data labels, giving you extended customization choices to seamlessly match your reporting needs, and it brings the simple integration of additional metrics, dramatically improving your data analysis experience. Here are this month’s highlights!
Key Enhancements:
- Title: The new Title card feature enables the display of legend fields on your data labels, offering the option to omit the legend for a cleaner look. It also allows customized data label titles using different data fields, adjustable font style, color, and transparency.
- Value: Easily spot the active field for your data label in the field well. Modifying it to another field is quick and easy, with customization options like font, color, transparency, and more, plus a new feature for blank value display.
- Detail: Our new Detail card feature unlocks new potential by allowing the inclusion of a secondary metric to your data labels. Choose any field to add an extra value, with the full range of formatting options at your disposal.
- Visual label layout: A new feature that lets you choose between a sleek single-line data label or a multi-line data label, where desired.
We invite you to explore these remarkable new features of Data labels to significantly enhance the data density in your charts. Available for Columns, Bars, Lines, and Ribbon charts, these advancements are ready to transform your reporting. We’re excited to hear your thoughts on these innovations and value your input on future Data Label enhancements!
For more detailed information about these new Data label features, look for our dedicated blog post.
Azure Maps clustering aggregation support for bubble layer
We are thrilled to introduce a powerful enhancement to the Power BI Azure Maps visual – the clustering aggregation feature for the bubble layer with customizable styling options. This innovative capability not only enables users to dynamically aggregate data within bubble layers based on distinct zoom levels but also empowers you to tailor the visual representation of clustered data.
Users can now effortlessly style their cluster bubbles, choosing from a range of options including bubble colors, font size and more. This level of customization enhances the user experience by allowing them to visually highlight and emphasize specific aspects of the data, creating more impactful and informative geospatial visualizations. With this added flexibility, the Power BI Azure Maps visual is poised to become an even more powerful tool for users seeking dynamic and engaging geospatial analyses.
Alerting on your Power BI reports with Data Activator
In October we announced the public preview of Data Activator. Many Power BI users have been asking for alerting capabilities within reports, and Data Activator is here to help! From the visual context menu or report toolbar you can choose ‘Set alert’. This opens a pane where you can set the measure you want to monitor and the conditions you want to detect:
When you select Continue, it will create a Data Activator reflex and connect it to the data in your Power BI dataset. Data Activator will automatically start monitoring your data and notify you when the condition is met!
If you want to further customize the trigger to watch for more complex conditions, notify different users, or even launch a Power Automate workflow, or for more information on Data Activator, you can read our documentation at https://aka.ms/dataActivatorDocs, or watch our intro video:
Data Connectivity
OneLake data hub in Power BI Desktop
Connectivity to Fabric items (Datamarts, Lakehouses, Warehouses, KQL Databases) via the OneLake data hub is now generally available, following the earlier announcement of Fabric’s general availability.
Power BI Connector Updates
Amazon Athena (Connector Update)
The Amazon Athena connector has been updated. Please find below update release notes from the Amazon team.
In this Power BI connector release, we eliminate an issue that prevents users from connecting to Athena using AAD authentication without a role parameter.
BQE Core (Connector Update)
The BQE Core connector has been updated. Please find below update release notes from the BQE Core team.
Added Target Utilization and Birth Date properties to Employee table.
Databricks (Connector Update)
The Databricks connector has been updated. Please find below update release notes from the Databricks team.
In this release we add AAD auth support for Databricks AWS workspace
Inwink (New Connector)
We are excited to announce the release of the inwink connector. Please find below release notes from the inwink team.
Visualize your inwink data and create dashboards using the inwink connector for Power BI. Combine data in inwink with data from other business apps (like Salesforce, SAP, etc) to drive better decision making and analyze the performance of your events and communities.
Starburst Enterprise (Connector Update)
The Starburst Enterprise connector has been updated. Please find below update release notes from the Starburst team.
All changes are backwards compatible:
Added support for canceling queries using “CancelQueryExplicity”.
Added “Use default proxy” checkbox.
Added support for Starburst Galaxy OAuth authentication.
Removed “Safe metadata read (optional)” from connector dialog, it’s now enabled by default for all queries.
Public change log: https://docs.starburst.io/clients/powerbi.html#version-53-public-preview-under-microsoft-certification
Snowflake (Connector Update)
The Snowflake connector has been updated to enable cancellation of a Snowflake query when the operation is cancelled in the Power BI user interface.
Eduframe Reporting (Connector Update)
The Eduframe Reporting connector has been updated. Please find below update release notes from the Eduframe team.
Added teacher enrollments.
Added teacher roles.
Added catalog variants.
Added planning event materials.
Added meeting materials.
Fixed the loading of teachers.
Service
Storytelling in PowerPoint – Suggested Content
This month we’re introducing a new feature for the Power BI add-in for PowerPoint that makes it easier to find and insert the Power BI reports you need for your presentation.
When you add the Power BI add-in to your slide, the add-in automatically scans the title of the slide and suggests Power BI content that might be relevant. For example, if the title of your slide is “Holiday campaign status”, the add-in will show you a list of Power BI reports that contain data about campaigns.
The add-in also shows you reports you’ve recently visited in Power BI, so you can easily access reports you’re working on or frequently use.
Note: If the report you wanted to add doesn’t appear in the list of suggestions, you can just copy the report’s URL and paste it into the input box, as you’ve always done.
Once you find the report you want to insert, you can choose to insert a complete page or a specific visual from the report. You can also update filters or slicers before you insert your selection. Filter and slicer values are saved together with the report or visual you picked.
And, if you have permission to grant others access to this report, you can also select the “Give people automatic access to this data” checkbox, so that when others in your org view this presentation, they’ll be able to see the data you’ve just added to the slide.
See the Storytelling in Power BI documentation here.
Ability to search and sort in paginated reports
This month, we introduced the ability to search for content across all pages of a paginated report on the PBI service. The paginated report can either be published to the service or can be authored on the PBI service. Specify any required parameters to view the report before searching. The search works across all pages of the report.
You can also sort on a column by simply clicking on the sort icon when the report is rendered on the PBI service.
Note: Both sort and search are available only in the Web layout. Read more about viewing a paginated report.
Developers
Power BI Custom Visuals Authentication API
This API allows Custom Visuals to obtain Azure Active Directory (AAD) access tokens through single sign-on (SSO), facilitating secure and efficient user-contextual operations. The API will be controlled by a global admin setting. Learn more about this API
Dynamic drill control
This feature allows the visual to enable or disable the drill feature dynamically using an API call.
- When the drill feature is enabled, all the functionalities of drilldown and expand/collapse features are available. These functionalities include API calls, context menu commands, header drill buttons, and support for hierarchy data.
- When the drill feature is disabled, these functionalities aren’t available.
Find more information here
Check out the coming soon APIs
Git integration conflict resolution
When working with Git integration in Fabric, conflicts can occur if the same items are edited in both the Fabric workspace and in the connected Git repository. To avoid any mistakes, items in conflict are marked as such, and the conflicts must be resolved before any further changes can be made. In these cases, it is up to you to decide which changes to keep and which to discard.
To make things easier, we’ve added the option to resolve conflicts and select which content to keep directly within the Fabric workspace, so that you don’t have to navigate elsewhere or revert to previous versions.
Accessed by updating items from the source control pane, resolving conflicts is as simple as selecting whether to accept incoming changes or keep the current content for each item in conflict:
Once the selections are made, all items can be merged and updated, and any further changes can be made in the workspace.
Resolving conflicts can also be done by reverting either the workspace or the Git branch to a previous state, or directly in Git if you need to further inspect what changes were made and which version to choose. Read more on how to Resolve conflicts with Git integration – Microsoft Fabric | Microsoft Learn.
New supported items for git integration and deployment pipelines
Microsoft Fabric’s lifecycle management tools, Git integration and deployment pipelines, enable seamless communication and collaboration among all members of the development team. We are thrilled to share that we’ve expanded support for additional Fabric items across these tools.
Git integration can now be used with Notebooks and Lakehouses, in addition to the already available Reports, Paginated Reports and Semantic Models. You can back up and version your notebooks and lakehouses, revert to previous stages, collaborate, or work alone using Git branches, and manage your content lifecycle entirely within Fabric.
For deployment pipelines, we’ve added support for Notebooks and Lakehouses as well, enabling you to deploy these items across different environments such as development, test, and production and enabling you to streamline your development process.
Learn more about Lakehouse and Notebook source control and deployment to understand the capabilities of each item type within Git integration and deployment pipelines.
Visualizations
New visuals in AppSource
ilionx ZorgControl KPI Gauge
Tabulator for PowerBI
Funnel Charts
Activity Gauge by Powerviz
The Activity Gauge by Powerviz is an advanced visual used to measure the progress against a set target or goal for multiple categories. It is typically presented in the form of a radial bar chart/gauge, displaying a range of values or percentages. This visual is commonly used for project management, performance tracking, and monitoring Key Performance Indicators (KPIs).
Key Features:
- Gauge Customization: Shape, style, and add icons for a unique design.
- Targets: Choose from options or use a custom field to set a target.
- Data Colors: Offers 7 schemes and 30+ color palettes.
- Center Circle: Customize center circle with text, icons, measures, and images.
- Interactive Mouseover: Hover for Interactive Info Display.
- Conditional Formatting – Create rules based on measure or category rules.
Other features included are ranking, annotation, grid view, show condition, and accessibility support.
Business Use Cases:
- Project Management: Track task progress for efficient management.
- Financial Dashboard: Gain insights into key financial metrics.
- Employee Performance: Evaluate contributions and identify areas for improvement.
Try Activity Gauge for FREE from AppSource
Check out all features of the visual: Demo file
Step-by-step documentation and Feature Video
Follow Powerviz for more updates. Learn more about visuals.
Navigate Financial Risks and Rewards with Profitbase’s Power BI Visualization
Risk is an inherent part of doing business. That doesn’t mean you have to settle for total unpredictability. Profitbase’s new Risk Matrix visualization for Power BI provides businesses with a better way to visualize and assess potential risks. This helps them handle and minimize potential damage, in addition to discovering opportunities that the business can benefit from.
Advantages of the Risk Matrix by Profitbase:
- Efficient Risk Assessment: Quickly evaluate risks by impact and likelihood.
- Focused Prioritization: Spotlight and tackle high-priority risks, focusing your resources effectively.
- Strategic Allocation: Employ our tool to guide wise resource distribution for better efficiency.
- Harmonized Communication: Establish a common understanding among teams and stakeholders with a well-recognized method.
- Flexible Customization: Personalize the matrix size, from 5×5 to 3×3 or 3×5, and colors to align with your company’s branding.
- Dynamic Visualization: Adjustable bubble sizes animate your data, offering a clear financial perspective.
View an introduction video of the visual here.
Intuitive multi-level donut charts for your reports
Drill Down Donut PRO for Power BI guarantees the best user experience when it comes to multi-level donut charts. All interactions take place as on-chart interactions, ensuring a quick and easy data exploration experience. Control every aspect of the chart through extensive customization, and even select from multiple chart types (donut, pie, gauge). Get Drill Down Donut PRO now from AppSource.
Main features:
- Adjustable “Others” slice – set the number of visible slices and group the rest.
- On-chart interactions – no need to learn hidden controls.
- Custom tooltips – select ZoomCharts custom tooltip or Power BI built-in tooltip.
- Full customization – customize every slice, label, and legend.
- Desktop and mobile device navigation – explore charts the same way on any device.
Popular use cases:
- Sales and marketing – measuring campaign performance and research results.
- Human resources – staff composition, salary distribution, performance data.
- Accounting and finance – income and expense analysis, billings, creditors, and debtors.
- Project management – risk distribution, resource allocation.
For the full list of features, see ZoomCharts’s official product page.
ZoomCharts Drill Down Visuals are known for interactive drilldowns, cross-filtering, and rich customization options. They support interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu.
Inforiver Analytics+ 3.2
Inforiver Analytics+ (50+ Charts + Cards + Tables) by Lumel is now a complete No-Code storytelling visualization platform that is designed to help customers migrate legacy dashboards from Tableau, Spotfire, Qlik, Cognos, SAP Lumira, or their static PowerPoint dashboards from Think-cell or Mekko Graphics and consolidate them with Microsoft Power BI.
Why choose Inforiver Analytics+ as your migration companion to Power BI:
- Small Multiple & Trellis everything like Tableau and Spotfire
- “Visual PivotTable” for your Hierarchical data on all three axes (x,y,z) like Spotfire.
- Split and Group Measures into Trellis panels like Tableau
- Storytelling features like Think-cell and Mekko Graphics
- Dynamic Deviations (CAGR),
- Annotations,
- Series connector lines,
- Automatic series and categorical sorting.
- Trellis Tables with Excel-like formatting
- Advanced visualizations:
- Box and Whisker Plot – dynamic and pre-calculated quartiles
- Marimekko Chart (Variable Width column or Cascade chart) – stacking and 100% scales
- Advanced Waterfall- Measure-based Interim, Stacked & Dual Breakdown, Variance, and Combination
- Special Charts for Story Telling:
- Slope graph,
- Lollipop and dumbbell,
- Dot Plot,
- Arrow & range Plots
Multivariate visualizations (Bubble/Scatter and Radar/Polar) and pie/donut types will be added to this visual soon with our vision and roadmap here.
Try Inforiver Analytics+ today after watching the 2-minute video highlights.
The Microsoft Purview hub in Fabricfor creators 
The Microsoft Purview hub is a centralized page in Fabric that helps Fabric administrators and users gain insights about their Fabric data estate. It contains reports that provide insights into sensitive data and item endorsement and serves as a gateway to more advanced capabilities in Microsoft Purview. Following the public preview release of the Microsoft Purview hub for Fabric tenant admins in May, the hub is now available in public preview for all Fabric users. 
See Microsoft Purview hub documentation here.
VideoPlayer Visual
We are excited to introduce the VideoPlayer visual for Power BI Desktop! This new custom visual allows you to seamlessly integrate videos into your Power BI reports, enhancing your data storytelling capabilities. With VideoPlayer, you can enrich your reports with dynamic multimedia content to engage your audience effectively.
Key Features
- Video Integration: Easily embed videos from various sources, such as YouTube, Vimeo (including password protected videos), and many others, directly into your Power BI reports.
- Interactive Playback: Control video playback using intuitive play, pause, and seeking functionality, ensuring your audience can interact with the content as needed.
- Customization: Customize the appearance of the VideoPlayer to match your report’s style and design, including changing the video player’s dimensions and controls.
- Data-Driven Video Selection: Utilize data-driven techniques to determine which video to play based on your dataset, making your reports more insightful.
Link: https://appsource.microsoft.com/en-us/product/power-bi-visuals/pbicraft1694192953706.videoplayer
Ridgeline Plot by Powerviz
A Ridgeline Plot or Joy plot is an advanced Power BI visual, that visually represents numeric variable distributions across multiple categories on a continuous axis. The overlapping mountain ranges allow easy comparison. Ridgeline Plot effectively highlights patterns of distribution change across different categories, providing insights into the variability and trends within the data.
Key Features:
- Multiple Distribution: Ability to plot more than one value in Measures.
- Ridge Style: Different lines, and marker styles available.
- Scaling: Control Y-axis scaling and overlaps.
- Colors: Apply color schemes across X-axis, Y-axis and based on FX rules.
- Ranking: Filter Top/Bottom N values, with “Others”.
- Reference Line: Highlight ranges or important data points on X and Y axis.
- Mode Line: Emphasize peak points and display labels.
- Custom Tooltip: Add highest, lowest, mean, and median points without any DAX.
- Themes: Save designs and share seamlessly with JSON files.
Other features included are ranking, annotation, grid view, show condition, and accessibility support.
Business Use Cases:
Time Series Data, Statistical Analysis, Distribution Data
🔗 Try Ridgeline Plot for FREE from AppSource
📊 Download: Demo file
📃 Step-by-step instructions: Documentation
💡 Feature video: Video Link
📍 Learn more about visuals: https://powerviz.ai/
✅ Follow Powerviz: https://lnkd.in/gN_9Sa6U
Core
Changes to workspace retention settings in Fabric and Power BI
Today when a Microsoft Fabric workspace is deleted, the workspace and its contents are retained for 30 days before being permanently deleted. Starting November 15th, the retention period for collaborative workspaces will be configurable from 7 to 90 days. The workspace retention setting will be enabled by default and the default retention period will be 7 days.
Fabric administrators will be able to specify the workspace retention period in the Admin Portal. After the configured retention period, workspaces will be permanently deleted. The retention period can be modified anytime.
Fabric administrators can permanently delete workspaces that are in the retention period through Admin portal. When a workspace is permanently deleted by the Fabric administrator or automatically (after the retention period), the workspace and its contents become irrecoverable.
Fabric administrators can also restore workspaces that are in the retention period through the Admin Portal. Administrators must provide a user principle that will be assigned the workspace admin role in the recovered workspace. When a workspace is recovered, the roles and permissions associated with the workspace are not recovered.
Note that this change does not affect product behavior when a Premium or Fabric capacity is deleted. When a Premium or Fabric capacity is deleted, its workspace is moved to a Shared capacity. My Workspaces will be retained for 30 days before being permanently deleted.
Fabric administrators should review the workspace retention admin setting in Fabric admin portal. In the Fabric admin portal, go to Workspace settings > Define workspace retention period.
- Turn on the setting and enter the number of days for desired retention period. You can choose anywhere from 7 to 90 days.
- When it is done, select Apply.
Learn more about workspace retention in Fabric and Power BI here: Manage workspaces – Microsoft Fabric | Microsoft Learn.
Introducing Fabric Admin APIs
Admin APIs play an important role in automating essential admin and governance tasks, including activities such as monitoring, auditing, compliance, access controls, etc. We are delighted to announce the General Availability of Microsoft Fabric, accompanied by the introduction of a new Fabric Admin APIs designed to streamline administrative tasks.
Fabric Admin APIs: Engineered for Performance and Flexibility
Our Fabric Admin APIs are designed with a focus on performance, featuring fewer call limitations and extensible interfaces to provide administrators with enhanced control. The initial set of APIs is tailored to simplify the discovery of workspaces, Fabric items, and user access details.
Roadmap: Functional Parity with PowerBI Admin APIs and Beyond
Looking ahead, our roadmap includes achieving functional parity with the existing PowerBI Admin APIs. Additionally, we aim to extend functionality to cater to a broader range of automation use cases, ensuring that Fabric Admin APIs evolve with your administrative needs.
Transitioning Seamlessly: Support for PowerBI Admin APIs
While we are committed to supporting all existing PowerBI admin APIs for continued functionality, we strongly encourage our users to transition to the new Fabric Admin APIs. As we progressively add support for existing functionalities through Fabric APIs, adopting the new set ensures a future-proof approach to administration.
Explore the Documentation: Learn More about Fabric Admin APIs
To learn more about the new APIs, check out the public documentation for Fabric Admin API.
Thank you for being a part of this exciting phase in Microsoft Fabric’s evolution.
Synapse
Data Warehouse
Automatic Log Checkpointing
We are excited to announce automatic log checkpointing for Data Warehouses!
One of our goals with the Data Warehouse is to automate as much as possible to make it easier and cheaper for you to build and use them. This means you will be spending your time on adding and gaining insights from your data instead of spending it on tasks like maintenance. As a user, you should also expect great performance which is where log checkpointing comes in!
What is Log Checkpointing and why is it important?
To understand what log checkpointing is and why it is important, we need to first talk about how tables are stored and how they are queried.
When you create a table and add data to it, the data is stored in parquet files on OneLake. Internally, there is also a log file that keeps track of which parquet files, when combined, make up the data that is in the table. These log files are internal and cannot be used directly by other engines. Instead, we automatically publish Delta Lake Logs so that other engines can directly access the right parquet files.
Now, imagine that you load data into your table every 5 minutes. That means over the course of a year, you would have loaded data to your table 105120 times. Each time, a new log file would be created that tells the system that when reading the table, the new parquet files need to be read as well. That means when reading the table, the system first needs to read all 105120 log files which are not very performant.
This is where log checkpointing comes in! As of the time of this blog, after every 10 transactions, we automatically and asynchronously create a new log file that is called a checkpoint. This file is basically a summary of all the previous log files. Now when you query the table, the system needs to read the latest checkpoint and any log files that were created after. Instead of having to read 105120 log files, we would typically need to read 10 or less files!
Conclusion
Log Checkpointing is one of the ways that we help your Data Warehouse to provide you with great performance and best of all, it involves no additional work from you! This helps give you more time to work on leveraging your Data Warehouse to gain more value and insights!
Please look forward to more announcements about more automated performance enhancements!
The Art of Data warehouse recovery within Microsoft Fabric
In the ever-evolving landscape of data management, data warehouses play a pivotal role in storing and processing vast volumes of data. This enables organizations to derive meaningful insights and make well-informed data driven decisions. The necessity for a quick and reliable data warehouse recovery solution becomes increasingly crucial to not only safeguard against data corruption but also to ensure business continuity. We are thrilled to launch the ability to perform the restore in-place of the data warehouse within Microsoft Fabric.
Restore in-place is an essential part of data warehouse recovery which allows to restore the data warehouse to a prior known reliable state by replacing or over-writing the existing data warehouse from which the restore point was created. Restore points are recovery points that can be leveraged to restore the data warehouse. From the moment the data warehouse is created, system generated restore points are created every 8 hours. There will be a total of 42 system generated restore points at any given point in time.
Furthermore, you have the flexibility to create any number of user-defined restore points aligned with your specific business or organizational recovery strategy. Both the system-generated and user-defined restore points come with a retention period of seven calendar days, after which they will expire.
In an era where resilience of data aligns with resilience of business, having a well-established restore plan for data warehouse is critical for organizations. Explore the capability to perform restore in-place using Rest API.
Data Engineering
Rich dataframe Preview in Notebook
We are thrilled to announce the upgraded display () function started public preview on Fabric Notebook, we call it Rich dataframe preview. Now when you use display () to preview your dataframe, you can easily specify the range, view the dataframe summary and column statistics, check invalid values or missing values, and preview the long cell with just a few clicks and a much nicer UX!
%%configure – Personalize Spark session in Notebook
Now you can personalize your Spark session with the magic command %%configure. Fabric notebook supports customized vCores, Memory of the Driver and Executor, Spark properties, mount points, pool, and the default lakehouse of the notebook session. They can be used in both interactive notebook and pipeline notebook activities.
You can also parameterize the session configuration, and it can allow you to pass parameters from the data pipeline and override default values in Notebook parameter cell.
Check the document to learn more about the usage.
KQL DB read/write code snippets in Notebooks
Two new code snippets are now available as part of PySpark Notebooks to help you read data from a KQL Database and write data to it.
Write data to KQL Database (Kusto): like reading, write data snippet allows you to create PySpark code to write read data to a KQL Database. All you need to provide is KQL Database URI, database name and table.
Data Science
Semantic Link Update
We’re excited to announce the latest update of Semantic Link! Apart from many improvements we also added many new features for our Power BI engineering community that you can use from Fabric notebooks to satisfy all your automation needs.
- Trigger semantic models refresh using enhanced refresh
- Update any Tabular Object Model (TOM) property via TMSL using execute_tmsl
- Capture traces to profile and optimize your DAX queries and semantic model refreshes using create_trace_connection.
- Additional data discovery methods: list_columns, list_expressions, list_hierarchies, list_partitions, list_reports, list_refresh_requests, list_translations and get_roles
- Make REST requests to the new Fabric REST APIs and existing Power BI REST APIs using our new Fabric and PowerBI clients.
- Optimized access for OneLake integrated semantic models, without putting additional load on your Power BI capacity (blog)
Real-time Analytics
Enabling Data Availability of KQL Database in OneLake
Enabling data availability of KQL Database in OneLake means customers can enjoy the best of both worlds:
- They can query the data with high performance and low latency in their KQL database.
- Query the same data in Delta Parquet via Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks and more.
This capability is part of the one logical copy promise in Fabric. It is important to note that users will only be charged once for the data storage. No extra charge is required once the data is made available also in Delta Parquet format.
Enabling the feature is quite simple. All that is needed is for the user to enable the Data Availability option of the selected KQL Database. Read this blog for more information.
Create a Notebook with pre-configured connection to your KQL DB
You can now just create a new Notebook from KQL DB editor with a pre-configured connection to your KQL DB and explore the data using PySpark. This option creates a PySpark Notebook with a ready-to execute code cell to read data from the selected KQL DB.
KQL Database schema validation
Like any piece of code, your database schema is as good as the intent you convey when you wrote it. But over time, the intent diffuses, and different priorities, authors and just plain miscommunication can diminish the quality of your code.
The new Kusto command. show database schema violations was designed to validate the current state of your database schema and find inconsistencies.
- Does the function associated with an update policy still have the same schema as the target schema or do the two drift away?
- Function A calls Function B which calls Function C: Does Function B still exist? Is the schema of each output chain in a consistent manner?
- Does your materialized view source table still exist? Is the schema compatible?
- Etc.
The command returns a table with schema validation issues. Since that is a KQL query result set, you could filter / format it using usual KQL query capabilities.
You can use this command for a spot check on your database. You could also use it in CI/CD automation: for instance, you could validate your schema in a dev environment before deploying the schema to production, as a quality gate.
Calculating distinct counts in Power BI running reports on KQL Databases
Calculating distinct counts on massive, distributed datasets is not trivial. Fabric KQL database dcount and dcountif functions use a special algorithm to return an estimate of distinct counts , even in extremely large datasets.
The new functions count_distinct and count_distinctif were recently added to calculate exact distinct counts. These two functions are much more expensive than the original ones.
To bridge the gap between these two approaches, we added a setting in Power BI to control the level of accuracy of dcount. This will allow users to optimize the accuracy of the dcount function vs the runtime of the query. The setting also allows calling count distinct when absolute accuracy is needed.
Read this for more information.
Data Factory
Data pipelines
Azure Databricks Activity
You can now create powerful data pipeline workflows that include Notebook executions from your Azure Databricks clusters using Fabric Data Factory. Add a Databricks activity to your pipeline, point to your existing cluster, or request a new cluster, and Data Factory will execute your Notebook code for you!
Connection editing in pipeline editor now supported
You can now edit your existing data connections while you are designing your pipeline without leaving the pipeline editor!
When setting your connection, simply click Edit and a pop up will appear.
Read and Write to the Fabric Lakehouse using Azure Data Factory (ADF)
You can now read and write data in the Microsoft Fabric Lakehouse from ADF (Azure Data Factory). Using either Copy Activity or Mapping Data Flows, you can read, write, transform, and process data using ADF or Synapse Analytics, currently in public preview.
Set activity state for easy pipeline debugging
In Fabric Data Factory data pipelines, you can now set an activity’s state to inactive so that you can save your pipeline even with incomplete, invalid configurations. Think of it as “commenting out” part of your pipeline code.
