Microsoft Fabric Updates Blog

Shortcuts to Google Cloud Storage, now available in Public Preview

We are excited to announce that you can now create OneLake shortcuts to your Google Cloud Storage (GCS) buckets. With the addition of GCS, you can now utilize cross-cloud shortcuts to analyze your data across all three major cloud platforms. Shortcuts in OneLake allow you to connect to your existing data through a single unified name space without having to copy or move data. Just open a Lakehouse in Fabric, create a shortcut to GCS, AWS S3 or ADLS Gen2 and immediately start analyzing your data through Spark, SQL and Power BI.

A diagram of a cloud computing system

Description automatically generated

How to create GCS shortcuts

Open a Lakehouse in Fabric and create a new shortcut. If your target data is already in the Delta/Parquet format, create the shortcut in the “Tables” section of the Lakehouse. This will allow metadata synchronization across Spark, SQL and Power BI. If you target data is in another format, create the shortcut in the “Files” section of the Lakehouse.

Screenshot of right click context menu showing where to select New shortcut from the Lake view.

Under External Sources, select Google Cloud Storage.

Screenshot of the New shortcut window showing the available shortcut types. The option titled Google Cloud Storage is highlighted.

Configure your connection settings.

Screenshot of the New shortcut window showing the Connection settings and Connection credentials.

In the connection URL, you can specify either the global endpoint for the GCS storage service or a bucket specific endpoint.

  • Global endpoint: https://storage.googleapis.com
  • Bucket specific endpoint: https://<BucketName>.storage.googleapis.com

GCS shortcuts utilize HMAC keys for authorization with Google. HMAC keys can be created for both user accounts and service accounts. Follow the guide here to create an HMAC key: Manage HMAC keys for service accounts.

The account associated with the HMAC key must have permission to access the data within GCS. If you provided a bucket specific URL, the account needs the following permissions:

  • storage.objects.get
  • stoage.objects.list

If you provided the global endpoint in the URL, the account would need the following additional permissions:

  • storage.buckets.list

Once your connection is configured, you can browse your storage account in GCS.

Screenshot of the storage browse window with multiple folders selected.

In this view, you can select multiple target locations to create shortcuts to.

The review page allows you to see all of the selections you made.

Screenshot of shortcut review page with options to rename and delete shortcuts.

Here you can use the pencil button to rename any of your selections. The trash button can be used to remove any unwanted selections. Click create and your shortcuts will appear in your lakehouse.

Screenshot showing a Lake view list of tables that display the shortcut symbol.

Now start analyzing your data! Open the SQL endpoint and start writing new queries. Create a new Notebook and analyze your data with Spark. Open the semantic model and create a new Power BI report.

Caching

Just like AWS S3 shortcuts, GCS shortcuts also support caching. With caching enabled, egress costs are greatly reduced. As files are read through the GCS shortcut, the files are stored in a cache for the Fabric workspace. Subsequent read requests are served from cache rather than GCS directly.

Cache enablement is configured at a workspace level. To enable shortcut caching, workspace admins can start by opening a Fabric workspace and selecting “Workspace settings

In the workspace settings panel, select the “OneLake” tab. Switch the toggle for “Enable Cache for Shortcuts” to “On”. Then click the “Save” button.

Screenshot of workspace settings panel with OneLake tab selected.

Caching is now enabled for all GCS, S3 and S3 compatible shortcuts in that workspace. To learn more about shortcut caching see our blog here: Reduce egress costs with S3 shortcuts in OneLake | Microsoft Fabric Blog | Microsoft Fabric.

References

相關部落格文章

Shortcuts to Google Cloud Storage, now available in Public Preview

10月 7, 2024 作者: Alex Lin

Introducing Managed VNet Support for Fabric Eventstream! By creating a Fabric’s Managed Private Endpoint, you can now securely connect Eventstream to your Azure services, such as Azure Event Hubs or IoT Hub, within a private network or behind a firewall. This integration ensures your data is securely transmitted over a private network, enabling you to … Continue reading “Secure Data Streaming with Managed Private Endpoints in Eventstream (Preview)”

10月 4, 2024 作者: Jason Himmelstein

We had an incredible time in our host city of Stockholm for FabCon Europe! 3,300 attendees joined us from our international community, and it was wonderful to meet so many of you in person. Throughout the week of FabCon Europe, our teams published a wealth of valuable content, and we want to ensure you have … Continue reading “Fabric Community Conference Europe Recap”