Microsoft Fabric Updates Blog

Shortcuts to Google Cloud Storage, now available in Public Preview

We are excited to announce that you can now create OneLake shortcuts to your Google Cloud Storage (GCS) buckets. With the addition of GCS, you can now utilize cross-cloud shortcuts to analyze your data across all three major cloud platforms. Shortcuts in OneLake allow you to connect to your existing data through a single unified name space without having to copy or move data. Just open a Lakehouse in Fabric, create a shortcut to GCS, AWS S3 or ADLS Gen2 and immediately start analyzing your data through Spark, SQL and Power BI.

A diagram of a cloud computing system

Description automatically generated

How to create GCS shortcuts

Open a Lakehouse in Fabric and create a new shortcut. If your target data is already in the Delta/Parquet format, create the shortcut in the “Tables” section of the Lakehouse. This will allow metadata synchronization across Spark, SQL and Power BI. If you target data is in another format, create the shortcut in the “Files” section of the Lakehouse.

Screenshot of right click context menu showing where to select New shortcut from the Lake view.

Under External Sources, select Google Cloud Storage.

Screenshot of the New shortcut window showing the available shortcut types. The option titled Google Cloud Storage is highlighted.

Configure your connection settings.

Screenshot of the New shortcut window showing the Connection settings and Connection credentials.

In the connection URL, you can specify either the global endpoint for the GCS storage service or a bucket specific endpoint.

  • Global endpoint: https://storage.googleapis.com
  • Bucket specific endpoint: https://<BucketName>.storage.googleapis.com

GCS shortcuts utilize HMAC keys for authorization with Google. HMAC keys can be created for both user accounts and service accounts. Follow the guide here to create an HMAC key: Manage HMAC keys for service accounts.

The account associated with the HMAC key must have permission to access the data within GCS. If you provided a bucket specific URL, the account needs the following permissions:

  • storage.objects.get
  • stoage.objects.list

If you provided the global endpoint in the URL, the account would need the following additional permissions:

  • storage.buckets.list

Once your connection is configured, you can browse your storage account in GCS.

Screenshot of the storage browse window with multiple folders selected.

In this view, you can select multiple target locations to create shortcuts to.

The review page allows you to see all of the selections you made.

Screenshot of shortcut review page with options to rename and delete shortcuts.

Here you can use the pencil button to rename any of your selections. The trash button can be used to remove any unwanted selections. Click create and your shortcuts will appear in your lakehouse.

Screenshot showing a Lake view list of tables that display the shortcut symbol.

Now start analyzing your data! Open the SQL endpoint and start writing new queries. Create a new Notebook and analyze your data with Spark. Open the semantic model and create a new Power BI report.

Caching

Just like AWS S3 shortcuts, GCS shortcuts also support caching. With caching enabled, egress costs are greatly reduced. As files are read through the GCS shortcut, the files are stored in a cache for the Fabric workspace. Subsequent read requests are served from cache rather than GCS directly.

Cache enablement is configured at a workspace level. To enable shortcut caching, workspace admins can start by opening a Fabric workspace and selecting “Workspace settings

In the workspace settings panel, select the “OneLake” tab. Switch the toggle for “Enable Cache for Shortcuts” to “On”. Then click the “Save” button.

Screenshot of workspace settings panel with OneLake tab selected.

Caching is now enabled for all GCS, S3 and S3 compatible shortcuts in that workspace. To learn more about shortcut caching see our blog here: Reduce egress costs with S3 shortcuts in OneLake | Microsoft Fabric Blog | Microsoft Fabric.

References

Related blog posts

Shortcuts to Google Cloud Storage, now available in Public Preview

May 23, 2024 by Will Thompson (HE/HIM)

Microsoft Fabric’s new Real-Time hub makes it easier than ever to discover, preview, and use streaming data from around your organization. But the data in these streams is only as useful as the actions and changes it can drive in your business. Data Activator provides a no-code experience for automatically taking actions when patterns or … Continue reading “Alerting and acting on data from the Real-Time hub”

May 23, 2024 by Salil Kanade

We are excited to announce the public preview release of Copilot for Data Warehouse in Microsoft Fabric, a groundbreaking AI assistant designed to transform your data warehousing tasks. Data warehouse development can be daunting for SQL developers, especially under tight timelines where insights are needed “yesterday”.  Developers may spend hours writing code, building schemas, documentation, … Continue reading “Announcing the Public Preview of Copilot for Data Warehouse in Microsoft Fabric”