Unleashing the Power of mssparkutils in Microsoft Fabric Notebooks

Microsoft Fabric notebooks provide a powerful and versatile platform for data engineering and data science tasks. To enhance the capabilities of these notebooks, Microsoft has introduced mssparkutils, a built-in package designed to simplify common tasks and streamline your workflow. In this blog post, we’ll explore the functionalities of mssparkutils and how you can leverage its power to optimise your Microsoft Fabric notebook experience.

What is mssparkutils?

Microsoft Spark Utilities (mssparkutils), recently renamed to notebookutils but still backward compatible, is a powerful built-in package within Microsoft Fabric notebooks. It provides a set of utilities and enhancements built on top of Apache Spark, specifically tailored for the Microsoft Fabric environment. mssparkutils is designed to help you perform common tasks with ease, including:

• File System Utilities: Manipulate and interact with file systems like Azure Data Lake Storage (ADLS) and Azure Blob Storage directly from your notebooks.
• Notebook Utilities: Chain and orchestrate notebook executions, enabling complex workflows and modular code design.
• Credential Utilities: Securely manage and access secrets and credentials, enhancing the security of your notebooks and data pipelines.

Key Features of mssparkutils

Let’s delve into the key features of mssparkutils and explore how they can empower your Microsoft Fabric notebooks.

1. File System Utilities (mssparkutils.fs)

The mssparkutils.fs module provides a comprehensive set of functions for interacting with file systems. It allows you to perform various file and directory operations, such as:

• File and directory manipulation: Move, copy, delete, and create files and directories within your storage accounts.
• Reading and writing files: Read data from files and write data to files in various formats.
• Mounting file systems: Mount external file systems to your notebook environment, providing seamless access to data.

Here are some examples of mssparkutils.fs functionalities:

• Listing files in a directory: mssparkutils.fs.ls("abfss://your-storage-account.dfs.core.windows.net/path/to/your/directory/")
• Copying files: mssparkutils.fs.cp("abfss://your-storage-account.dfs.core.windows.net/path/to/source/file.txt", "abfss://your-storage-account.dfs.core.windows.net/path/to/destination/file.txt")
• Moving files efficiently: For enhanced performance, especially with large datasets, use fastcp: mssparkutils.fs.fastcp("abfss://your-storage-account.dfs.core.windows.net/path/to/source/large_file.parquet", "abfss://your-storage-account.dfs.core.windows.net/path/to/destination/large_file.parquet")
• Appending content to a file: mssparkutils.fs.append("abfss://your-storage-account.dfs.core.windows.net/path/to/output/file.txt", "Content to append", True) # Creates the file if it doesn't exist

2. Notebook Utilities (mssparkutils.notebook)

The mssparkutils.notebook module empowers you to orchestrate and chain notebook executions, enabling you to build modular and reusable data workflows. Key functionalities include:

• Running a notebook: Execute another notebook from within your current notebook, passing parameters and capturing exit values.
• Running multiple notebooks: Execute multiple notebooks in parallel or with predefined dependencies, optimizing resource utilization and workflow efficiency.
• Exiting a notebook: Programmatically exit a notebook and return a value, enabling conditional workflow execution.

Here are some examples of mssparkutils.notebook functionalities:

• Running a single notebook: mssparkutils.notebook.run("MyChildNotebook", 90, {"input_param": "example_value"}) # Runs "MyChildNotebook" with a 90-second timeout and input parameter
• Running multiple notebooks in parallel or with dependencies: Define a Directed Acyclic Graph (DAG) in JSON format to specify notebook execution order and dependencies, then use runMultiple: DAG = { "activities": [ { "name": "Notebook1", "notebookPath": "Notebook1" }, { "name": "Notebook2", "notebookPath": "Notebook2", "dependencies": ["Notebook1"] # Notebook2 runs after Notebook1 completes } ] } mssparkutils.notebook.runMultiple(DAG)
• Exiting a notebook with a value: mssparkutils.notebook.exit("Notebook completed successfully") # Exits the notebook and returns the message

3. Credential Utilities (mssparkutils.credentials)

The mssparkutils.credentials module enhances the security of your notebooks by providing functionalities for managing and accessing secrets and credentials. It allows you to:

• Retrieve secrets from Azure Key Vault: Securely access secrets stored in Azure Key Vault without hardcoding them in your notebooks.
• Get access tokens: Obtain access tokens for various Azure services, enabling secure authentication and authorization.

Here’s an example of retrieving a secret from Azure Key Vault:

Python

mssparkutils.credentials.getSecret("YourKeyVaultName", "YourSecretName") # Retrieves the secret from Azure Key Vault

Benefits of Using mssparkutils

Leveraging mssparkutils in your Microsoft Fabric notebooks offers numerous benefits:

• Simplified File Operations: mssparkutils.fs streamlines file system interactions, making data handling more efficient and less complex.
• Enhanced Workflow Orchestration: mssparkutils.notebook enables you to create robust and modular data workflows by chaining and orchestrating notebook executions.
• Improved Security: mssparkutils.credentials enhances security by providing secure access to secrets and credentials, reducing the risk of exposing sensitive information.
• Optimized Resource Utilization: The runMultiple function optimizes the use of Spark compute resources by allowing parallel notebook executions.
• Increased Productivity: By simplifying common tasks and providing powerful utilities, mssparkutils boosts your productivity and accelerates your data engineering and data science workflows.

Get Started with mssparkutils

Getting started with mssparkutils is straightforward. The package is readily available in Microsoft Fabric notebooks, and you don’t need to install anything. Simply import the mssparkutils library and start exploring its functionalities.

Python

import mssparkutils

To discover the available functions within each module, use the help() function:

Python

mssparkutils.fs.help()
mssparkutils.notebook.help()
mssparkutils.credentials.help()

Conclusion

mssparkutils is a game-changing package for Microsoft Fabric notebooks, offering a wealth of utilities to simplify common tasks, enhance security, and optimize workflows. By unleashing the power of mssparkutils, you can significantly improve your productivity and build more robust and efficient data solutions in Microsoft Fabric.

We encourage you to explore the capabilities of mssparkutils and incorporate it into your Microsoft Fabric notebooks to experience its transformative impact firsthand. Remember to check the official Microsoft Fabric documentation for the most up-to-date information and detailed usage examples.

Important Note: As of January 2025, mssparkutils has been renamed to notebookutils. While mssparkutils remains backward compatible, it is strongly recommended to migrate to notebookutils to ensure continued support and access to the latest features and updates.

Start leveraging the power of notebookutils (formerly mssparkutils) today and unlock a new level of efficiency and productivity in your Microsoft Fabric notebooks!