How to Set Dynamic File Names in Azure Data Factory: A Step-by-Step Guide

Creating dynamic file names is essential in data integration workflows, especially when managing large datasets in cloud platforms like Azure Data Factory (ADF) or Azure Synapse Pipelines. This guide will walk you through setting up dynamic file names using expressions, making your data processes more organized and efficient.

Why Use Dynamic File Names?

Dynamic file names ensure each file generated during a data pipeline run is unique. This is particularly important when working with frequently updated data, such as logs, daily reports, or data exports. It helps avoid overwriting files and makes tracking and managing data easier.

Step 1: Understanding Dynamic File Name Expressions

To generate dynamic file names, we use expressions that combine text, date, time, and other pipeline parameters. The following JavaScript expression is commonly used to add the current date and time to file names:

javascript:

@concat('customer_data_', formatDateTime(utcNow(), 'yyyy-MM-dd_HH-mm-ss'), '.csv')

This expression will create file names like:

customer_data_2024-09-17_14-45-00.csv

Step 2: Setting Up Dynamic File Names in Azure Data Factory

1. Navigate to the Copy Data Tool:

• In the Azure Data Factory, go to your pipeline and select the activity where you want to configure the file name, typically under the Destination tab.

2. Configure File Name with Dynamic Content:

• Click on the File name field in the destination settings.
• Select Add dynamic content to open the expression editor.

3. Use Date and Time in File Names:

• To create a file name that includes the current date and time, use the following expression: @concat('customer_data_', formatDateTime(utcNow(), 'yyyy-MM-dd_HH-mm-ss'), '.csv')
• This will generate unique file names such as: customer_data_2024-09-17_14-45-00.csv

Step 3: Adding Pipeline Parameters to File Names

If you need to include specific identifiers like a customer ID, you can modify the expression to include pipeline parameters:

@concat('customer_', pipeline().parameters.CustomerID, '_', formatDateTime(utcNow(), 'yyyy-MM-dd_HH-mm-ss'), '.csv')

This will create a file name like:

codecustomer_12345_2024-09-17_14-45-00.csv

Where 12345 is dynamically passed from the pipeline parameter.

Step 4: File Format Considerations

When setting dynamic file names, it’s essential to ensure the correct file extension is used based on your data format. For example:

• CSV Files: Use .csv at the end of your expression.
• Parquet Files: Change the extension to .parquet: @concat('customer_data_', formatDateTime(utcNow(), 'yyyy-MM-dd_HH-mm-ss'), '.parquet')

This will create files like:

customer_data_2024-09-18_14-45-00.parquet

Step 5: Applying and Testing the Expression

1. Apply the Expression:

• Once you’ve configured the dynamic expression, click Finish to apply it.

2. Test the Pipeline:

• Run the pipeline to ensure the dynamic file naming works as expected. Check the output folder to confirm that files are being saved with the correct dynamic names.

Example Scenario

Imagine you’re working on a project where you need to export customer data to Azure Data Lake in the following path:

 datalake/bronze/public/customer/

By setting up dynamic file names, each file will look like:

customer_data_2024-09-18_14-45-00.parquet

This approach makes it easy to identify when each file was generated, ensuring proper data organization and management.

Benefits of Using Dynamic File Names

1. Prevents Overwriting: Dynamic names ensure that each run of your pipeline generates a unique file, avoiding overwriting previous data.
2. Improves Organization: Including date, time, and identifiers in file names helps keep data organized and easy to navigate.
3. Enhances Tracking: Dynamic names provide context, making it easier to track and audit data for analysis and compliance.

Conclusion

Using dynamic file names in Azure Data Factory is a powerful way to manage your data workflows more effectively. By incorporating expressions that include dates, times, and other parameters, you ensure each file is uniquely identifiable, enhancing both data organization and operational efficiency.

This guide provides the foundational steps to implement dynamic file names in your data pipelines. Experiment with different expressions to tailor file naming conventions that best suit your project needs!