Parallel Processing in Azure Data Factory

If you’re using Azure Data Factory and make use of a ForEach activity in your data pipeline, in this post I’d like to tell you about a simple but useful feature in Azure Data Factory. This tip will give you the ability if you’re using a ForEach activity within a data pipeline to decide whether to process each item in your ForEach loop sequentially or in parallel.

In the screenshot below, you’ll see I’m in a pipeline and I’ve configured a ForEach loop. Notice the box with the sequential option. If you leave that box unchecked, Azure Data Factory will process each item in the ForEach loop in parallel up to the limits of the Data Factory engine.

In most cases where we have a looping mechanism, including tools like SSIS, each item in the loop was processed in sequence and in a certain order. At times this can be pretty handy, and you can check the box and have that option.

Other times, the order doesn’t matter and in fact, it may be better to spread the workload and do multiple things at one time.

I’ve recently used this quick but powerful feature and wish we had this option before in SSIS, but I’m glad to see it now in Azure Data Factory.

So, again, a quick feature tip that I hope you’ll find handy and helpful. If you have any questions about Azure Data Factory or about how to use the Azure platform in your organization, you’re in the right place. We are your best resource for anything within Azure and we’re here to help.