Sign up for your free trial and begin automating your processes today.

Asynchronous n8n Workflows: A Parallel Processing POC

Achieve parallel processing in n8n. Learn to use a powerful asynchronous workflow pattern for long-running tasks with this detailed POC.

n8n is an incredibly powerful tool for workflow automation, but out of the box, its execution model is sequential. For most use cases, this is exactly what you want. However, what if you have multiple long-running tasks that you need to kick off and then wait for their results? For example, processing a large batch of items, each of which takes several minutes to complete? Waiting for each item to finish one by one can be highly inefficient.

Fortunately, for advanced users, there is a powerful pattern to achieve a form of asynchronous, parallel execution using built-in n8n nodes. This article will walk you through a proof-of-concept (POC) that demonstrates how to spin up several sub-workflows without waiting for them to complete, and then gracefully collect their results.

The Core Concept

The magic of this approach lies in a specific configuration of two nodes:

  1. Execute Workflow: By default, this node waits for the sub-workflow to finish. However, if you disable the “Wait For Sub-workflow Completion” option, it will run the sub-workflow and immediately proceed to the next node in the main workflow.

  2. Wait: This node can pause a workflow execution and later be resumed by an external trigger, like a webhook call.

By combining these two, the main workflow can kick off multiple sub-workflows and then wait for each one to “check back in” with its result via a webhook.

Workflow 1: The Main Orchestrator

This is the main workflow that will initiate the parallel tasks. Its job is to generate the data for the sub-workflows, call them, wait for all of them to report back, and finally, process the results.

Nodes:

  1. Execute Workflow:

    • Workflow Name: Choose from the list of available sub-workflows.

    • Wait For Sub-workflow Completion: Set this to false. This is the most crucial step!

    • Parameters:

      • wait: pass the amount of secons to wait.

      • webhook: The webhook URL should be taken from $execution.resumeUrl with a unique suffix, allowing us to link the result back to the correct “wait” step.

  2. Wait: This is where the main workflow will pause.

    • Resume: Select On Webhook Call.

    • Webhook Suffix: This must match the suffix passed to the sub-workflow.

  3. Merge: After the Wait nodes resume and return their results, we’ll use a Merge node to combine all the returned data into a single output. Set the Mode to Append to gather the output from all the asynchronous calls.

  4. Summarize: Finally, we’ll use a Summarize node to calculate the sum of the returned values and confirm that all results were successfully received.

    • Aggregation: Set this to Sum.

    • Field: Select the field containing the result, which in this case is body.result.

The entire n8n Main Orchestrator workflow showing the Execute Workflow, Wait, Merge, and Summarize nodes.

A close-up of the Execute Workflow node parameters, with “Wait For Sub-workflow Completion” set to false.

A close-up of the Wait node parameters, with “Resume” set to “On Webhook Call” and the “Webhook Suffix” field.

Workflow 2: The Asynchronous Worker

This sub-workflow is the actual worker that performs the long-running task and reports back to the main workflow.

Nodes:

  1. Execute Sub-workflow Trigger: This is the entry point for the sub-workflow. It receives the wait and webhook parameters from the main workflow.

  2. Wait: This node simulates the workload.

    • Resume: Select After Time Interval.

    • Wait Amount: Use the value passed from the main workflow ({{ $json.wait }}).

    • Wait Unit: Choose Seconds.

  3. HTTP Request: This node is the “callback” that resumes the main workflow.

    • URL: Use the webhook URL passed from the main workflow ({{ $json.webhook }}).

    • Body Parameters: The wait value multiplied by two will be sent in the request body as a JSON key named result to confirm the sub-workflow processed the data correctly.

    • Retry on Error: Crucial for robustness. Connect the Error output of this node back to a new Wait node with a small delay (e.g., 5 seconds). This will create a retry loop in case the main workflow’s webhook is temporarily unavailable. The output of this Wait node should be merged back into the HTTP Request node’s input.

The entire n8n Asynchronous Worker workflow, showing the Execute Sub-workflow Trigger, Wait, and HTTP Request nodes with an error retry loop.

A close-up of the Execute Sub-workflow Trigger node, highlighting the ‘wait’ and ‘webhook’ parameters.

A close-up of the HTTP Request node’s Body Parameters section, showing the result field with the expression $json.wait * 2.

Testing

For demonstration purposes, you can place the main workflow and the worker sub-workflow on the same canvas. The main workflow will execute the worker sub-workflow twice. For the first call, pass a wait parameter of 5 and a webhook suffix of call1. For the second call, pass a wait parameter of 3 and a suffix of call2. After execution, the Summarize node in the main workflow will show a result of 16, which is the expected sum of the two returned values: (5 * 2) + (3 * 2) = 16.

A side-by-side view of the two workflows, Main and Worker, on the same canvas after a successful execution, with the green checkmarks indicating completion.

The output data of the Summarize node, showing a final value of 16.

Conclusion

This pattern provides a powerful way to handle concurrent tasks in n8n. While it’s a manual workaround, it offers a robust solution for use cases that require parallel execution without waiting for one step to finish before starting the next. It’s a great example of how you can extend n8n’s capabilities with a bit of creative node configuration. This POC can serve as a foundation for more complex, real-world asynchronous processing jobs.

n8n is a trademark of n8n GmbH. This site is not officially associated with n8n GmbH.
Built with Hugo
Theme Stack designed by Jimmy