Distributed Task Execution

Nx Cloud provides config-free distributed task execution (DTE). Nx can run any task graph on multiple machines instead of running them on a single machine.

Imagine you are running nx affected --build --parallel. Normally this command runs the build target for the affected projects in parallel on the same machine. However, if you enable distributed task execution, the command will send the task graph to Nx Cloud. Nx Cloud agents will then pick up the tasks to execute them.

Note that this happens transparently. If an agent needs the output of lib1 to build app1, and some agent built lib1, the first agent is going to fetch the needed output before running the build task.

As agents complete tasks, the main job where you invoked nx affected --build will start receiving created files and terminal outputs. After nx affected --build completes, the machine will have the build files and all the terminal outputs as if it ran it locally.

Enabling Distributed Task Execution

Let's use Circle CI as an example.

version: 2.1 jobs:   agent: steps: - checkout - run: npm install - run: npx nx-cloud start-agent pr-main: environment: NX_CLOUD_DISTRIBUTED_EXECUTION: 'true' steps: - checkout - run: npm install - run: npx nx-cloud start-ci-run - run: npx nx affected --target=build --parallel --max-parallel=3 - run: npx nx affected --target=test --parallel --max-parallel=2 - run: npx nx affected --target=lint --parallel --max-parallel=4 - run: npx nx-cloud stop-all-agents workflows:   build: jobs: - agent: name: 'agent1' - agent: name: 'agent2' - agent: name: 'agent3' - pr-main

To enable distributed task execution, the following has to be true:

  1. NX_CLOUD_DISTRIBUTED_EXECUTION has to be set to true.

  2. Agents must be started using the nx-cloud start-agent command, and must be stopped at the end using nx-cloud stop-all-agents.

The configuration above uses 3 agents to run all builds, tests, and lints.

Note that only cacheable operations can be distributed because they have to be replayed on the main job.

You don't have to distribute every single command. For instance, it's pretty common to send test coverage reports from a single machine. You can do it as follows:

- run: npx nx affected --target=test --parallel --max-parallel=2 - run: NX_CLOUD_DISTRIBUTED_EXECUTION=false npx nx affected --target=send-coverage-reports --parallel

Distributed task execution will copy all artifacts from agents to the main job, so after the affected --target=test command completes, all the coverage reports will be on the main job.

Note, the --parallel and --max-parallel flags are propagated to Nx Cloud agents. In this example above each agent will test 2 projects in parallel.


In the example above, the CircleCI config creates one main job and three agent jobs. All four jobs have the same environment and the same source code. The four jobs will start around the same time. And, once the main job completes, all the agents will be stopped.

It's important to note that an Nx Cloud agent isn't a machine but rather a long-running process that runs on a machine. In the example above, the CircleCI config provisions three machines that run nx-cloud start-agent. Nx Cloud does not start agents. It simply coordinates work between the agents that you start in your CI. So you are in control of how many agents you want to use and where they are located.

Additional Configuration Options

  • Setting NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT to say 8 will tell Nx Cloud assume that there are 8 agents running.

  • Setting NX_CLOUD_DISTRIBUTED_EXECUTION_STOP_AGENTS_ON_FAILURE to false will tell Nx Cloud not to stop agents if a command fails. You need to make sure to invoke nx-cloud stop-all-agents even if CI fails.

Distributed Task Execution and Sharding

The Nx Cloud distributed task execution model is akin to what is used at Google or Facebook. It scales better and provided more flexiblity than sharding tasks across agents. Read more about Distributed Task Execution and how it compares to binning/sharding.

Select an article to learn more about Nx Cloud