Splitting a long pipeline into parallel jobs

Pipelines become serial the way code becomes tangled: one step at a time, each added where the previous one ended, without anyone asking whether it needed to wait. The result is a twenty-minute pipeline whose critical path is eight minutes, with twelve minutes of waiting for steps that had no dependency on each other. Splitting it is usually straightforward — the hard parts are not paying for dependency installation several times and knowing whether the change helped. This recipe covers both, within CI caching and runner performance.

When to use this approach

  • Total pipeline duration is much larger than the longest single step.
  • Lint, unit tests and build all run one after another with no real ordering need.
  • One slow suite dominates and could be sharded.
  • Runner capacity is available — parallelism you cannot schedule is not parallelism.
  • If the pipeline is already at its critical path, splitting further adds overhead per job and makes it slower.

Step 1 — Draw the real dependency graph

Ask of each step: what does it read that a previous step wrote? Anything with no answer can start immediately.

# Current structure, in order
gh run view "$RUN_ID" --json jobs \
  --jq '.jobs[] | {name, started: .startedAt, finished: .completedAt}'
# The arithmetic that reveals the opportunity
gh run view "$RUN_ID" --json jobs --jq '
  {wall: ((.jobs | map(.completedAt | fromdate) | max)
        - (.jobs | map(.startedAt | fromdate) | min)),
   serial_sum: ([.jobs[] | ((.completedAt | fromdate) - (.startedAt | fromdate))] | add)}'

If wall and serial_sum are close, nothing is running in parallel. If wall is much smaller, you are already parallel and the remaining time is the critical path.

Declared dependencies against real onesLinting reads only the source and needs nothing from the build. Unit tests need dependencies installed but not the compiled bundle. Only packaging genuinely needs both the build output and a green test run, so three of the four steps can start at once.Declared (serial)Actually requiredlintafter install and buildafter installunit testsafter lintafter installbuildafter testsafter installpackageafter buildafter build and teststhree of four steps were waiting on something they never read

Step 2 — Install once and share the result

The naive split makes every job install dependencies, which can cost more than the serialisation it removed.

jobs:
  setup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - uses: actions/upload-artifact@v4
        with: { name: node-modules, path: node_modules, retention-days: 1 }

  lint:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with: { name: node-modules, path: node_modules }
      - run: npm run lint
# Verification: only the setup job should show an install step taking real time
gh run view "$RUN_ID" --json jobs \
  --jq '.jobs[] | {job: .name, steps: [.steps[] | select(.name | test("ci|install")) | .name]}'

Whether artefact transfer beats a cache restore depends on the size of the tree; measure both. For large native dependency trees the cache is usually faster, and for small ones the artefact is.

Step 3 — Shard the suite that dominates

When one suite is the critical path, splitting the pipeline no longer helps — splitting the suite does.

  test:
    needs: setup
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npx jest --shard=${{ matrix.shard }}/4 --ci
# Verification: shards should finish within a minute of each other
gh run view "$RUN_ID" --json jobs \
  --jq '[.jobs[] | select(.name | startswith("test")) |
         {name, seconds: ((.completedAt | fromdate) - (.startedAt | fromdate))}]'

fail-fast: false matters here. With the default, one failing shard cancels the others and you learn about one failure per run instead of all of them, which turns a single fix into four round trips.

Critical path as the pipeline is restructuredSerialised, the pipeline takes the sum of its steps. Running independent jobs in parallel drops it to the longest of them, which is the test suite. Sharding that suite four ways cuts it again, until the fixed setup cost becomes the floor.minutes of wall-clock timefully serial19 minindependent jobs parallel9 minplus four test shards5 mintheoretical floor (setup)3 mineach step approaches the fixed cost, which is what caching attacks instead

Step 4 — Avoid the traps that make parallel slower

Three patterns routinely make a split pipeline slower than the serial one it replaced.

# 1. Checkout in every job on a large repository — trim it
      - uses: actions/checkout@v4
        with: { fetch-depth: 1 }

# 2. Matrix jobs that each restore a large cache — measure the restore
#    against simply running two shards in one job.

# 3. A fan-in job that waits for the slowest branch of the graph while
#    holding a runner. Use `needs` precisely rather than listing everything.
# Which job is the critical path?
gh run view "$RUN_ID" --json jobs \
  --jq '[.jobs[] | {name, seconds: ((.completedAt | fromdate) - (.startedAt | fromdate))}]
        | sort_by(-.seconds) | .[0:3]'

Step 5 — Re-measure, and keep the measurement

The failure mode of pipeline optimisation is a restructure that feels faster and is not. Record the numbers before and after.

# Median duration over the last thirty runs, comparable before and after
gh run list --workflow ci.yml --limit 30 --json createdAt,updatedAt \
  --jq '[.[] | ((.updatedAt | fromdate) - (.createdAt | fromdate))] | sort | .[15]'
How the restructured pipeline runsThe setup job installs once and publishes the dependency tree. Lint, build and four test shards start together the moment it finishes. Packaging waits only for build and tests, which is the one genuine dependency in the graph.setuplinttest shardspackagedependencies readydependencies readylint result (not required)all shards greenartefact publishedlint informs but does not gate packaging — declaring that removes a wait

Validation checklist

Frequently Asked Questions

How many shards is too many?

When the per-shard fixed cost — runner acquisition, checkout, dependency restore — approaches the per-shard test time, adding shards stops helping and starts costing. For most suites that is somewhere between four and eight. Measure the shard duration spread; when the fastest shard is mostly setup, stop.

Should lint block the tests?

No. They read the same source and answer different questions, so running them in sequence adds latency without adding information. Both should gate the merge; neither should gate the other.

Does parallelism increase cost?

Total machine time goes up slightly because of duplicated setup, while wall-clock time goes down substantially. Whether that trade is right depends on whether you are paying for minutes or for people waiting — usually the second is far more expensive.