Why Your CI/CD Pipeline Is Slower Than It Should Be

A 45-minute CI pipeline is a tax on every engineer every day. If your team runs 50 builds per day and each build takes 45 minutes instead of 12 minutes, that is 27.5 hours of wall-clock time lost daily, plus the context-switching cost of engineers waiting for CI before they can merge and move to the next task. CI pipeline performance is engineering leverage, and most teams leave significant time on the table.

"CI pipeline speed is a proxy for how seriously an engineering organization takes developer experience. Every minute of unnecessary CI wait time is a minute of context switching, a minute of multitasking, a minute of reduced flow state. The cumulative cost is measured in engineering productivity, not pipeline minutes."

— Nicole Forsgren, PhD, VP of Research and Strategy, GitHub, co-author of Accelerate (2023)

Parallelization: the highest-leverage intervention

If your pipeline runs tests sequentially, fixing that is the highest-leverage change you can make. Most test suites can be split across multiple runners with 80 to 90 percent efficiency. A test suite that takes 30 minutes on one runner might take 5 minutes across eight runners, with the remaining time being setup overhead rather than test execution.

Test splitting strategies: random split by test file is simple and usually good enough. Timing-based split, which splits to equalize execution time across runners, is better if your tests have high variance in execution time. One very slow test file can create a straggler runner that gates the whole build. Most CI platforms including GitHub Actions, CircleCI, and Buildkite support parallelism natively.

Caching layers

Dependency installation is often the first 5 to 10 minutes of a build. If your pipeline downloads and installs dependencies from scratch on every run, you are paying that cost on every build even when nothing in your dependency manifest has changed. Cache the dependency layer keyed to the lockfile hash. When the lockfile does not change, restore the cache instead of installing. This is a standard feature of most CI platforms and the setup is typically under an hour.

Test selection

Not every code change should run the full test suite. A change to a frontend component should not trigger backend integration tests. A change to documentation should not trigger anything. Test selection, running only the tests relevant to the changed code, can reduce build time by 50 to 80 percent for targeted changes. Monorepos have good tooling for this: Nx, Turborepo, and Bazel all support dependency graph-aware test selection.

Flaky test debt

Flaky tests are a pipeline performance problem as well as a reliability problem. A test that fails 10 percent of the time forces either automatic retry (extending pipeline time) or manual re-runs (developer time). Track flaky tests explicitly. Quarantine flaky tests while fixing the flakiness rather than letting them pollute the main pipeline. Assign ownership: flaky tests do not get fixed without someone responsible for fixing them.

Measure before optimizing

Measure where your pipeline actually spends its time before optimizing. Many teams optimize the test stage when the actual bottleneck is container build time, or optimize caching when the actual bottleneck is test suite parallelism. CI platforms expose timing data for every step. Spend 30 minutes looking at the actual data before deciding which intervention to make first.

📊By the numbers