Your Build Is Slow Because Everyone Owns It

TL;DR - Nobody decides to have a twenty-five-minute build. It gets there one reasonable step at a time: a new scanner here, an integration suite there, a cache that quietly stopped hitting, a runner pool sized to last year’s headcount. Every team owns a step, so nobody owns the total, and the total is the only number developers experience. Flaky tests finish the job by teaching everyone that red means “press retry”, which is the moment the pipeline stops being a signal. The fix is not a quarterly “speed up CI” sprint. It is treating feedback-loop latency as a platform capability with an owner: a published time budget for push-to-signal, caching and test selection as shared primitives rather than per-repo heroics, runners sized on queue time rather than on the invoice, and flaky tests quarantined with an owner and an expiry date. Measure the wait the developer actually sits through, not the average job duration on a dashboard.

I build and run platforms for a living, and one of the most reliable ways I have of reading an engineering organisation is to push a one-line change and time how long it takes for the pipeline to tell me something. Not the deploy. Just the first honest signal: does this compile, do the tests pass, may I merge?

In a healthy setup that number is a few minutes. In most of the places I walk into it is somewhere between fifteen and forty, with a second run “because that test is flaky”, and a queue in front of it at ten in the morning. When I ask who owns that number, I get a pause and then a list of names, each of whom owns one step.

That pause is the whole problem.

The tell

Three questions about the pipeline behind your busiest repository:

• What is the p90 time from push to first useful signal, including the queue? Not the average job duration on the CI vendor’s dashboard. The wall-clock time a developer waits, from git push to a red or green they can act on. If nobody can answer without going to look, nobody is watching it.
• What share of pipeline runs are re-runs of the same commit? A re-run of an unchanged commit is either a flaky test or a flaky runner. Either way it is someone saying “I do not believe what the pipeline just told me”. If that share is above a few percent, you have a trust problem, not a speed problem.
• Who would be in trouble if the build doubled in length next quarter? If the honest answer is “nobody, but everyone would complain”, the build is a commons, and commons degrade.

How a build gets slow without anyone deciding

There is no villain here. Every step that makes a build slow was added by someone sensible for a good reason.

• Each addition is locally rational. The security team adds a scanner, which costs ninety seconds. Someone adds an end-to-end suite after an incident, which costs four minutes. A team adds a second architecture to the image build. None of these is wrong, and none of them is noticed, because a ninety-second regression is invisible inside a single pull request. Nobody reviews the build’s total length, because nobody’s job is to.
• The cache rots silently. Dependency caches, layer caches and build caches all degrade without failing. A key changes, a base image moves, a lockfile is regenerated on every run, and the build still goes green. It just takes eight minutes longer, forever, and there is no alert for “this used to be fast”.
• The queue is not in the duration. The CI dashboard reports how long jobs ran. It does not report how long they waited for a runner. At nine in the evening the build takes eleven minutes. At ten in the morning it takes eleven minutes plus twelve in the queue, and only the developer sees the second number.
• The invoice is visible and the wait is not. Runner minutes arrive as a bill with a name on it. Engineers waiting arrive as nothing at all. So the runner pool gets sized by whoever reads the invoice, which is the same broken feedback loop I described in why your cloud bill is not a finance problem, pointing in the opposite direction: here the cheap-looking number is the expensive one.
• Retry is free. A test fails, someone presses retry, it passes. The cost of the flake moves from the person who wrote the test to the person who happened to trip over it, and the test stays exactly as flaky as it was.

The result is a build that is slow in a way that is everybody’s fault and nobody’s problem. That is the definition of something a platform should own.

The arithmetic nobody does

The reason this matters more than it looks is that the cost is not linear in minutes. There is a threshold.

Below roughly ten minutes, people wait. They read the diff again, glance at a review, and pick the signal up the moment it lands. Above it, they switch. They start another task, and the context they had when they pushed is gone by the time the build answers. A red build after twenty-five minutes is a red build for someone who is now thinking about something else.

And then the second-order effect, which is the one that actually hurts: when feedback is slow, people batch. They stop pushing small changes because every push costs half an hour of latency, so they accumulate three changes into one. Larger changes fail more often, are harder to review, and are harder to roll back. That is how a slow build shows up in your change failure rate and your lead time without anyone connecting the two, and it is exactly the kind of causal link that gets lost when the DORA numbers are treated as a dashboard rather than a diagnosis.

Do the sum once, for your own organisation, with your own numbers. A couple of hundred engineers, a handful of pushes each per day, fifteen minutes of avoidable wait per push, a fraction of which becomes a lost context switch. The figure is always larger than the runner bill, usually by an order of magnitude. I have yet to see an organisation where buying more runners was not cheaper than the waiting they replaced.

Flaky tests are a trust problem

A slow build costs time. A flaky build costs belief, and belief is harder to get back.

The first time a developer sees a test fail on a change that could not possibly have caused it, they press retry. The tenth time, they press retry without reading the failure. At that point the pipeline has stopped being a signal and become a slot machine, and a real regression gets retried past along with everything else. Every gate downstream depends on green meaning something: you cannot auto-merge the dependency bot’s routine updates on a green build that nobody trusts, and you cannot deploy continuously behind one either.

When I wrote about running the platform team’s own pager, I argued that every pipeline failure has to be classifiable as “the road broke” or “the change was bad”. Flaky tests are the third bucket that classification leaves out: the change was fine, the road was fine, and the test lied. That bucket needs its own process.

• Detect flakes from the data you already have. A test that fails and then passes on the same commit, without a code change in between, is flaky by definition. Every re-run is a free experiment. Record per-test pass and fail by commit and the list of your worst offenders writes itself.
• Quarantine, do not delete, and do not ignore. A detected flaky test moves out of the blocking gate automatically and into a non-blocking run that still reports. The gate goes back to meaning something today, and the test keeps generating evidence.
• Every quarantined test gets an owner and an expiry. The owning team, from the inventory, gets a ticket. If the test is not fixed or deliberately deleted within a set window, it is deleted on the date, with a note. A quarantine without an expiry is a graveyard, and a graveyard of skipped tests is worse than no tests, because it looks like coverage.
• Treat the retry button as telemetry. Count it. The re-run rate per repository is one of the most honest numbers you have about how much the team believes its own pipeline.

Own the total, not the steps

The structural fix is to give the end-to-end feedback loop an owner, and the natural owner is the platform team, because the pipeline is already a shared product whether anyone admits it or not. Owning the total does not mean owning every step. It means owning the budget and the primitives that make the budget achievable.

• Publish a time budget. For example: first signal in under five minutes, merge-ready in under ten, at p90 including the queue. Write it down where the road is documented. A budget turns “the build feels slow” into a regression with a date, and it gives anyone adding a step a number to stay inside. Without the road to meet it, though, a budget is just a rule nobody can follow, so it has to ship with the next three bullets.
• Caching is a platform primitive, not a per-repo heroic. Dependency caches, image layer caches and, where the build tool supports it, a shared remote build cache are infrastructure. They should be on by default in the shared pipeline, keyed correctly, and monitored for hit rate like any other production service. A cache hit rate that drops from 90% to 40% is an incident with a measurable cost, even if nothing is red.
• Size runners on queue time, not on the invoice. Set an objective on how long a job waits for a runner at peak, and scale to it. If the cost conversation comes up, bring the arithmetic from the section above. Then make the build cheap by construction in other ways: right-sized runner classes, scale-to-zero overnight, and no permanent fleet idling for a peak that lasts an hour.
• Run less, earlier. Order the pipeline so the cheapest, most-likely-to-fail checks run first and fail fast. Run only the tests affected by the change where the build tool can tell you which ones those are. Split the gate honestly: what must block a merge, and what can run after it, against main, with a fast revert if it fails.

The way to deliver all of that once instead of two hundred times is the same pattern as everywhere else in the cluster. The pipeline should be a thin template over fat shared primitives, so that when the platform team fixes caching, adds a fast-fail stage or moves to a better runner class, every repository on the road gets it without a pull request. If each repository carries four hundred lines of copied pipeline YAML, you cannot own the total, because there is no total, only two hundred forks of it.

Who owns what

As with every shared capability, the line needs drawing on purpose rather than wherever the last complaint left it.

• The platform owns the loop. The shared pipeline, the runners, the caches, the time budget, flake detection and quarantine mechanics, the per-repo latency and re-run numbers, and the stage ordering defaults.
• The team owns the tests. What they test, how long their own suites take, fixing or deleting the flaky ones they own, and keeping their repository inside the budget once the road makes that possible.
• Anyone adding a blocking step pays for it. The security scanner, the licence check, the new compliance gate: each one comes with its time cost stated, and either fits inside the budget or comes with the speed-up that makes room for it.

That third rule matters most. Most slow builds are slow because adding a step is free and removing one requires a meeting.

How you know it is working

• p50 and p90 push-to-first-signal, including queue time. The number a developer actually experiences, per repository, tracked over time.
• Queue time at peak. On its own line, so it cannot hide inside an average.
• Re-run rate on unchanged commits. The trust number. It should trend towards zero.
• Quarantined tests, by age. The count matters less than the age. Anything past its expiry is a broken promise.
• Cache hit rate on the shared pipeline. A leading indicator. It falls before the build gets slow.

Notice what is not on the list: average job duration. It excludes the queue, it averages away the long tail that people actually remember, and it gets better when you skip tests. It is the CI equivalent of host uptime.

When this is the wrong answer

• Your build is already fast. A small team with a four-minute build and no flakes does not need a latency programme. Write the budget down so you notice when it drifts, and go and work on something else.
• There is no shared pipeline yet. If every repository has its own hand-written CI, owning the total is not possible yet. Build the first paved road with a pipeline worth sharing, and the latency work plugs into it afterwards.
• A dedicated build team already exists. Large monorepo shops often have a build or developer-productivity team that owns exactly this. Do not fork it. The platform’s job is to feed it runners, identity and capacity, and get out of the way.
• The tests are slow because they are honest. Some integration suites take time because they exercise real things. Do not trade them for a fast build that proves nothing, and do not move them out of the gate just to hit a number. Move them after the merge only if you can revert quickly and have decided that the risk is acceptable.
• Regulated, validated pipelines. In some environments every pipeline change is itself a controlled change. Fine. The budget and the measurement still apply. Just batch the improvements into the validation cycle rather than shipping them weekly.

One guardrail for the other direction. Once there is a number, there is a temptation to hit it by deleting tests, skipping stages on “small” changes, or quietly marking gates as non-blocking. That is speed bought with confidence, and it is a bad trade. The goal is a fast loop that still tells the truth, and the honest metric set above is deliberately built so that cheating on one number shows up in another.

The bottom line

A slow, flaky build is not a technical failure. It is an ownership failure. Every step in the pipeline has an owner and the sum of the steps has none, so the sum drifts upwards one sensible decision at a time, retries hide the flakes, and the invoice argues against the only fix that works. Meanwhile engineers wait, switch, batch, and ship bigger changes that fail more often.

So put a name on the total. Platform engineering exists to make the right thing the easy thing, and a fast, trustworthy feedback loop is the thing every other good habit depends on: small changes, continuous deployment, auto-merge, honest DORA numbers. Publish a time budget including the queue. Make caching and runners shared primitives the platform runs like production. Quarantine flakes automatically, with an owner and an expiry. Make anyone adding a blocking step pay for it in minutes. And in the platform team’s second year, when the work is mostly running what you built, treat the build’s latency as one of the ratios you report.

The question I would ask any platform team: what was the p90 wait from push to signal on your busiest repository last week, and who would have noticed if it had doubled? If the answer to the second half is “everyone, but no one in particular”, that is the next thing to own.

If your pipeline has crept past the point where people wait for it, or the retry button has become part of everyone’s workflow, that is exactly the kind of problem I help teams work through. Let’s talk.