The Change That Skipped the Pipeline

TL;DR - Most platform teams have spent years making code deploys safe: review, tests, a canary, a one-command rollback, a record of who shipped what. Then production changes in a dozen other ways that touch none of it - a feature flag flipped to everyone at once, a global config value edited in a web UI, a security group changed in the cloud console, a manual UPDATE in a database shell, a DNS record, an IAM policy, a vendor setting. In most organisations I have worked with, those side doors start a large share of the incidents, and the change failure rate cannot see any of them because it only counts deploys. The cure is not to force every change through a twenty-minute pipeline. It is to give every change channel the properties that made deploys safe - a record, a proportionate review, a staged rollout and a fast undo - and to make “what changed in the last hour?” a single query instead of a question in Slack.

I build and run platforms for a living, and the incident timeline I have read most often in my career has the same line near the top. Somebody checks the deploy history for the window when the errors started and writes, with visible relief, “no deploys in the window”.

That line is supposed to narrow the search. In practice it sends everyone in the wrong direction for forty minutes, because the cause was a change - just not one the pipeline knew about. A flag went from 5% to 100% because the experiment “looked fine”. A timeout in a shared config service was lowered from thirty seconds to three, for every consumer at once. Someone opened a port in the console while debugging last Thursday and closed the wrong one on the way out. The database lost an index because a migration was “just run by hand, it was quicker”.

The deploy pipeline was perfect. It was also guarding one door out of twelve.

The tell

Three questions, asked of whoever runs your incident process:

• When an incident starts, how do you find out what changed? If the honest answer is “we look at the deploy history, then ask around”, you have one channel with a record and many without. The asking around is where the time goes.
• How many ways are there to change production? Not deploy. Change. Count them out loud: deploys, flags, runtime config, infrastructure applies, console clicks, DNS and certificates, CDN rules, access grants, manual data changes, scheduled jobs, third-party dashboards. Most teams have never written the list down, and the first draft is always too short.
• Which of those can you undo in under five minutes, by someone who did not make the change? That is the real coverage of your production-safety stack. For deploys it is usually yes. For the rest it is usually “whoever did it remembers what the old value was”.

If the list is long and the undo column is mostly blank, your road guards the change that was already the safest.

Why the road only guards deploys

Nobody chose this. It happens for reasons that each sound reasonable.

• The pipeline is the visible door. Platform teams pave what they can see, and the deploy is the one change that already flows through infrastructure they own. Everything else happens in tools that other teams bought, in consoles, or in someone’s terminal.
• “It is just config.” A config value feels smaller than code, so it gets less ceremony. It is not smaller. A shared config value is code that runs everywhere at once, with no tests, deployed to every consumer in the same second. It is the opposite of a canary.
• The fast tools were sold as escaping the pipeline. Flag services and dynamic config exist precisely so you can change behaviour without waiting for a build. That is a real benefit - I argued for feature flags by default because separating deploy from release is the cheapest safety there is. But “faster than a deploy” quietly became “outside every safety a deploy has”.
• Each channel has a different owner. The flag tool belongs to product engineering, the config service to one backend team, the cloud console to whoever has admin, DNS to the network team, the database to the DBAs. No single team owns “change”, so nobody sees that all of them are the same problem.
• The metric only counts deploys. The pillar’s DORA list defines change failure rate as the percentage of deploys that cause an incident. If the flag flip that took checkout down is not a deploy, it is not in the numerator or the denominator. The number looks great, and it is measuring the safest channel you have.

That last one matters more than it looks. When a metric is blind to the most dangerous changes, it is dashboard theatre by construction, without anyone gaming anything.

The arithmetic

Do this once with your own data. Take the last twenty incidents and, for each one, write down the change that triggered it - not the root cause, the trigger. Then sort the triggers into “came through the pipeline” and “did not”.

The split varies, but in every organisation where I have done it, the side doors were somewhere between a third and well over half. And the side-door incidents had two properties the deploy incidents did not. They took longer to diagnose, because the first forty minutes went on the deploy history. And they were bigger, because a global config change or a flag at 100% has no blast-radius limit at all.

That is the uncomfortable shape. You have a road with seat belts for the change that already had a small blast radius and a fast rollback, and an open field for the changes that hit everything at once and cannot easily be undone. If your postmortems keep producing action items that change nothing, check how many of them were about a change nobody could see.

Make “what changed?” one query

Before you change any process, fix the record. It is the cheapest, highest-leverage step in this whole post, and it requires no one to change how they work.

• Every channel emits a change event to one place. Deploys, flag changes, config edits, infrastructure applies, access grants, DNS updates. Each event carries the same fields: what changed, from what to what, who, when, which services or environments it touches, and how to undo it. Most flag and config tools already have a webhook or an audit log. The work is collecting them, not inventing them.
• Side doors get observed, not just declared. Console changes show up in the cloud provider’s audit log, and infrastructure drift shows up when you compare live state to code. If you are fluent in Terraform, you already know drift is the thing that bites; run the drift check on a schedule and turn every finding into a change event. A manual change you detect is still a change on the record.
• Put the feed where incidents happen. Overlay change events on the service dashboards, and make the first step of every incident a query: “everything that changed for this service and its dependencies in the last two hours”. The useful version knows the dependency graph, which is one more reason to have an estate you can actually enumerate.

The test is simple. In the next incident, does anyone have to ask in a chat channel what changed? If yes, a channel is missing from the feed.

Give config the properties of a deploy

The deploy pipeline is safe because of a handful of properties, not because of the pipeline itself. Any change channel can have them, at a fraction of the ceremony.

• Versioned. Every config value has a history, and the previous value is one click away. “Whoever changed it remembers” is not a version history.
• Validated before it lands. Config gets a schema: types, ranges, required fields. A timeout of 3 where the unit is milliseconds should fail at write time with a message that names the fix, not at runtime in forty services.
• Staged, not global. A config change goes to one cell, one region, one percentage of instances, or one internal tenant first, and waits long enough to see the metrics that matter. That is a canary. A global config service with no staging is a deploy to everything at once with no rollback plan.
• Reviewed in proportion to blast radius. A per-service setting can be self-approved. A value read by thirty services gets a second pair of eyes. Review is cheap when it is proportionate and gets rubber-stamped when it is not.
• Reversible by someone else. One command, or one button, restores the last known good value, and the on-call engineer can do it without finding the author.

None of this needs to go through the build pipeline. The point is to bring the deploy’s properties to config, not to bring config to the deploy.

Flags are production changes too

Feature flags deserve their own section, because they are the side door that was built on purpose and is now the busiest.

• A flag change is a release. Moving a flag from 5% to 100% is the moment new code meets all your users. Treat it like one: staged steps, a wait at each step, and an automatic stop if error rate or latency moves.
• Kill switches are a separate kind of flag. A flag that turns off a dependency or a feature in an emergency should be documented, tested and listed in the runbook. It should not be the same mechanism as the marketing experiment.
• Every flag has an owner and an expiry. Flags that outlive their rollout become permanent, undocumented branches in the code. Stale flags are config debt, and the one that bites is always the flag nobody remembers creating, flipped by someone who thought it was a different one.
• Flag changes are on the change feed. If your flag tool cannot emit events, that is a reason to change flag tools, not a reason to skip the feed.

Close the side doors you do not need

Some channels should not exist at all, or should exist only for emergencies.

• Consoles are read-only by default. Engineers can look at everything and change nothing through the web UI in production. Changes go through code. When someone needs write access in an emergency, they get it through a break-glass path that is time-boxed and opens a ticket automatically, the same pattern I described for the platform team’s own pager.
• Data changes go through a recorded runner. A manual UPDATE in a shell is the least reversible change there is. Give people a paved way to run a one-off data fix: the script in a pull request, reviewed, run by a job with its own identity, output logged, and a backup or a dry run first. If you have a paved road for the data layer, the restore drill is what makes this safe enough to allow at all.
• Access is a change. Granting a role or opening a network path is a production change with a security blast radius. It belongs on the feed, it expires, and, as with every control that comes with a road, the compliant way should be faster than asking a friend with admin rights.

The guiding rule: do not ban a side door until there is a road that is at least as fast. Ban it first and people will find a smaller, darker door.

Who owns what

As with every shared capability in this cluster, draw the line on purpose:

• The platform owns the mechanism. The change feed, the event format, the drift detection, the staged-rollout machinery for config and flags, the break-glass path, the data-change runner, and the “what changed?” query.
• Teams own the meaning. Which of their config values are dangerous, what their flags do and when they expire, which kill switches exist and when to use them.
• Whoever runs a shared change channel owns its safety. If your team runs the config service thirty teams read from, staged rollout and validation are part of your service, not a favour to your users.

How you know it is working

• Share of incidents triggered by a change that was not on the feed. The headline number. It should fall towards zero, and every time it does not, you have found a missing channel.
• Time to answer “what changed?” at the start of an incident. Measured in the incident timeline. Seconds, not forty minutes.
• Change failure rate across all channels, not just deploys. Count flag changes, config changes and infrastructure applies in the denominator and the numerator. The number will get worse first. That is the point: it was lying before.
• Drift findings per week, and how old they are. Each one is a side door someone used. A steady rate tells you which road is too slow.
• Flags past their expiry, and break-glass uses per month. Both should be small and boring. A rising break-glass count is a product signal about the normal path.

Notice what is not on the list: number of change approvals, or tickets raised. An approval board measures ceremony, not safety.

When this is the wrong answer

• Twelve engineers and one service. If everyone can see every change in the team channel and the person who made it is always awake, a shared changelog and read-only consoles are enough. Build the feed when you can no longer remember who changed what.
• You have no deploy road yet. If code deploys still happen from a laptop, start there. Paving side doors next to an unpaved main road is the wrong order.
• Your side doors are genuinely rare. If the incident sort says nearly everything came through the pipeline, believe it. Spend the effort on the pipeline, and run the sort again in six months.
• Regulated change management already exists. If you have a mandated change process, feed it from the change events rather than building a second one. Two records of the same change is how both go stale.
• The emergency is now. During an incident, use whatever door fixes it fastest. The work is making sure that door is on the record afterwards, not slowing down the fix.

One guardrail for the other direction. The worst version of this is a change advisory board in platform clothing: every flag flip waits for a ticket, every config edit waits for a pipeline, and teams respond by hard-coding values and shipping fewer, bigger changes - which is how you get the incidents you were trying to prevent. Flags and dynamic config exist because fast change is safer than slow change. Keep them fast. Make them visible, staged and reversible.

The bottom line

The deploy pipeline is usually the safest door into production, because it is the one we have spent a decade paving. The others - flags, config, consoles, data, DNS, access - change production just as surely, often with a bigger blast radius and no undo, and they do it out of sight of the metric that is supposed to tell you whether change is safe.

So widen the road to cover change, not just deploys. One change feed that every channel writes to. Config that is versioned, validated, staged and reversible. Flags treated as releases, with owners and expiry dates. Consoles read-only, break-glass logged, data changes through a recorded runner. Then count every channel in your change failure rate, and watch the number get honest. That is platform engineering doing what it is for: making the safe way to change production the easy way, whichever door you came through.

The question I would ask any platform team: in your last incident, how long did it take to find out what changed - and was the answer in the deploy history? If it was not, the next road to pave is not the pipeline.

If your incidents keep starting with “no deploys in the window”, or you suspect your change failure rate is only measuring the safest door, that is exactly the kind of problem I help teams work through. Let’s talk.