TL;DR - Almost every security control I meet in the wild was shipped as a rule and never as a road. The policy exists, the scanner runs, the build fails, and there is still a long-lived API key in a .env file, because nobody ever built the thing a developer is supposed to do instead. A rule without a road is a tax you levy on developers, and taxes get evaded, creatively and quietly. Treat secrets and policy the way you’d treat any other platform capability: distribute identity instead of secret values so the credential is short-lived and nobody handles it; ship every new control with a default that already satisfies it, an error message that names the fix, and a migration for what already exists; roll it out observe, then warn, then enforce; and make exceptions a first-class artefact that expires. The security team writes the rule. The platform team owes the road. Ship them in the same change or don’t ship the rule.
I build platforms for a living, which means I spend a lot of time in the seam between the security team and the people who ship code. That seam has a reliable shape. Every org I walk into has a secrets policy. It is usually a good policy, written by someone competent, and it says sensible things about rotation intervals and approved stores. And in the same org, roughly four minutes of grepping turns up a production credential in a repo, a shared vault entry that eleven people can open, and a CI variable somebody pasted in eighteen months ago that no longer maps to a human.
The instinct is to read that as a discipline problem. It isn’t. It is a product verdict. Every one of those secrets is a developer who, at some specific moment, weighed the paved way against the fast way and chose the fast way, because the paved way was a ticket and a wait and the fast way was a paste. The policy told them what not to do. Nothing told them what to do instead, at 6pm, without asking permission.
That’s the failure mode this post is about, and it applies just as hard to policy-as-code as it does to secrets. The pillar names both in its list of what an internal platform actually contains - “SBOMs, image signing, secrets management, policy-as-code. Not optional, but also not a 12-step manual process” - and that last clause is the entire argument. A control that costs twelve manual steps has not been implemented. It has been announced.
Secret sprawl is a map of where your platform stopped
Start with secrets, because they’re the most honest diagnostic instrument in your organisation.
Go and find where the credentials actually live. Not where the policy says they live. In my experience the list is always some version of: a .env file that’s gitignored on most machines, CI variables set by hand through a web UI, a shared password-manager folder, a Terraform state file, a Slack DM from 2024, and a Kubernetes Secret that is base64 and therefore, in the mind of at least one person on the team, encrypted.
Now overlay that list on your platform. Every one of those locations is a place where a developer needed a credential and the paved road did not reach. The sprawl isn’t random. It is a precise map of the gaps, and it’s more accurate than any survey you could run, because people were answering the question honestly with their hands instead of politely with their mouths.
Which reframes the work. The goal is not to hunt down the sprawl, because the sprawl regenerates as fast as you clear it, the same way a wiki page rots faster than you can update it. The goal is to make the sanctioned way to get a credential the cheapest way available at the moment of need. If getting a database password through the platform takes ten seconds and getting it wrong takes ten seconds, you win permanently. If the sanctioned way involves a ticket, you lose permanently, and you will keep losing while writing increasingly stern policies about it.
There’s a test I like here, borrowed from the same instinct as the error-budget question of whether a number has ever changed a decision: can you rotate a production credential this afternoon, unilaterally, without coordinating with anyone? Nearly every team has a rotation policy. Nearly none can pass that test, because rotation is a rare, scary, manual operation involving several people and a maintenance window, so it gets deferred until it becomes an incident. A rotation policy you can’t execute on a normal Tuesday is decoration.
Distribute identity, not secrets
The way out of that is a genuine shift in model, and it’s the single highest-leverage security change a platform team can make: stop trying to distribute secret values safely and start distributing identity instead.
The old model treats a credential as a thing you copy from a secure place to the place it’s needed, then protect at every point along the way. That model is unwinnable, because the number of copies only ever grows and each copy is long-lived. The workload-identity model inverts it: the workload proves what it is to an issuer, using a signed token from a runtime it doesn’t control, and the issuer mints a credential scoped to that workload, valid for minutes. Cloud IAM roles for a compute instance, OIDC federation from your CI provider to your cloud, service-account identity in a cluster, dynamic database credentials from a vault. The mechanisms differ, the shape is the same.
Three properties of that shape are worth naming, because together they’re why it wins:
- It’s an identity, not a value. There is nothing to copy, paste, screenshot or leak, because no human ever sees a credential. You cannot lose a secret you never had. This is what actually kills secret sprawl, and nothing else does.
- It’s short-lived, so rotation stops being an event. A credential that lives fifteen minutes is rotated constantly, by the normal operation of the system, exercised thousands of times a week. The scary annual operation becomes the boring default path. You never have to test your rotation procedure because you never stop running it.
- It’s issued to a workload, not a team. Scope collapses from “the service tier” to “this deployment of this service in this environment”, and revocation becomes surgical instead of an all-hands migration.
You will not get to zero real secrets, and you shouldn’t pretend otherwise. Third-party API keys, that payment provider with no federation story, the legacy system nobody may touch: there is always an irreducible core of genuine static credentials. Run a real secrets manager for those, and buy it rather than building it - a secrets store is exactly the undifferentiated substrate I’d argue you should never hand-roll, because a vendor whose entire business is that component will always operate it better than you will as a side quest. Your build budget goes into the thin layer above it: the bit that makes the store reachable from a service template on the first commit, wired to your identity provider, with defaults nobody has to look up.
Which is the actual deliverable. The first golden path already lists “secrets from the secret store, not the env file checked into Git” as a default the template ships with. This is what makes that line true rather than aspirational: a new service comes into the world already holding a workload identity, already able to read the config it needs, with no step in the README that begins “ask the platform team for”. The developer never makes a decision about credentials, which means they can never make the wrong one.
Policy as code is not the PDF, rewritten in Rego
Now the other half. Policy-as-code is having a moment, and mostly for good reasons, but there’s a specific way teams cash out the idea that produces nothing.
The pattern goes: security has a control document, someone competent translates the controls into Rego or Kyverno or a pile of Conftest tests, wires them into CI as a blocking check, and declares the control automated. And it is automated, in the narrow sense that a machine now evaluates it. But look at what changed for the person subject to the rule. Before, there was a document they didn’t read. Now, there’s a build that fails at 5pm on a Friday with deny: image.signature.required (SEC-014) and no indication of what to actually do. The friction wasn’t removed. It was industrialised, and made faster and colder.
Let me be fair about what policy-as-code genuinely buys you, because it’s real: an executable rule is versioned, reviewable, testable, and diffable. You can propose a control as a pull request, argue about it in a code review, write test cases proving what it does and doesn’t catch, and know with certainty what is enforced today, which is not something a PDF has ever been able to tell you. That’s the same jump in kind that Terraform gave us over hand-built infrastructure, and it’s worth having.
But executable is only half the job, and it’s the half that serves the security team. The other half serves the developer, and it’s the half that gets skipped: what does a person do when the rule says no? A control that can only refuse is a gate. Platform work is the business of making the right thing the easy thing, and a gate does exactly one of those two.
Never ship a control without shipping the road
So here’s the discipline, and it’s the whole post in one line: never ship a control without shipping the compliant path in the same change. Not next quarter. Same change.
Concretely, a control isn’t ready to enforce until it has all four of these:
- A default that already satisfies it. The service template, the shared pipeline, the base image, the Terraform module: whatever the paved road is, it complies out of the box, so the overwhelming majority of teams inherit compliance without ever knowing the rule exists. This is the actual goal. Compliance should be a side effect of using the platform, not an achievement unlocked by reading a policy.
- An error message that names the fix. Not the control ID. “This image isn’t signed. The
build-and-signstep in the shared pipeline does this for you: . If you’re on a custom pipeline, here’s the three-line diff.” Write the message for the tired developer who has never heard of your control, because that is who will read it. - A migration for what already exists. There are N repositories that violate this rule today, written by people who broke no rule at the time they wrote them. Enforcing on day one punishes them for your new idea. Either you bring them over, or you grandfather them explicitly with a deadline, but “everyone fix your own” is how a control becomes a thing developers route around.
- An owner for the false positives. Every non-trivial policy is wrong sometimes. Somebody has to be responsible for that, quickly, or the first bad denial teaches the whole org that the safe move is to work around the platform.
And roll it out in three stages, never one. Observe first: run the policy in dry-run against the entire estate and count the violations, changing nothing. That count is the single most valuable number in the project, and it is routinely a shock. If a rule you thought was uncontroversial has two hundred violations, it isn’t a rule, it’s a programme of work, and now you know before you’ve broken anybody’s build. This is the same instinct as measuring what’s actually happening before you wire an alert to it: find out the true state of the world first, then decide what to act on. Warn second: the check runs, reports, annotates the pull request, and passes anyway, while you fix the false positives and watch the violation count fall as the paved default spreads. Enforce last, when the count is near zero and enforcement is a formality rather than an event. Done this way, turning enforcement on is a non-event, which is exactly what a good rollout feels like.
One more thing to steal from the paved-road world: your policies are a portfolio, and they sprawl the same way paved roads sprawl once you have more than one. Every enforced control is a permanent maintenance line, not a one-off delivery. It will produce false positives, need exceptions, and break when the ecosystem underneath it moves. Twelve well-maintained controls that everyone understands beat ninety inherited from a compliance template, of which four are load-bearing and the rest are noise nobody can explain but nobody dares delete. Apply a paving bar to controls, and prune the zombies.
Exceptions are a feature, and they expire
Every control you enforce will, at some point, be wrong for someone with a legitimate reason. What happens next determines whether you have a security practice or a folk tradition.
If exceptions are hard to get, developers don’t stop needing them. They get creative. The base image gets pinned to a fork nobody scans. A # nosec comment appears with no explanation. The pipeline step gets commented out with a # temporary note dated two years ago. Now the control still shows green, the work still happened, and you’ve lost the one thing the control was actually giving you: knowledge of where the risk is. A hostile exception process doesn’t reduce exceptions. It moves them somewhere you can’t see them, which is strictly worse than granting them.
So build the exception path deliberately:
- Requestable where the work happens. A file in the repo, reviewed like code, approved by the owning team plus security. Not an email, not a form on an intranet nobody can find, not a DM to whoever is known to say yes.
- Expiring by default. Ninety days, and then it lapses and has to be renewed by a human who states the reason again. Permanent exceptions outlive the person who understood them and become archaeology. An expiry date is what stops your control set from quietly becoming fiction.
- Counted, and read as product feedback. The exception list is the best backlog the platform team has. When ten teams exempt themselves from the same rule, the rule is wrong or the road is missing, and either way that’s your next piece of work. One team exempting itself is a special case. Ten is a design defect in the control.
The contrarian bit: an organisation with zero exceptions is not disciplined, it’s uninformed. The exceptions didn’t vanish, they went dark. A visible, expiring, well-reviewed exception is a control working correctly.
Who owns this, and how you’d know it’s working
The ownership question resolves cleanly if you use the Team Topologies interaction modes: security is an enabling team, not a gate operator. Their job is to know the threat model, write the rule, and pair with the platform team for the weeks it takes to pave the road that satisfies it. The platform team then runs that capability X-as-a-Service, like every other capability. When security owns the gate but not the road, you get an organisation where the security team’s only available verb is “no”, which is bad for the platform, worse for the developers, and honestly worst for the security engineers, who did not get into this field to operate a turnstile.
For signals, be careful which number you chase, because this area is unusually rich in the same theatre that afflicts DORA dashboards. “98% policy compliance” is a deeply suspicious metric: it is trivially satisfied by exempting the hard cases and counting the easy ones. Better questions:
- What fraction of services get the control by default, without doing anything? This is the real number. It measures the road, not the rule.
- How long from a new control being agreed to it being enforced? If it’s a year, your paving capability is the bottleneck, not your policy-writing capability.
- Are live exceptions trending down, and does each one have a name and a date?
- Is time-to-first-deploy still good? If security controls quietly added three days to shipping a service, the org will route around the platform, and you’ll have traded a visible risk for an invisible one.
When this is the wrong answer
The honest off-ramps, because “pave it, don’t gate it” is a default with real edges.
- You’re small enough that the rule is the road. With five engineers and one repo, a shared convention and your PaaS’s built-in secrets feature is a complete secrets strategy. Standing up a policy engine and an identity broker for three services is the same category error as running Kubernetes for three of them: you’ve bought the operating cost of a scale you don’t have. Use the managed thing, write the convention down, move on.
- Some controls really are non-negotiable, road or no road. Customer data leaving the boundary, production credentials in a public repo, unreviewed access to the payments database. A handful of controls are existential, and you enforce those on day one and eat the disruption. The pairing rule is for the ninety-odd percent that are hygiene. Just be honest about which bucket a control is in, because the failure mode is classifying everything as existential precisely to avoid doing the paving work.
- You don’t have a platform to put the default into yet. If there’s no service template and no shared pipeline, there is nowhere for a compliant default to live, and policy-as-code becomes enforcement with no delivery mechanism attached. Build the first paved road, then hang the controls off it. The order matters more than the tooling.
- Your cloud already sells you the whole capability. Workload identity federation, a managed policy service, org-level guardrails: if the platform you’re on has it, adopt it rather than assembling your own from parts. Keep your opinion layer thin, keep the stack boring, and spend the saved effort on the paved roads that are genuinely yours.
You start earning the full apparatus when the same security question is being answered differently by every team, when onboarding a service means a human decides where its credentials come from, or when your security team has become a queue that developers plan around. That’s the moment security stops being a document and starts being a platform capability.
The bottom line
Security controls don’t fail because developers are careless or because the rules are wrong. They fail because a rule was shipped without a road, and a rule without a road is a tax: it makes the correct behaviour the expensive behaviour and then acts surprised when people don’t pay. The .env file, the shared vault entry, the commented-out pipeline step - none of those are discipline failures. They’re the map of where your platform didn’t reach.
So do the platform version instead. Distribute identity rather than secret values, so the credential is short-lived and nobody ever holds it. Put the compliant default in the template, so the majority of your org complies by accident. Ship every control with a fix-naming error message, a migration for what already exists, and an observe-warn-enforce rollout that makes enforcement day boring. Let exceptions exist, in code, with an expiry date, and read them as feedback rather than failure.
The question to ask before you enforce anything: when a developer hits this rule, what exactly do we want them to do, and have we built it yet? If you can’t answer the second half, you don’t have a control. You have a tax with a scanner attached.
If you’re standing up secrets or policy-as-code and it’s turning into a queue that teams route around - or you’ve got a green compliance dashboard and a nagging suspicion the real credentials are somewhere else entirely - that’s exactly the kind of thing I help teams sort out. Let’s talk.