TL;DR - Every service in your estate has a disaster recovery story, and almost all of them begin with the same sentence: “redeploy it through the platform”. The platform itself usually has no such story. It was assembled over two or three years, partly from code and partly by hand, and it is full of quiet loops: the pipeline deploys the pipeline, the secret store is unsealed with keys kept in the secret store, the recovery runbook lives on a wiki hosted on the cluster it describes. Nobody has ever rebuilt it from an empty account, so nobody knows whether they could, or how long it would take. The fix is not a two-hundred-page DR plan. It is a map of the cycles with a named break point for each, a deliberately small “layer zero” that lives outside everything it bootstraps, a written bootstrap order, and a timed rebuild drill that proves the whole thing works before the day you need it.
I build and run platforms for a living, and there is one question that reliably makes a confident platform team go quiet. Not “what is your uptime” or “how fast is your pipeline”. Just this: if the cloud account the platform runs in were gone tomorrow, what would you do first?
Usually someone starts answering with “we would redeploy…” and then stops, because they have just realised that the thing they would redeploy with is the thing that is gone.
That pause is the finding. The platform is the restore tool for everything else, and it is the one system nobody has tried to restore.
The tell
Three questions, asked of the platform team rather than of any product team:
- Where is the runbook for restoring the secret store, and what does it run on? If the answer is “the wiki” and the wiki sits behind your SSO, behind your ingress, on your cluster, then the runbook is only readable while the thing it restores is working.
- If your identity provider were down, could anyone log into the cloud account? Break-glass credentials exist in most organisations. In my experience they have been tested in very few, and when they are tested the first time, they do not work: the MFA device belonged to someone who left, or the password was rotated by a policy nobody remembered.
- What is the oldest piece of the platform that has never been recreated? There is always one. The state bucket created by hand in week two. The DNS zone imported “temporarily”. The CI runner image somebody built on their laptop in year one, still the base for everything.
If those answers are vague, you do not have a recoverable platform. You have a platform that has not been asked yet.
Why nobody has a recovery story for the platform
Nobody plans to leave the platform out of disaster recovery. It happens for reasons that each sound sensible.
- DR plans are written per service, and every one assumes the platform is up. Each product team writes “restore from backup, redeploy from the pipeline” and is quite right to. The platform team reviews those plans and does not notice that its own system appears in every one of them as a given, never as a subject.
- The platform grew by accretion. Year one was a construction project under time pressure. Some of it was code, some was clicked into existence to unblock the first team, and the clicked parts were always going to be codified “next sprint”. That backlog is the platform’s real recovery gap.
- The platform deploys itself. Self-hosting is elegant and usually right: the platform team should eat its own cooking. It also means the normal path for changing the pipeline, the cluster add-ons or the secret store runs through the pipeline, the cluster and the secret store. That is a loop, and loops only break cleanly if you designed the break.
- “We are in the cloud” sounds like a recovery strategy. The provider has multiple zones and very good durability. That protects you from the provider losing a disk. It does not protect you from an account compromised and wiped, a region-wide control-plane incident, a destructive
terraform apply, or a billing suspension. Those are all things I have seen happen to real teams. - The test is expensive and the risk is rare. A full rebuild drill takes days the first time, and the event it prepares for might never come. So it loses every prioritisation meeting, quarter after quarter, until the quarter it would have mattered.
Find the cycles
The first useful artefact is not a plan. It is a dependency map of the platform’s own components, drawn specifically to find loops. Take a whiteboard and list what the platform is made of: the cloud accounts and their root credentials, the identity provider, DNS, the infrastructure-as-code state, the container registry, the CI system and its runners, the clusters, the secret store, the ingress, observability, the catalogue, the docs.
Then, for each one, write down what it needs to come back. The cycles appear quickly, and they are nearly always the same handful:
- The pipeline deploys the pipeline. If CI is gone, what deploys CI?
- The secret store holds its own keys. The unseal keys, or the credentials the bootstrap job uses, are stored as secrets in the store they unlock.
- The state is created by the code that needs the state. The bucket holding your infrastructure state was created by the same configuration that stores its state in that bucket. Anyone fluent in Terraform has met a version of this, and most teams solved it once, by hand, and did not write down how.
- The registry is pulled from by the cluster that hosts it. The images the cluster needs to start include the image for the registry the cluster runs.
- Identity sits behind the ingress it protects. The SSO login page is served through the ingress controller that only lets in people who have logged in.
- The docs are hosted on the thing they document. The wiki, the catalogue and the runbooks run on the platform and are only reachable through it.
None of these cycles is a mistake on its own. Most of them are the right design for normal operation. The mistake is a cycle with no documented, tested way to break it. For each loop on the map, write one line: “to break this cycle, start here, with this credential, using this command”. If you cannot write the line, you have found the first piece of work.
Keep layer zero small and outside
Every rebuild has to start somewhere that does not depend on the platform. I call that layer zero, and the whole design exercise is keeping it small, boring and genuinely outside.
- What belongs in layer zero. The root and break-glass credentials for the cloud accounts. The domain registrar and the delegation for your top-level DNS. The bucket or backend that holds infrastructure state for the bootstrap layer. A copy of the bootstrap code and the bootstrap runbook. The offline recovery material for the secret store. Very little else.
- Where it lives. Somewhere that survives the platform, your identity provider and your main cloud account all being unavailable at the same time. That usually means a separate account with separate credentials, plus a sealed, physically held copy of the most critical keys, split between named people. It does not mean a folder in the wiki.
- How it is protected. Layer zero is the most powerful thing you own, so it is also the most dangerous. Access is rare, logged, requires more than one person, and raises an alert every time it is used. The identity-first approach to secrets covers almost everything else on the platform; layer zero is the small, honest exception, and it should stay small.
- How it is kept true. Every layer-zero credential has a “last tested” date, and someone tests it every quarter by actually using it. A break-glass password nobody has typed in a year is not a break-glass path. It is a hope.
The temptation is to let layer zero grow, because everything feels critical in a disaster. Resist it. Every item in layer zero is something you maintain by hand, outside every road you have built. Ten items is manageable. Fifty is a second, worse platform.
Write the bootstrap order down
With the cycles broken and layer zero defined, the rebuild becomes a sequence, and the sequence is an artefact you can version, review and test.
- Layer zero. Recover access to the accounts, the registrar and the bootstrap state.
- Foundations. Networking, DNS zones, the identity integration with the cloud, the state backends for everything else, all from code applied from a known machine with layer-zero credentials.
- The substrate. Clusters or the compute layer, the registry, the secret store, unsealed from layer-zero material.
- The road. CI and its runners, the templates, the deploy machinery. From this point the platform can deploy itself again, and everything after this runs through the normal path.
- The rest of the platform. Observability, ingress for internal tools, the catalogue, the docs.
- The services. Now, and only now, every product team’s own recovery plan becomes executable.
Two properties matter more than the exact steps. First, the order lives in the same repository as the platform code, so it changes when the platform changes. A bootstrap document in a separate wiki drifts out of date within a quarter. Second, the first three steps must run from something other than the platform, usually a pinned container image and a laptop with layer-zero credentials. If step two needs your CI system, it is not step two.
Code is only the truth if you check
“It is all in Terraform” is the sentence I hear most often in response to the opening question, and it is rarely entirely true. Code describes what someone declared. The live estate contains whatever actually happened, including the hand-made resources from year one, the console fixes during incidents, and the settings that were changed outside code because the change skipped the pipeline.
- Run drift detection on the platform’s own infrastructure, on a schedule. Every finding is a part of the platform that a rebuild from code would not reproduce.
- Hunt for the resources with no code at all. Compare what exists in the accounts with what is under management. The difference is your un-rebuildable estate, and an inventory built from what is observed rather than declared is exactly the tool for finding it.
- Watch for hidden inputs. A module that reads a value from a parameter someone set by hand, a pipeline that depends on a runner image nobody can rebuild, a config file that was copied from a server once. Each one is a step in the bootstrap order that exists only in someone’s memory.
The platform has state too
Rebuilding infrastructure from code is the easy half. The platform also holds data that code cannot regenerate: the contents of the secret store, the images in the registry, the infrastructure state files, the CI configuration and its history, the catalogue’s ownership records, the dashboards and alert rules if they were built by hand.
Treat that data the way a properly paved data road treats a service’s database: backups taken automatically, stored outside the blast radius, and restored on a schedule to prove they work. The secret store backup in particular deserves a drill of its own, because restoring it usually needs exactly the unseal material that layer zero is supposed to hold. That is the cycle most likely to be broken in practice and the one most likely to be discovered during the incident.
Some of this data can be regenerated instead of restored, and that is often the better design. Images can be rebuilt from source if the builds are reproducible. Short-lived credentials can simply be reissued. Prefer rebuild over restore wherever you can, because a rebuild proves itself every day and a restore only proves itself when you test it.
Drill it, small first
A platform that has never been rebuilt is in the same position as a backup that has never been restored: a hypothesis. The only way to turn it into a fact is to run the experiment, and the trick is to start small enough that the experiment actually gets scheduled.
- Rebuild one component. Start with something whose loss would hurt but not stop the world: tear down and recreate the CI runners, or a non-production cluster, purely from code and the bootstrap order. Time it. Write down every step that was not in the document.
- Rebuild one cycle. Pick a loop from the map, such as the secret store’s unseal path, and break it deliberately in a scratch environment. This is where most of the surprises are.
- Test layer zero. Every quarter, someone logs in with the break-glass credentials, confirms the sealed keys are where they should be, and records the date. Fifteen minutes, and the most valuable fifteen minutes of the quarter.
- Then the full rebuild. Once a year, rebuild the whole platform into an empty account, follow the bootstrap order, deploy one real service through the rebuilt road, and stop the clock when it serves traffic. That number is your platform’s recovery time. Everything else is a guess.
Run the drills in working hours, against scratch accounts, with the whole team watching the first time. The goal is to find the missing steps, not to prove courage. A drill that is run as a surprise chaos stunt in production teaches people to fear the drill.
Not everything needs the same recovery time
The platform is not one service, and its recovery objectives should not be one number. The tiering that makes sense for the platform team’s own pager works here too.
- Runtime primitives - the ingress, the secret store, the identity path, anything customer traffic depends on. These need the shortest recovery time and the most rehearsed path, and may justify standby capacity in another region.
- The change path - CI, the registry, the deploy machinery. Losing these stops change but not traffic. A recovery time in hours, rebuilt from code, is usually fine.
- The conveniences - the catalogue, the docs site, the dashboards for the platform itself. These can be rebuilt in days. Keep an offline export of the docs that matter for recovery, and accept the rest.
Write the targets down and get the people who fund the platform to agree with them. A recovery objective is a business decision about how long the organisation can survive without the ability to change software. The platform team should propose it, not set it alone.
Who owns what
- The platform team owns the mechanism. The cycle map, the bootstrap order, the drift checks on its own infrastructure, the backups of platform state, and the drills.
- Security co-owns layer zero. Who holds the sealed keys, how access is logged, how often it is tested, and how the material is rotated when people leave.
- Leadership owns the objective. How long the organisation can go without deploying, and what it is willing to spend to shorten that.
- Product teams own their own recovery, on top. Their plans stay “redeploy through the platform”, which is now a sentence that is actually true.
How you know it is working
- Time to rebuild from an empty account, from the last full drill. One number, with a date.
- Cycles on the map with a tested break point. The goal is all of them, and new cycles get added to the map when new components arrive.
- Share of platform infrastructure under code with no drift, measured from observation, not from what the repository claims.
- Days since each layer-zero credential was last used successfully. Nothing should be older than a quarter.
- Steps added to the bootstrap order during the last drill. This should fall towards zero. Every non-zero entry is something the document did not know.
Notice what is not on the list: the length of the DR plan, or whether it has been signed off. A plan nobody has executed measures paperwork, not recoverability.
When this is the wrong answer
- Your platform is one box. A boring stack on a single server does not need a cycle map. It needs a tested backup, a short rebuild script, and someone who has run the script on a fresh machine. That is the same idea at the right size.
- Your platform is mostly bought. If CI, the registry, identity and secrets are all SaaS, much of the rebuild is the vendor’s problem, and buying the substrate was partly a recovery decision whether you said so or not. Your layer zero becomes the admin accounts for those vendors and an export of their configuration. Test those exports.
- There is no road yet. If services still deploy by hand, there is no platform to rebuild. Build the first road from code from the start, and this post becomes cheap later.
- A real DR function already exists. If your organisation runs regulated disaster recovery with its own testing calendar, add the platform to it rather than starting a parallel programme. The platform’s job is to bring the cycle map and the bootstrap order to their exercise.
- The platform is changing shape this year. If you are mid-migration to a new substrate, do not perfect the rebuild of the old one. Make the new one rebuildable from its first day, and keep the old one’s backups honest until it is gone.
One guardrail for the other direction. The worst version of this is a platform team that disappears into DR for two quarters: active-active everything, a second region running idle at full cost, and a plan document that becomes the team’s main product. Most platforms do not need hot standby; they need to know their rebuild time and have it be a number the business can live with. Spend on recovery in proportion to the tier, and be honest when the answer to a rare risk is “a few days, and we accept that”.
The bottom line
The internal platform is the tool every other team uses to recover, which is exactly why it is so easy to forget that it needs recovering too. It was built partly by hand, it deploys itself, it keeps its own keys and it hosts its own runbook, and none of that is wrong until the day the account is gone and the first step of the plan needs the thing you just lost.
So make it rebuildable on purpose. Draw the dependency map and find the cycles. Give every cycle a written, tested break point. Keep layer zero small, outside, and exercised every quarter. Version the bootstrap order next to the code. Check drift so the code is the truth. Back up the platform’s own state and prefer rebuilding over restoring. Then run the drill, small first, and publish the time. That is what platform engineering promises to everyone else - a reliable, repeatable way to stand things up - finally applied to itself.
The question I would ask any platform team: if your cloud account were empty tomorrow morning, how long until the first product team could deploy again, and when did you last measure it? If the honest answer is a guess, that is a better use of next quarter than another feature on a road you cannot rebuild.
If your platform has never been rebuilt from scratch, or you suspect the break-glass path is a password nobody has typed in years, that is exactly the kind of problem I help teams work through. Let’s talk.