Your Platform Stops at the Database

TL;DR - Most internal platforms will get a stateless service to production in an afternoon and then hand you a ticket form the moment you ask for a database. Platform teams pave the half that is cheap to get wrong, because a stateless mistake is undone by a redeploy and a data mistake is not undone at all. So the riskiest part of every service, the part where the mistakes are permanent, is the part with no road: provisioned by hand, accessed with a shared password, migrated by whoever is brave, and protected by backups nobody has ever restored. “We use a managed database” does not fix this, because the provider manages the engine and nothing about how your teams use it. Pave the data road the same way you paved the rest: a database from the template, access by identity, schema migrations run as deploys, restores as a drilled self-service command with a measured time, and a written line between what the platform owns and what the team owns.

I build and run platforms for a living, and there is a test I like more than any maturity model. I ask a team to show me how a new service gets to production. Usually it goes well. There is a template, a pipeline, a dashboard, and twenty minutes later a hello-world is running. Then I ask them to do it again, for a service that needs a database. The demo stops. Somebody opens a ticket form, or a wiki page titled “Requesting a database (temporary process)”, last edited two years ago.

That moment is the most consistent finding I have. The road is paved right up to the edge of the data, and then it stops.

The tell

Three questions, asked about the database behind any service that matters:

• How long does a new service wait for its database? If the answer is longer than the rest of the road put together, the data layer is where your lead time actually lives, and no amount of pipeline tuning will touch it.
• When did you last restore a production backup into something that worked, and how long did it take? Not “the backups are green”. A restore, into a running service, against a clock. If nobody can name a date, you do not know whether you have backups. You know that a job exits zero.
• Who runs schema migrations, and how? If the answer is “the team, from a laptop, during a quiet hour” or “it depends who is on”, the most dangerous change your organisation makes has no road at all.

If all three answers are vague, the platform is not finished. It has paved the part that was easy to pave.

Why the road stops there

Nobody decides to leave the data out. It happens for reasons that each look sensible on their own.

• Stateless is forgiving. A bad deploy of a stateless service is undone by deploying the previous version. That makes it safe to pave quickly, iterate on, and get wrong in public. Data has no previous version you can redeploy. A bad migration or a dropped table is undone by a restore, if you are lucky, and with data loss if you are not. Platform teams, quite reasonably, are slower to put their name on something whose mistakes are permanent.
• Somebody already owns it, sort of. There is a DBA, or there was. There is a ticket queue with a person behind it. It is slow, but it exists, so paving it never feels urgent. That queue is exactly the human bottleneck platform engineering exists to dissolve, just in a corner nobody looks at.
• “Managed” sounds like “solved”. The organisation moved to a managed database service and quietly concluded the data problem was handled. The engine is handled. That is all that is handled.
• The first road was built for the first workload. The first golden path is usually a stateless HTTP service, because that is the most common shape and the easiest to get adopted. That was the right call. The mistake is treating it as the finished platform rather than the first road.

The result is an inversion I see everywhere: the platform’s coverage is inversely proportional to the cost of getting things wrong. Deploys, which are cheap to reverse, have a beautiful road. Data, which is not, has a form.

What “managed” does not manage

I am firmly in favour of buying the database engine. Running your own replication, failover and storage is undifferentiated work, and the honest build-versus-buy line puts it squarely on the buy side. But the line has two sides, and the thin layer of opinion on top is still yours. Here is what a managed service does not do for you:

• Restore. It takes snapshots. It does not prove that a snapshot, restored, gives you a working service with the data you expected, in a time anyone has agreed to.
• Schema change. It will happily execute a migration that locks your largest table for forty minutes at peak.
• Access. It gives you a master password. What happens to that password next is entirely up to you, and in most estates it ends up in six places.
• Engine upgrades. It will tell you a major version is going end of life. It will not migrate your extensions, test your queries or schedule the work.
• Sizing and connections. It will let a team pick the largest instance class “to be safe”, and it will let a service open connections until the database falls over.
• The data’s lifecycle. Retention, deletion, what counts as personal data and where the copies went. The provider has no opinion.

Every one of those is a decision someone in your organisation is making, per database, by hand, mostly without noticing.

A backup is a hypothesis

If I could change one thing in most estates it would be this. A backup that has never been restored is not a backup. It is a hypothesis that you have a backup, and the experiment that tests it usually runs for the first time during the worst incident of the year.

What I would put in place, cheapest first:

• An automated restore drill. Once a week, a job picks a production database, restores the latest backup into a scratch environment, starts the owning service against it, runs a handful of read queries that should return known shapes of data, records how long it all took, and throws it away. It is the data equivalent of the synthetic hello-world that deploys through the real road every thirty minutes, and it should page someone when it fails.
• A measured restore time per database. Not a target in a document. The number from last week’s drill. For the largest database in the estate, this number is almost always much larger than anyone assumed, and it is the number that decides how long your worst outage lasts.
• Restore as a self-service command. A team that needs to recover from a bad migration should be able to restore to a point in time, into a side-by-side copy, without a ticket. If restoring requires the platform team, then a team’s worst day is gated on yours.
• Restores in the runbook. The generic half of the runbook, the half the platform should generate rather than ask every team to write, should include the exact restore command for this service’s database and the date it was last drilled.

The drill will fail the first few times. That is the point of running it on a Tuesday afternoon rather than finding out at 3am.

Schema migrations are deploys

The other gap is schema change, and it is where I see the most avoidable incidents. Application deploys get canaries, health checks and rollback. Migrations, which cannot be rolled back, get run from a laptop.

Treat them as what they are: the highest-risk deploys you make.

• Run them in the pipeline, not by hand. A migration step on the road, with the same audit trail as any other change, run by the pipeline’s identity rather than a human’s.
• Expand, then contract. Add the new column, deploy code that writes both, backfill, switch reads, and only then drop the old one, in a later release. Every step is backward-compatible with the code that is running, so the application can roll back even though the data cannot.
• Lint for the dangerous operations. Most of the outages I have seen from migrations come from a short list: adding an index without doing it concurrently, rewriting a large table, adding a constraint that validates every row under a lock. A check in the pipeline that recognises those and blocks them with a message naming the safe alternative is cheap, and it is a perfect example of a control that comes with its own road rather than a rule nobody can follow.
• Test against realistic volume. A migration that takes two seconds against a staging database with a thousand rows can take an hour against production. Staging tells you what you want to hear unless it has production’s shape. The restore drill gives you a free, recent, production-sized copy to rehearse against, which is one more reason to build it first.

Pave the data road

With restores and migrations in place, the rest of the road is the familiar work, applied to state:

• A database from the template. Asking for a database is a line in the service’s config, not a ticket. One module, with the defaults that matter already on: encryption, backups, retention, monitoring, a connection pooler, tags that say who owns it and what it costs. The team picks a size from three, not from forty.
• Access by identity, not by password. The service reaches its database with a workload identity and short-lived credentials. Humans get time-boxed read access through a logged path, and write access through break-glass that opens a ticket automatically. Nobody has the master password, including the platform team, most days.
• Engine upgrades on a cadence. The database engine is substrate, and substrate is upgraded by the platform on a schedule with a canary cohort, the same way the rest of the fleet stays current. A major engine upgrade done every year is a routine. One done after five years is a forced migration.
• Every database on the road shows up in the inventory. Owner, engine version, size, last drilled restore, cost. Databases created off-road show up too, because the inventory is built from what is observed, not what was declared, and an untracked production database is the most expensive kind of shadow estate there is.

Who owns what

Paving the data road does not mean the platform team owns everyone’s data. It means drawing the line deliberately instead of leaving it wherever the last incident left it.

• The platform owns the mechanism. The engine, its upgrades, the provisioning module, the backup schedule, the restore command and the drill that proves it works, the access road, the migration step and its lint.
• The team owns the meaning. The schema, the queries, the indexes their access patterns need, the correctness of the data, the retention decisions, and the content of every migration.
• The data specialist enables. If you have DBAs or a data reliability engineer, the best use of them is not the ticket queue. It is building the lint rules, reviewing the gnarly migrations, teaching teams to read a query plan, and then stepping back. That is an enabling team with an exit date, not a permanent gate.

Write that split down, put it next to the escalation boundary, and the 2am question “is this ours or theirs?” becomes a lookup.

How you know it is working

• Time to a new database. Measured the same way you measure time to a new service, and ideally indistinguishable from it.
• Restore drill success rate, and the restore time of the largest database. Both from the last drill, not from a document.
• Share of production databases created on the road. The rest are either legacy you have a plan for or a shadow estate you need to find.
• Engine version skew. How many major versions are running. One or two is a converged fleet.
• Migrations that took a lock in production. This number should trend to zero, and every non-zero entry is a lint rule you have not written yet.

Notice what is not on the list: number of databases provisioned. Making databases easy to create is the goal, not the scoreboard.

When this is the wrong answer

• You run one database. A small company with one well-run Postgres, or a boring stack with SQLite on a single box, does not need a provisioning road. It still needs the restore drill. That part is never the wrong answer.
• New databases are rare. If the organisation creates three a year, do not build self-service provisioning. Write a good module, have someone experienced run it, and spend the effort on restores and migrations instead, which happen every week regardless.
• A mandated DBA function owns production data. In some regulated environments, a separate team is required to approve schema changes and hold production access. Fine. Pave the road up to their door: the migration step, the lint, the drills and the evidence, so their review is a fast check of a well-formed change rather than an archaeology exercise.
• The analytics warehouse is a different problem. Pipelines, lineage and a data platform for analysts are their own discipline. This post is about the operational databases behind services. Do not let one programme swallow both.
• There is no road yet. If stateless services still have no paved road, build that first. The data road plugs into the template, the identity model and the pipeline. Without those, you are building a database portal in a vacuum.

One guardrail for the other direction. Once databases are easy, people create a lot of them, and “database per service” turns into a hundred tiny instances, each with its own backups to drill and its own engine to upgrade. Easy creation needs the same portfolio discipline as easy roads: sensible sharing where the boundaries allow it, idle instances flagged and reclaimed, and a cost line nobody can miss.

The bottom line

Platform teams pave the stateless half of a service because it is forgiving, and in doing so they leave the unforgiving half to tickets, shared passwords, migrations from laptops and backups that have never met a restore. That is backwards. The value of a road is highest exactly where mistakes are most expensive, and nowhere are they more expensive than the data.

So extend the road past the edge. Buy the engine and own the layer on top. Prove backups with a weekly restore drill and publish the time. Run migrations as deploys, expand then contract, with a lint that knows which operations lock. Provision from the template, connect by identity, upgrade on a cadence, and write down who owns the mechanism and who owns the meaning.

The question I would ask any platform team: if your largest production database disappeared at noon today, how long until the service is back, and when did you last measure it? If the honest answer is a guess, that is the next thing to build, before the next feature on the part of the road that already works.

If your paved road stops at a ticket form the moment a service needs state, or nobody can say when a backup was last restored, that is exactly the kind of problem I help teams work through. Let’s talk.