Oct 14, 2026 · 1 min read
The migration that ran halfway
We ran a schema migration on a Friday. It stopped at 61%. Here is what I check now before anything touches a big table.

It was a Friday, which is the first mistake, and the second was that the migration looked trivial: add a column with a default, backfill it, add an index.
On our staging copy it took four minutes. In production it ran for forty, held a lock it should not have held, and then the deploy pipeline timed out and killed it at 61%. Half the rows had the new value. Half did not. The index did not exist.
What actually went wrong
The backfill was one UPDATE statement over 300 million rows.
Staging had a tenth of the data and none of the write traffic.
Nobody had decided what "rolled back" meant for a half-finished backfill.
Nothing was lost, but we spent the weekend writing a script to finish the job in batches.