Priyanka R.
HomeUsesTalksBlogContact
HomeUsesTalks
All posts

Oct 14, 2026 · 1 min read

The migration that ran halfway

We ran a schema migration on a Friday. It stopped at 61%. Here is what I check now before anything touches a big table.

It was a Friday, which is the first mistake, and the second was that the migration looked trivial: add a column with a default, backfill it, add an index.

On our staging copy it took four minutes. In production it ran for forty, held a lock it should not have held, and then the deploy pipeline timed out and killed it at 61%. Half the rows had the new value. Half did not. The index did not exist.

What actually went wrong

  • The backfill was one UPDATE statement over 300 million rows.

  • Staging had a tenth of the data and none of the write traffic.

  • Nobody had decided what "rolled back" meant for a half-finished backfill.

Nothing was lost, but we spent the weekend writing a script to finish the job in batches.

What I check now

© 2026 Priyanka R. · Built on a Sunday. Written in Berlin.

priyanka.raghunathan@example.com

Blog
Contact
  • Can this run in batches, and can it stop and resume at any batch?

  • Does any step take a lock stronger than it needs? CREATE INDEX CONCURRENTLY, always.

  • What does the application do if it sees the old schema, the new schema, or both?

  • Who is awake when it runs?

  • A migration is not done when it is written. It is done when it has failed once in your head.

    None of this is new. I knew all of it before that Friday. Knowing is not the same as having a checklist that somebody actually reads in code review, so now the checklist is in our pull request template.

    • postgres
    • migrations

    Keep reading

    • Jul 29, 2026 · 1 min read

      SKIP LOCKED is a perfectly good queue

      Before you add a message broker for background jobs, try the database you already run.

    • Mar 6, 2026 · 1 min read

      Seven years of job interviews from both sides

      I have been interviewed about forty times and have interviewed maybe two hundred people. Some notes, not advice.