Rollback And Recovery
Rolling back a bad deploy per platform, how rollback interacts with an already-applied migration, backups and disaster recovery, and secret rotation.
Shipping is only half of operations. This page covers what happens when a deploy goes wrong: how to roll back on each target, why a rollback does not undo a database migration, how to back up the two stateful stores the kit depends on (Postgres and your S3 buckets), and how to rotate secrets safely.
The one idea that ties it all together: migrations run before the new app version takes traffic and are forward-only, so your app must always tolerate a schema that is one step ahead of the code. Get that right and rollbacks are boring; get it wrong and a rollback restores old code against a schema it can't read.
Migrations, Rollback, And Expand/Contract
Every target runs prisma migrate deploy (via scripts/deploy/migrate-database.sh) as a step that completes before the new app version starts serving — the Fly release_command, the Render preDeployCommand, the manual .github/workflows/migrate.yml workflow on Vercel, and the one-shot migrator service on Docker Compose. This is not Vercel-specific: on Fly and Render the migration runs while the old version is still serving traffic, so for a window both the old code and the new schema are live at the same time.
Two consequences fall out of that, and they apply to every rolling deploy:
- The old version must work against the new schema. During a rolling deploy (and during the brief pre-promote window on Vercel), requests hit the old code against the already-migrated database.
- A code rollback does not roll the schema back.
prisma migrate deployis forward-only — the kit ships no down-migrations. Rolling back the app (redeploying the previous image) restores old code but leaves the newer schema in place. So the version you roll back to must also tolerate the new schema.
The discipline that makes both safe is expand/contract (a.k.a. backward-compatible migrations): never remove or repurpose a column, table, or constraint in the same release as the code change that depends on it.
Expand
Add the new column/table/index as nullable or with a default, and deploy code that writes to both the old and new shape. The old code ignores the new fields; the new code tolerates rows that predate them.
Migrate and backfill
Backfill existing rows in a separate step. Reads should not assume the backfill has finished.
Contract (a later release)
Only once every running version reads the new shape — and you're confident you won't need to roll back past this point — ship the destructive change (drop the old column, add the NOT NULL, remove the constraint) as its own migration in a later release.
A destructive (contract) migration is the one thing a rollback can't save you from. If you drop a column in the same release as the code that stops using it and then roll the code back, the restored old version queries a column that no longer exists. Keep contract migrations in their own release, well after the expand, so there's always a safe version to roll back to. To actually undo a schema change you must write and apply a new forward migration; restoring data lost by a destructive change means going to backups (below).
Rolling Back A Deploy
Rolling back restores a previous application version. On every platform it reverts code and the build, not the database — re-read Migrations, Rollback, And Expand/Contract before you pull the trigger.
Fly has no native rollback command — you redeploy a previous image. List the release history with its image references, pick the last good one, and deploy it:
fly releases --image -c apps/docs/fly.toml # or the web app's ./fly.toml
# copy the DOCKER IMAGE of the last good release, then:
fly deploy -i registry.fly.io/<your-app>@sha256:<digest>fly deploy -i reverts only the image (code). It does not revert fly secrets, [env], or other fly.toml changes, and it does not run a down-migration — the schema stays wherever the last forward migration left it. If the bad deploy changed a secret, roll that back separately with fly secrets set.
Render keeps previous deploys and offers a one-click rollback. In the dashboard: open the service → Deploys → find the last healthy deploy → Rollback to this deploy. Render redeploys that image.
A Render rollback redeploys a previous image and does not re-run the preDeployCommand migration in reverse. As on Fly, the database keeps the schema from the most recent forward migration — the version you roll back to must tolerate it.
Vercel keeps every production build, so rollback is instant: Project → Deployments → the last good deployment → ⋯ → Instant Rollback (or Promote to Production). The previous build is already on the CDN, so it takes effect in seconds.
Vercel runs no migrations, so a rollback can't touch the database at all — whatever the .github/workflows/migrate.yml run applied stays applied. Only roll back to a build whose code works against the current schema.
Redeploy the previous image tag. The GHCR workflow tags each build with :<sha> (and :<version>), so pin the last good one instead of :latest:
docker compose pull # or set the pinned tag in your compose/env
docker compose up -dPoint web/docs at ghcr.io/<owner>/<repo>/web:<good-sha> rather than :latest, then bring the stack back up. The database is untouched; do not run the migrator as part of a rollback.
Backups And Disaster Recovery
The kit is stateless; your data lives in two places the kit does not back up for you. Own a backup and restore-test policy for both before launch.
Postgres
Postgres is entirely your responsibility (the kit ships no Postgres service — see Deployment). At minimum:
- Point-in-time recovery (PITR). Most managed providers (Neon, Supabase, RDS) offer PITR — enable it and know your recovery window (e.g. 7 days). PITR is what lets you recover from a bad migration or an accidental
DELETE, not just a lost server. - Scheduled logical dumps. Run
pg_dumpon a cadence (nightly is a common floor) and store the dumps off the database host — ideally in a different region/account than the DB. Dumps are your escape hatch if the managed provider itself has an incident. - Test the restore. A backup you've never restored is a hypothesis. Periodically restore the latest dump into a scratch database and boot the app against it. Track your RPO (how much data you can afford to lose) and RTO (how long a restore takes) against real numbers.
- Migrations run against a direct connection. If you restore into a pooled setup, keep the migrate-vs-runtime
DIRECT_URLsplit from Deployment → Database Connections and Pooling.
S3 buckets
Uploaded objects are not in Postgres. The kit uses two buckets (see ADR 0005): the public images bucket (avatars, org logos) and the private attachments bucket (S3_BUCKET_NAME_ATTACHMENTS, served only via signed URLs). Both hold user data that a database restore won't bring back.
- Enable object versioning on both buckets so an overwrite or delete is recoverable.
- Add a cross-region replication or lifecycle backup policy appropriate to your provider.
- Keep the
tmp/-expiry lifecycle rule from Storage; it's orthogonal to backups but part of a healthy bucket policy.
Back up Postgres and the buckets together and consistently. A row in User that references an avatar key is only meaningful if the object still exists — restoring the database to a point the bucket has since diverged from leaves dangling references. Snapshot both at compatible points in time.
Secret Rotation
Secrets should be rotated on a schedule and immediately after any suspected exposure. The blast radius differs per secret — know it before you rotate in production.
| Secret | What rotating it does |
|---|---|
BETTER_AUTH_SECRET | Signs and encrypts every session cookie. Rotating it invalidates all active sessions — every user is signed out and must log in again. Also invalidates in-flight email verification / password-reset links (their tokens are signed with the old secret). Rotate during a low-traffic window and expect a login spike. |
NEXT_SERVER_ACTIONS_ENCRYPTION_KEY | Encrypts server-action payloads across replicas. Rotate it on all replicas at once; a split fleet (some old, some new) can't decrypt each other's actions and you'll see intermittent form failures until the roll completes. |
OAUTH_PROXY_SECRET | Shared secret for the preview OAuth proxy. Must be identical across production, preview, and any local proxy. Rotate everywhere in one change or previews fail with state_mismatch. Safe to rotate independently of BETTER_AUTH_SECRET. |
STRIPE_WEBHOOK_SECRET | Verifies inbound Stripe webhooks. Roll the endpoint's signing secret in the Stripe dashboard, then update the runtime env; there's a brief window where in-flight events signed with the old secret return 400 (Stripe retries them). |
STRIPE_SECRET_KEY, OAuth client secrets, AWS_*, UPSTASH_*, provider API keys (RESEND_API_KEY, …) | Provider-side credentials. Create the new credential first, deploy it as a runtime secret, confirm traffic works, then revoke the old one — this overlap avoids downtime. None of these sign session cookies, so rotating them does not sign users out. |
DATABASE_URL / DIRECT_URL | Rotate the DB password at the provider, update the runtime secret (and the migrate-time DIRECT_URL wherever migrations run), then redeploy. Long-lived container targets pick it up on restart. |
All of these are runtime secrets, not build args (except the NEXT_PUBLIC_* values, which are build-time — see Deployment → Build-Time vs Runtime Env). Rotating a runtime secret takes effect on the next restart with no rebuild. After any rotation, run pnpm setup:doctor against the environment to confirm nothing is left half-rotated.
Where To Go Next
Deployment
Per-target migration handling, the release hooks referenced here, and where each runtime secret gets set.
Going To Production
The pre-launch checklist and post-deploy smoke test that this page backstops.
Database
How prisma migrate deploy works, the migrator script, and the pooled-vs-direct connection split.
Monitoring
The /api/health readiness probe that gates a rollout and the error tracking you'll watch during a rollback.