Migration Rollback Playbooks

Rollback runs as a gated sequence: a breached threshold forces an explicit go/no-go before parallel DNS and redirect reversals, then verification.

Executive Summary

A rollback is a pre-authorised, rehearsed reversal of a migration to a known-good state, not an improvised scramble. This runbook gives the release owner, on-call SRE, and SEO lead a single source of truth: when to abort, who decides, and the exact order in which DNS, redirects, and server configuration revert. The outcome is a bounded recovery window — typically under 15 minutes for redirect-layer faults and under one TTL cycle for DNS faults — with link equity and crawl signals preserved. Treat every cutover as reversible until the post-launch soak window closes.

The word “pre-authorised” is carrying the most weight in that summary. Every element of a rollback — the trigger values, the decision-maker, the exact commands, the communication template — has to exist before the window opens, because the defining property of the situation in which you need it is that nobody has spare attention for authoring anything. A migration where the reversal is designed during the incident does not have a rollback plan; it has an outage with good intentions.

It is also worth separating rollback from failure. Teams resist reverting because it feels like an admission that the migration did not work, so they spend the cheapest thirty minutes of the recovery arguing about whether things are really that bad. Framed correctly, a rollback is a scheduled, rehearsed operation that returns the system to a state you know works, at a cost you measured in advance. Rolling back at minute ten and retrying two nights later is an ordinary outcome; pressing on because reverting felt like giving up is how a two-hour incident becomes a two-week reconciliation.

Prerequisites

A pre-migration snapshot of every authoritative zone file, redirect map, and server config under version control
Lowered DNS TTLs in place before cutover (see TTL planning below)
Access to DNS provider APIs (Cloudflare API token, aws CLI with Route 53 permissions)
Server reload rights (nginx -t, systemctl reload) on every origin and edge node
Baseline metrics for error rate, p95 latency, conversion, and crawl errors captured before launch
A named decision-maker with explicit authority to call the rollback

Two of these are routinely present in name only. Version-controlled snapshots are worthless if nobody has restored one — check out the pre-migration zone file and redirect map into a scratch environment and confirm they load, because a config that references a path or an upstream that no longer exists will fail nginx -t at precisely the wrong moment. And baseline metrics need to be stored figures rather than dashboard views: during a rollback the question is “is p95 back to normal”, and “normal” has to be a number somebody can read out, not a shape somebody remembers.

Step-by-Step Execution

1. Define Objective Rollback Triggers

Codify the numeric signals that force a reversal before launch, never during the incident. Set ceilings for 5xx/4xx rate, p95 latency, traffic drop, conversion drop, and crawl-error spikes, each tied to a monitoring window. Document these in Rollback Trigger Thresholds so on-call staff act on data, not intuition.

Good triggers share three properties: a metric, a threshold, and a window. Drop any one and the trigger stops working. “5xx rate above 2%” without a window fires on a single bad minute during a deploy; “sustained errors” without a threshold is an opinion; and a threshold on a metric nobody is already collecting is a trigger that will never fire because nobody thought to look. Derive each threshold from the baseline rather than from a round number — if your normal 5xx rate is 0.4%, a 2% ceiling is a meaningful five-fold degradation, whereas on a site that normally sits at 1.8% it is noise.

Include at least one business metric alongside the technical ones. Error rate and latency catch infrastructure faults, but the most expensive migration failures are functionally silent: a checkout that returns 200 while failing to charge, a search page that renders with zero results, a form that submits into nothing. Conversion rate or completed-order count against the same hour of the previous week catches these when nothing else will, and it is usually the trigger that fires first on a genuinely bad migration.

2. Assign Decision Authority

Name one accountable owner per phase with the power to abort, plus a clear escalation tier if recovery stalls. Ambiguity here is the single largest cause of prolonged outages during failed migrations. Pre-agree that the owner’s call is final within the soak window to avoid debate while error rates climb.

The authority has to be genuinely delegated, not nominally assigned. If the named owner believes that reverting will require them to justify the decision afterwards to someone who was not in the room, they will hesitate — and hesitation is the entire cost. Make it explicit in advance that invoking a documented trigger is always the correct action even when the fault later turns out to be minor, and that the post-mortem examines the trigger’s calibration rather than the owner’s judgement. A team that has been burned once for “rolling back unnecessarily” will not roll back promptly again.

3. Prepare the DNS Rollback Path

Keep the legacy A/AAAA/CNAME values staged as a ready-to-apply change set so reversion is a single API call. The speed of this path depends entirely on the TTL you set beforehand via TTL Optimization Strategies. Detailed reversion steps live in DNS Rollback Procedures.

Stage the reversion as a committed artefact — a JSON change batch, a saved API payload — rather than a set of instructions for reconstructing one. The difference matters because reconstructing the legacy record set requires knowing what it was, and the moment you most need that information is the moment your attention is least available. Generate the rollback payload at the same time you generate the cutover payload, from the same source of truth, and store them side by side. Then verify the rollback payload parses and targets the right zone before the window, since a malformed change batch is discovered only when it is rejected.

4. Prepare the Redirect Rollback Path

Stage the previous redirect map and server config as an atomic swap so the routing layer reverts without downtime. Coordinate this with your live URL Mapping & Redirect Architecture so the restored rules match the last verified state. Recovery specifics — config restore, cache purge — are in Redirect Rollback & Recovery.

5. Rehearse the Reversal in Staging

Treat the drill as the deliverable of this phase rather than the documentation. A runbook that has been read is not a runbook that has been run, and the gap between the two is where recovery time hides. Run a full rollback drill against staging and time each step end to end. A rollback that has never been executed is an assumption, not a plan. Capture the wall-clock duration of DNS reversion plus one TTL cycle to set realistic recovery expectations with stakeholders.

Time the drill honestly and publish the number. Rehearsals have a way of producing optimistic figures because the person running them knows exactly which command comes next and has every credential already open in a terminal. Run the drill with the on-call engineer who will actually be paged, from a cold start, following only the written runbook — and if that takes twenty minutes against a fifteen-minute promised recovery window, the correct response is to fix the runbook or widen the promise, not to record the fast number.

The drill is also the cheapest place to discover the dependencies nobody documented: an API token that expired, a CDN purge that requires a permission the on-call role does not have, a config that references a path removed in a later refactor. Every one of these is trivial to fix on a Tuesday afternoon and catastrophic to discover mid-incident. Re-run the drill after any significant infrastructure change, not just before migrations.

6. Execute and Verify Recovery

On a triggered abort, run DNS and redirect rollbacks in parallel, then verify with dig and curl against the legacy targets. Confirm error rate and latency return to baseline before declaring recovery. Reconcile crawl and index signals afterward through Search Console Handover to catch residual indexing drift.

Verification must be independent of the system that told you to roll back. If the monitoring pipeline itself is degraded — which is common when an origin is unhealthy — the dashboard returning to green may reflect missing data rather than recovered service. Confirm recovery with direct probes: dig against several resolvers for the record, curl against the legacy origin for status and headers, and one real transaction through the actual user path. Only then trust the dashboards to tell you the aggregate picture.

7. Communicate Status Continuously

Post a single authoritative status to stakeholders at trigger, mid-rollback, and recovery confirmation. Tie this to the broader plan in Pre-Migration Auditing & Risk Assessment so the rollback record feeds the post-mortem. Silence during a rollback erodes trust faster than the outage itself.

Those seven steps are not equally reversible, and the difference is what determines how long you actually have. Some layers revert at the speed of an API call; others revert at the speed of a cache you do not control.

Run the fast layers in parallel and treat the slow ones as damage you are racing to avoid, not as steps you can schedule.

Technical Configs

Each fragment below is a reversal, not a change: the value being applied is the one captured before the migration, and every one is designed to be executed without editing. Keep them in the migration branch alongside the cutover payloads so a reviewer can see both directions of every change in a single diff, and so the rollback is versioned with the thing it reverses rather than living in a document that drifts.

AWS Route 53 — revert an A record to the legacy origin via the CLI:

# UPSERT overwrites the live record with the staged legacy value
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456ABCDEFG \
  --change-batch file://rollback-a-record.json
# rollback-a-record.json contains action UPSERT, name www.example.com, value 203.0.113.10

Cloudflare API — patch a DNS record back to the previous IP:

# Restore the pre-cutover A record in a single call
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records/$REC" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"type":"A","name":"www","content":"203.0.113.10","ttl":60}'

Note the explicit ttl in the Cloudflare payload. A reversion that restores the record value but silently resets the TTL to the provider default undoes the agility you spent days buying, and it does so at exactly the point where you may need to revert the reversion. Whichever provider you use, assert the TTL in the rollback payload rather than relying on the record’s current setting being carried over.

Nginx — atomic redirect config swap with validation gate:

# Symlink the previous known-good map, then validate before reload
# ln -sfn /etc/nginx/maps/redirects.prev.conf /etc/nginx/maps/redirects.active.conf
map $request_uri $rollback_target {
    include /etc/nginx/maps/redirects.active.conf;
}
# nginx -t && systemctl reload nginx   (reload is zero-downtime)

Apache — restore prior rewrite rules and test config:

# Swap in the archived rules file, then validate syntax before graceful restart
# cp /etc/apache2/conf-available/redirects.prev.conf /etc/apache2/conf-available/redirects.conf
RewriteEngine On
RewriteMap legacy "txt:/etc/apache2/maps/legacy-map.txt"
RewriteRule ^(.*)$ ${legacy:$1} [R=301,L]
# apachectl configtest && apachectl graceful

Both server examples above share one property worth copying regardless of your stack: the reversal is a symlink or file swap followed by a validation gate, never an edit. Editing a live config under pressure means the rollback itself can introduce a syntax error, and nginx -t failing after you have already reverted DNS leaves you with neither the old routing nor the new. Stage the previous file whole, point at it atomically, validate, then reload.

The only sequential dependency is the abort call itself; everything after it runs on two independent layers and should be executed by two people at once.

One organisational note that belongs with the configs rather than the prose: whoever holds these credentials during the window must be awake and reachable for its entire duration. A rollback plan that depends on an API token held by one engineer, or on a CDN account with a single named administrator, has a single point of failure that no amount of rehearsal addresses. Provision at least two people with the necessary access for the soak period, confirm both can authenticate before the cutover begins, and include that confirmation in the pre-flight checklist alongside the technical items.

Validation & Rollback

Confirm the migration is healthy before closing the soak window, and verify the legacy state is fully restored after any reversal.

Post-Rollback Validation Checklist:

Confirm dig +short www.example.com @1.1.1.1 returns the legacy IP across 3+ resolvers
Verify curl -sI https://www.example.com | grep -i location shows pre-migration redirect targets
Confirm 5xx rate returns below baseline in the monitoring dashboard
Validate p95 latency is within the pre-launch ceiling
Purge CDN edge cache and confirm fresh objects via cf-cache-status: MISS then HIT

Common Pitfalls:

Setting triggers as vague “if it looks bad” criteria instead of numeric ceilings with windows
Forgetting that DNS rollback is bounded by the TTL set before cutover, not the moment of reversion
No single named owner, causing decision paralysis while errors compound
Reverting redirects without purging CDN cache, serving stale routing tables
Treating the rollback as untested theory rather than a rehearsed drill

Rollback Protocol:

Trigger fires when any threshold in Rollback Trigger Thresholds is breached for its full window
Named owner issues the abort and broadcasts the first status update
Apply the staged DNS reversion via Route 53 / Cloudflare API and start the TTL countdown
Swap redirect config atomically (nginx -t && systemctl reload) and purge CDN cache
Verify legacy responses with dig and curl, confirm metrics return to baseline, then broadcast recovery

One caveat on the checklist above: cf-cache-status: MISS followed by HIT proves the edge re-fetched, but it proves nothing about what it fetched. Pair the cache check with an assertion on the content itself — a build hash, an asset fingerprint, a distinguishing string that only the legacy build contains — so a successful purge that repopulated from the wrong origin is caught rather than celebrated. The same applies to the dig check: confirm the legacy IP specifically, not merely that the answer changed.

Finally, close the loop on the soak window explicitly. A rollback plan that is “armed until someone remembers to stand it down” tends to decay into a plan nobody is actually watching, while the staged artefacts quietly go stale — the redirect map diverges from production, the DNS payload references a decommissioned IP, the drill has not run in months. Give the soak window an end date, run a final verification against baseline metrics on that date, and formally retire the staged rollback assets in the same change that declares the migration complete. An expired rollback plan left in place is more dangerous than none, because someone will eventually apply it.

FAQ

How fast can a DNS rollback actually take effect? No faster than the TTL set before cutover allows. If you lowered the TTL to 60 seconds beforehand, most recursive resolvers adopt the reverted record within a few minutes; if the TTL was still 86400 seconds, some resolvers cache the broken record for up to a day. Always lower TTL before migrating, never after the fault appears.

Who should have authority to call a rollback? One named release owner per phase, with a documented escalation tier. The owner acts on the pre-agreed numeric triggers, not on opinion, and their call is final within the soak window. This removes the debate that otherwise lets error rates climb while stakeholders argue.

Should DNS and redirect rollbacks happen in sequence or in parallel? In parallel. They affect independent layers — name resolution versus HTTP routing — and serialising them doubles the recovery window. Run the DNS API change and the atomic redirect swap simultaneously, then verify both.

How long should the post-launch soak window stay open? Keep rollback fully armed for at least 72 hours, the period during which the majority of crawl re-indexing and traffic-pattern anomalies surface. High-traffic enterprise moves often hold the window for a full week before declaring the migration irreversible.

What is the difference between a rollback and a hotfix? A rollback returns the system to the previously known-good state wholesale; a hotfix patches forward. During the soak window, prefer rollback — it is rehearsed, bounded, and reversible — and reserve forward fixes for after the migration is declared stable. The exception is a narrow, well-understood fault affecting a small share of routes while traffic is still weighted low: reverting DNS there costs a full propagation cycle before you can retry, and fixing forward is genuinely faster. Write the distinction into the runbook so the on-call engineer is applying a rule rather than inventing a taxonomy at three in the morning.

What should we do if the rollback itself fails? Escalate immediately rather than iterating, and fall back to the crudest option that restores service — usually a maintenance page served from the edge, which is always available even when both origins are unhealthy. A failed rollback almost always means the “known-good” state was not actually known-good: a config referencing a decommissioned upstream, a snapshot that was never restored, credentials that expired. That is a rehearsal gap, and the post-mortem action is to add the missing drill rather than to add another step to the runbook.

Does rolling back damage search rankings on its own? Usually far less than leaving a broken migration in place. Crawlers tolerate a short period of instability; what they penalise is a sustained pattern of errors, redirect chains, or missing content. A revert that restores the previously indexed URLs within a TTL cycle typically leaves no lasting trace. The genuine risk comes from repeated flip-flopping — migrating, reverting, migrating again over several days — which produces conflicting canonical signals. Revert decisively, then wait until the fix is verified before retrying.

How does a rollback plan differ for a domain change versus a hosting move? A hosting move usually keeps the URLs identical and swaps the infrastructure beneath them, so reverting restores the previous origin and nothing external has to be told. A domain change alters the URLs themselves, which means search engines, inbound links, email signatures, and third-party integrations have all begun learning the new address. Reverting the infrastructure is equally fast, but the signals you sent outward cannot be recalled — which is why a domain change needs a longer soak window, a stricter set of triggers on crawl and index metrics, and a plan that assumes the old and new hostnames must both keep answering for months rather than days.

What belongs in the post-mortem after a rollback? Three things, in order. First, whether the trigger fired at the right moment: too late means the threshold or window needs tightening, too early means it needs calibrating against a real baseline. Second, how long each phase of the reversal actually took against the rehearsed figure, since a gap between the two is a rehearsal gap and not bad luck. Third, what the fault actually was and whether it was detectable before cutover — most migration failures are reproducible in staging once you know what to look for, and the value of the incident is the test you can now write. Deliberately exclude any assessment of whether reverting was the right call; treating that as settled is what keeps the next rollback fast.

← Back to Home