DNS Rollback Procedures

Context

When a cutover fails, reverting DNS is the bluntest and most reliable way to send all traffic back to the legacy origin — but its speed is decided hours earlier, by the TTL you set before the migration. This guide covers reverting A, AAAA, and CNAME records cleanly, the cache implications that bound your recovery window, and why a low pre-cutover TTL is the difference between a 2-minute and a 24-hour rollback. It operates under Migration Rollback Playbooks, which owns the trigger and decision-authority side of the reversal.

DNS rollback timeline A reverted record propagates to resolvers over one TTL cycle; a low pre-set TTL shortens the window from hours to minutes. DNS Rollback Timeline Apply revert Propagate Full recovery UPSERT legacy IP ≤ 1 TTL cycle dig confirms legacy TTL 60s → minutes to recover  ·  TTL 86400s → up to a day Lower the TTL before cutover, not after the fault.
The reverted record reaches resolvers within one TTL cycle; the TTL chosen before cutover sets the entire recovery budget.

The uncomfortable property of a DNS rollback is that almost none of its speed is under your control at the moment you invoke it. Applying the reverted record takes one API call and a second or two. Everything after that is waiting for caches you do not own to expire on a schedule you fixed days earlier. That inversion is worth internalising, because it changes what “preparing to roll back” means: the preparation that matters happened during the TTL reduction phase, and the artefacts you stage now only remove the seconds, not the minutes.

It also changes what you do while waiting. Since the reversion itself cannot be accelerated, the productive work during the TTL cycle is bounding the damage for users still pinned to the broken target — putting the new origin behind a maintenance page rather than leaving it serving errors, or where possible making it redirect to the legacy host. A broken origin that returns a clear holding page for the remainder of its cache window is a far better outcome than one returning 502s, and it costs nothing to arrange.

Pre-flight Checks

Confirm the legacy targets and TTL state before you need them, so reversion is a single staged change.

  • Snapshot current and legacy A/AAAA/CNAME values and store the legacy change set ready to apply.
  • Confirm the TTL was already lowered before cutover via TTL Optimization Strategies — verify the effective TTL with dig, not just the zone file.
  • Check for registrar TTL floors that may override your value and extend the window.
  • Verify API credentials (CF_TOKEN, aws Route 53 permissions) work with a dry-run.
  • Confirm at least two people hold working credentials for the whole soak window, so the reversion does not depend on one person being reachable.
  • Check that the legacy origin is still serving correctly right now, rather than assuming it is because nobody has switched it off.

DNS Rollback Readiness Checklist:

Execution Steps

Revert records, increment the zone serial, and verify propagation across multiple resolvers.

1. Apply the Reverted Records

Push the staged legacy A/AAAA/CNAME values via UPSERT so the change is idempotent and atomic per record. Update apex, www, and any service subdomains in one batch to avoid split routing. The full command-level walkthrough with verification is in Reverting DNS Records During a Failed Cutover.

Batch the records rather than applying them one at a time. A rollback that reverts the apex, then www, then the API subdomain over the course of a minute creates a window in which different hostnames point at different origins — which for a site whose pages load assets and API calls across subdomains is its own failure mode, and one that looks like an application bug rather than a routing state. Both Route 53 change batches and the equivalent bulk endpoints elsewhere apply atomically; use them, and include every hostname that moved during the cutover, including the ones nobody expects to matter.

Do not forget the records that are easy to overlook because they were changed by someone else. Mail hostnames, a status page, a CDN validation record, an SPF entry naming the new origin’s address — each of these may have been updated as part of the migration by a different person on a different day, and each will remain pointing at the new infrastructure after your rollback unless it is in the staged change set. Build that set from a diff of the zone before and after the migration rather than from memory.

2. Force a Zone Serial Increment

Bump the SOA serial on every authoritative nameserver so secondaries pull the reverted zone. A reverted record that never propagates to secondaries leaves a fraction of resolvers on the broken target. Confirm all NS hosts report the new serial before trusting propagation.

Query each authoritative nameserver by name to confirm the serial, rather than querying the domain and trusting whichever server answers. The failure this catches is specific and common: a secondary that missed the transfer continues serving the pre-rollback zone indefinitely, and because it is one of several nameservers, only a fraction of resolvers ever consult it. The result is a rollback that appears successful from your desk and leaves a slice of users on the broken origin for hours. Enumerate the NS set, query each in turn, and compare serials before you start the recovery clock.

3. Wait Out the TTL Cycle

Start the clock at the moment of reversion; resolvers holding the broken record release it only after their cached TTL expires. This is why the pre-cutover TTL governs everything — coordinate the wait with the cache mechanics in DNS Propagation Tracking. Do not declare recovery until at least one full TTL has elapsed.

Resist the temptation to keep re-pushing the record while you wait. Repeated UPSERT calls do not reach resolvers any faster, they generate noise in the change log that makes the incident harder to reconstruct afterwards, and on some providers a rapid sequence of changes triggers rate limiting at precisely the wrong moment. Apply the reversion once, confirm it landed on every authoritative nameserver, and then treat the remaining time as observation rather than action.

4. Verify Across Global Resolvers

Query 1.1.1.1, 8.8.8.8, and 9.9.9.9 plus regional resolvers to confirm the legacy IP is returned everywhere. A single outlier usually means a resolver-enforced minimum TTL or a stale secondary. Cross-check that the restored origin returns HTTP 200 before standing down, coordinating with Migration Rollback Playbooks for the recovery broadcast.

Those four steps divide cleanly into work you perform and time you endure, and confusing the two is what makes rollbacks feel longer than they are.

Which parts of a DNS rollback you control The reversion, serial bump and verification take seconds to minutes and are under your control, while the cache expiry that dominates the recovery window is fixed by the TTL chosen before the migration. You control the first minute; the TTL controls the rest Under your control seconds each Not under your control decided days ago UPSERT serial bump verify NS resolver caches expiring — one full TTL cycle abort called recovery confirmed Use the lower lane productively: put a holding page on the broken origin so pinned users get a clear message, not a 502.
Nothing you do after the abort shortens the lower bar. The only lever on it was set during the TTL reduction phase.

One more preparation worth making explicit: the staged change set should be generated from the zone as it existed before the migration, not hand-written from the values someone remembers. Export the zone before the cutover, keep that export in the migration branch, and derive the rollback payload from it programmatically. Hand-authored rollback payloads carry transcription errors — a transposed octet, a missing trailing dot on a CNAME target — that are invisible in review and fatal on application, and the moment you discover one is the moment you have the least capacity to debug it.

Configs / Commands

AWS Route 53 — UPSERT records back to the legacy origin:

# Apply the staged legacy values; UPSERT overwrites whatever is live
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456ABCDEFG \
  --change-batch file://dns-rollback.json
# Poll until INSYNC, then verify with dig
aws route53 get-change --id /change/C123456 --query 'ChangeInfo.Status'

Cloudflare API — revert apex and www in two PATCH calls:

# Restore legacy A records at TTL 60 for fast resolver adoption
for rec in "$APEX_REC:203.0.113.10" "$WWW_REC:203.0.113.10"; do
  id="${rec%%:*}"; ip="${rec##*:}"
  curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE/dns_records/$id" \
    -H "Authorization: Bearer $CF_TOKEN" -H "Content-Type: application/json" \
    --data "{\"type\":\"A\",\"content\":\"$ip\",\"ttl\":60}"
done

Verification — confirm the reverted record across resolvers with dig:

# Returned IP should equal the legacy origin on every resolver
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
  echo "$r:"; dig @$r www.example.com A +noall +answer
done

Be deliberate about what “recovery” means before you broadcast it. The natural instinct is to declare recovery when the dashboards return to baseline, but during a DNS reversion the aggregate error rate falls steadily as resolvers convert, so it will look substantially better long before the last resolver has moved. Pick a definition in advance — for example, every sampled resolver returning the legacy address, plus error rate at baseline, plus one successful real transaction — and hold the broadcast until all three are true. Declaring recovery early is worse than declaring it late, because it stands the team down while a minority of users are still failing.

Distinguishing a converged resolver from one that never moved Two resolvers both returning the legacy address, separated by whether the reported TTL decrements between polls, which indicates a genuine re-fetch rather than an entry that never changed. Both return the legacy IP — only one has actually reverted Reading TTL on second poll Means legacy IP, TTL 298 then 291 decrementing re-fetched — revert landed legacy IP, TTL 3600 both times static never adopted the migration new IP still returned decrementing cache not yet expired — wait Poll each resolver twice before drawing any conclusion; a single reading cannot separate the first row from the second.
The middle row is the one that matters during a reversion — it tells you nothing about whether your change propagated.

How long the cheap revert stays available is a calculation rather than a convention; Planning DNS Rollback Windows Around TTL shows how to size it from the record TTL, the negative-caching value and the first write on the new stack.

Validation

Every check below should be run from outside your own network, since an internal resolver may resolve the hostname through a private zone that never participated in the migration at all.

Prove the revert took effect globally and the legacy origin is serving traffic before declaring recovery.

  • dig @1.1.1.1 www.example.com +short returns the legacy IP, and the TTL counts down on repeat queries.
  • curl -sI https://www.example.com | head -1 returns HTTP 200 from the legacy origin.
  • dig +trace www.example.com confirms authoritative delegation and consistent NS records after the revert.
  • Authoritative query volume rises briefly, confirming resolvers are re-querying at the lowered TTL.

One verification subtlety: a resolver that returns the legacy IP is not necessarily one that has adopted your revert. It may simply never have adopted the migration in the first place, and would have reported the legacy address at any point in the last hour. Distinguishing the two matters, because the second tells you nothing about whether your reversion propagated. Check the TTL alongside the address — a resolver that has genuinely re-fetched will report a TTL at or near your published value and count down from there, while one that never moved carries whatever remains of an older entry.

Rollback Triggers

These conditions force or extend the DNS reversion during a cutover.

  • Origin failure: legacy IP must be restored if the new origin returns 5xx above 2% sustained for 5 minutes.
  • Propagation stall: if any major resolver still returns the broken IP after 2x the TTL, force a serial bump and re-push.
  • Resolver TTL override: if ISP resolvers ignore the low TTL, route via CDN to a static legacy origin IP during the wait.
  • Secondary desync: roll back and re-sync immediately if authoritative secondaries report mismatched SOA serials.

FAQ

How long does a DNS rollback take to fully propagate? It takes up to one full TTL cycle from the moment you apply the revert. With a pre-cutover TTL of 60 seconds, most resolvers adopt the legacy record within a few minutes; with a TTL still at 86400 seconds, some clients stay on the broken target for nearly a day. The TTL set before migration is the single biggest factor.

Can I shorten the window by lowering the TTL during the incident? No. Lowering the TTL now only affects future cache entries; resolvers that already cached the record at the old TTL hold it until that value expires. This is precisely why TTL must be lowered before cutover, not in response to the fault.

Should I revert AAAA and CNAME records too, or just A? Revert every record type that changed for the affected hostnames. Leaving a stale AAAA record sends IPv6 clients to the broken origin even after the A record is fixed, producing intermittent failures that are hard to diagnose.

What if the legacy origin has already been partially decommissioned? Then the DNS rollback is not available to you, and knowing that in advance changes the entire risk profile of the migration. This is the strongest practical argument for staged retirement: keep the legacy origin fully serving for at least one full soak window, then stop the application but leave the host and address allocated for considerably longer. A team that released the old IP on cutover day has converted every subsequent problem from a two-minute reversion into a rebuild. Verify the legacy origin actually answers correctly as part of the pre-flight checks, rather than assuming it does because nobody turned it off.

Should the reverted records go back at the original TTL or stay low? Stay low. The reversion is not the end of the incident — you will be retrying the migration, and possibly soon — so restoring an 86400 second TTL immediately after a failed cutover throws away the agility you will need for the second attempt. Apply the legacy values at the same short TTL you were running during the window, and only restore the normal value once the migration has succeeded and cleared its soak.

Do I need to purge anything besides DNS during a DNS rollback? Yes — purge the CDN edge cache for the affected hosts. DNS controls which origin resolvers reach, but a CDN may still serve cached objects generated by the broken origin. Trigger a cache purge via the provider API alongside the DNS revert.

Is it ever right to wait rather than roll back, given the TTL cost? Occasionally, and the deciding question is whether the fault is bounded and understood. If a single route is failing, the cause is identified, and a fix is minutes away, waiting is often faster than a reversion that will itself take a full TTL cycle and then need a second cutover later. If the fault is unbounded — you do not yet know what is failing, or the failure is spreading — reverting immediately is correct even though it is slower to take effect, because the alternative is an unknown fault running for an unknown duration. The trap is that unbounded faults reliably feel bounded in their first ten minutes.

How does DNSSEC change the rollback procedure? It adds a signing step that must not be skipped, and a failure mode that is louder than a stale record. Reverting a signed zone means re-signing it, and a zone whose RRSIG records do not validate against the published DS at the registry produces SERVFAIL from every validating resolver — meaning the rollback makes the site less reachable rather than more. Rehearse the signed reversion specifically, confirm the key material used to re-sign is current, and include a dig +dnssec validation check in the post-rollback verification rather than only checking the address.

What should the holding page on the broken origin actually say? Enough that a user knows the site is not gone and roughly when to return, and nothing that speculates about cause. A short maintenance notice with an approximate return time reduces support volume dramatically compared with a 502, and it prevents the far worse outcome where a user who reaches the broken origin concludes the business has failed. Serve it with a 503 status and a Retry-After header rather than a 200, so crawlers treat it as temporary and do not begin de-indexing the pages behind it — a maintenance page returning 200 is the fastest way to turn a two-hour incident into a ranking problem that outlasts it by weeks.

Related

← Back to Migration Rollback Playbooks

Explore Sub-topics

Reverting DNS Records During a Failed Cutover
Step-by-step reversion of A/AAAA/CNAME records during a failed cutover, with TTL-aware timing and dig verification across global resolvers.
Planning DNS Rollback Windows Around TTL
Size the window in which a DNS cutover can still be reverted cheaply — compute the real revert cost from TTL, negative caching and resolver behaviour.