Reverting DNS Records During a Failed Cutover

Problem Statement

The cutover is live, the new origin is failing, and every minute on the broken target costs traffic and revenue. You need to revert the A, AAAA, and CNAME records to the legacy origin immediately, then confirm the change has reached resolvers worldwide before standing down. This is the execution-level companion to DNS Rollback Procedures, focused purely on the commands and TTL-aware timing of the reversion itself.

TTL-aware DNS reversion timeline Five stages: detect failure, push legacy records, wait one TTL, verify with dig across resolvers, then stand down. TTL-Aware Reversion Timeline 1. Detect 2. Revert 3. Wait TTL 4. Verify 5. Stand down 5xx sustained push legacy A/AAAA 60–300 s dig @resolvers confirm 200 A pre-lowered TTL is what makes step 3 minutes, not hours.
Reversion only propagates as fast as the TTL set before cutover — keep it low until the migration is verified stable.

When to Use This Approach

  • The new origin returns sustained 5xx errors and a redirect-layer swap will not fix an origin fault.
  • DNS was pointed at a new IP during cutover and you have the legacy values staged.
  • Your pre-cutover TTL is low (60–300 s), so reversion can take effect in minutes.
  • Edge or load-balancer health checks cannot fail over fast enough on their own.
  • You need a clean, verifiable return to the known-good origin rather than a forward patch.

Step-by-Step Instructions

1. Apply the Staged Legacy Records

Push the legacy A/AAAA/CNAME values with an idempotent UPSERT so re-running the command is safe under pressure.

# Route 53: UPSERT all changed records back to legacy in one batch
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456ABCDEFG \
  --change-batch file://dns-rollback.json   # contains UPSERT for apex, www, AAAA, CNAME

Use a single batch containing every record that changed, and make sure that batch was generated from a pre-migration export rather than typed during the incident. UPSERT is idempotent by design so re-running is harmless, which matters because the natural response to uncertainty under pressure is to run the command again — but idempotence protects you only from repetition, not from a wrong value applied confidently.

2. Increment the Zone Serial and Confirm Propagation to Secondaries

Bump the SOA serial so authoritative secondaries pull the reverted zone; a record only reaches resolvers once every authoritative server serves it.

# Confirm every authoritative NS reports the new SOA serial
for ns in ns1.provider.com ns2.provider.com; do
  dig @$ns example.com SOA +short   # serial field must match across all nameservers
done

3. Verify the Revert Across Global Resolvers With dig

Query multiple public and regional resolvers; the answer must be the legacy IP everywhere, with a TTL that counts down on repeat queries.

# The returned IP must equal the legacy origin on every resolver
for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
  echo "== $r =="; dig @$r www.example.com A +noall +answer
done

Query each resolver twice, a few seconds apart. A single query returning the legacy address is ambiguous: it may be a resolver that has genuinely re-fetched, or one that never adopted the migration in the first place. The second query resolves the ambiguity, because a resolver that has just re-fetched reports a TTL at or near your published value and visibly decrementing, while one that never moved carries the remainder of an older entry. During a reversion this distinction is the difference between “propagation is working” and “I have no evidence either way”.

4. Confirm the Legacy Origin Is Serving and Purge the Edge

Once resolvers return the legacy IP, confirm the origin answers HTTP 200 and purge any CDN cache built from the broken origin.

# Confirm legacy origin health, then purge edge so no broken objects remain
curl -sI https://www.example.com | head -1   # expect HTTP/2 200
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" \
  -H "Authorization: Bearer $CF_TOKEN" -H "Content-Type: application/json" \
  --data '{"purge_everything":true}'

Sequence matters more than speed across those four commands, because two of them are only meaningful once the one before has landed. Verifying resolvers before the secondaries have pulled the zone produces a confusing mixture of old and new answers that looks like a failed revert; purging the edge before resolvers have moved simply repopulates the cache from the broken origin.

Order of operations for a DNS reversion Four steps in strict order — apply the staged records, confirm the serial on every nameserver, verify resolvers return the legacy address, then purge the edge — each showing what goes wrong if it is run before the previous one has completed. Each step is meaningless until the one before it lands 1. UPSERT legacy idempotent, one batch 2. Serial on every NS query each by name 3. Resolvers agree TTL counting down 4. Purge the edge clears cached 503s run too early: a stale secondary keeps serving the broken IP run too early: mixed answers look like a failed revert run too early: the edge refills from the broken origin Only step 1 is urgent. Steps 2 to 4 are gated on evidence, and running them ahead of that evidence costs time rather than saving it. The instinct under pressure is to fire all four at once — that is the mistake this ordering exists to prevent.
The reversion is one urgent command followed by three checks; treating all four as urgent is what produces the confusing half-reverted state.

Worked Example

At 02:14 UTC, www.example.com was cut over from the legacy origin 203.0.113.10 to a new origin 198.51.100.20. Within four minutes the 5xx rate crossed 6%, breaching the abort threshold. The pre-cutover TTL had been lowered to 60 seconds two days earlier.

Before reversion, dig @1.1.1.1 www.example.com +short returned 198.51.100.20 with curl -sI showing HTTP/2 503. The on-call ran the Route 53 UPSERT at 02:19, the change reached INSYNC in 38 seconds, and the SOA serial advanced from 2026061901 to 2026061902 on both nameservers. By 02:21 all four resolvers returned 203.0.113.10 with a decrementing TTL, and curl -sI https://www.example.com returned HTTP/2 200. A full edge purge cleared the cached 503 responses. Total recovery: 7 minutes from trigger to verified legacy serving — bounded by the 60-second TTL set ahead of time, as covered in TTL Optimization Strategies.

Seven minutes from trigger to verified recovery is a good number, and it is worth being precise about where it came from. Roughly forty seconds of that was the API change reaching INSYNC. Two minutes was resolvers releasing a sixty-second cache entry. The rest was verification and the edge purge. Only the first component is affected by how quickly the on-call engineer moved; the second was fixed two days earlier by whoever lowered the TTL, and the third is the cost of confirming rather than assuming.

Where the seven minutes of recovery time actually went A breakdown of a seven-minute recovery: five minutes detecting the breach, forty seconds for the API change to reach INSYNC, two minutes of resolver cache expiry set by the TTL, and the remainder verification and purge. Only one slice of a recovery is about reacting quickly detecting the breach — 5 min API 40 s TTL expiry — 2 min verify + purge shorten by tightening the trigger window already fast shorten only by lowering the TTL days beforehand do not skip The largest slice is detection, not reaction — which is why the trigger thresholds matter more than the runbook's execution speed. Had the TTL still been 86400 s, the third block alone would have been a day wide.
Two of the four blocks were decided before the cutover began. Optimising the runbook only shortens the two narrow ones.

Verification

  • dig @1.1.1.1 www.example.com +short returns the legacy IP, and the TTL decreases on repeated queries, confirming fresh resolver caching.
  • dig +trace www.example.com shows consistent authoritative delegation and matching NS records after the revert.
  • Track adoption across global points of presence using DNS Propagation Tracking until no resolver returns the broken IP.

FAQ

How quickly will the reverted record take effect? Within one TTL cycle of the moment you apply it. With a 60-second pre-cutover TTL, most resolvers serve the legacy IP within two to three minutes; with a high TTL still in place, some clients remain on the broken origin until that cache expires.

Why must I increment the SOA serial? Authoritative secondaries only pull a zone update when the serial advances. If you revert the record on the primary but the serial does not change, secondaries keep answering with the broken IP, leaving a fraction of traffic stranded until the next scheduled refresh.

Should I tell stakeholders before or after the revert has propagated? Before, and immediately. The first message costs nothing and prevents the most damaging failure mode of a rollback, which is a room full of people forming independent theories while the on-call engineer works. Send one line at the moment the abort is called — what broke, that the reversion is running, when the next update comes — then a second when resolvers have converged. Waiting until you can report success turns a seven-minute recovery into seven minutes of silence, and silence during an incident is read as loss of control regardless of how well the technical work is going.

Do I need to revert AAAA records as well as A records? Yes. If you only fix the A record, IPv6-capable clients following a stale AAAA record still reach the broken origin, producing intermittent failures that look random. Always revert every record type that changed for the affected host.

Should I purge everything, or only the affected paths? During an origin-level reversion a full purge is usually justified, because the broken origin may have served incorrect responses for any path, and you have no reliable list of which. That is different from a redirect rollback, where the affected routes are known and a targeted purge avoids stampeding the origin. Accept the cost: a full purge sends the whole traffic load to the legacy origin cold, so confirm that origin is healthy and warm before firing it rather than after.

What if the reverted records are correct but the site is still down? Then the fault was never in DNS, and the reversion has told you something valuable. Check the legacy origin directly by address, bypassing the hostname entirely — curl --resolve www.example.com:443:203.0.113.10 https://www.example.com/ — to separate a resolution problem from an origin problem. A legacy origin that fails when reached directly means it was degraded before the cutover started or was affected by the same underlying cause, and no amount of DNS work will help. That is the point to escalate rather than continue reverting.

Should the legacy records go back at their original TTL? No — put them back at the short TTL you were running during the cutover window. You are going to retry this migration, quite possibly within days, and restoring a long TTL immediately after a failed attempt throws away the agility the second attempt will need. Keep the low value in force until a cutover has succeeded and cleared its soak, then restore the normal TTL as a deliberate, scheduled step.

Related

← Back to DNS Rollback Procedures