Rollback Trigger Thresholds

Context

A migration aborts on data, not on nerves. The hardest decision during a cutover is whether a spike in errors is transient noise or a genuine regression, and that decision must be settled in advance with numeric ceilings and fixed observation windows. This guide defines the trigger set — error rate, latency, traffic, conversion, and crawl errors — that on-call staff watch during the soak window. It sits under Migration Rollback Playbooks; the playbook owns the reversal mechanics, while this page owns the firing conditions.

Each metric has its own ceiling and window; only a breach sustained across the full window fires the rollback.

Thresholds fail in two directions, and both are expensive. Set them too loose and the migration runs on through a genuine regression, because nothing ever formally breached and each individual reading looked survivable. Set them too tight and the first cache warm-up trips an abort, the team loses confidence in the alerting, and by the third false positive somebody has silenced the pager — which leaves you with no thresholds at all, only the appearance of them. The calibration work below exists to land between those failure modes, and it is why every threshold is expressed relative to a measured baseline rather than as a round number somebody liked.

There is a further reason to write them down early. Thresholds authored before the cutover are engineering decisions; thresholds proposed during one are negotiations, and they are negotiated by people who have just spent three months building the thing under test. The number that seems obviously too lenient on Tuesday is the number that seems perfectly reasonable at 02:00 on Saturday with a migration half-finished.

Pre-flight Checks

Establish baselines and alerting before launch so thresholds compare against real pre-migration behaviour, not guesses.

Capture 7-day baselines for 5xx rate, p95 latency, sessions, and conversion from your APM and analytics.
Confirm monitoring scrapes both edge and origin so a CDN-masked origin fault is still visible.
Verify alert routing reaches the named rollback owner via pager, not just a dashboard.
Validate clock sync across log sources so windowed evaluations align.
Agree which planned events fall inside the soak window — batch imports, marketing sends, scheduled reports — and annotate them in advance so their effects are not mistaken for migration faults.

Threshold Readiness Checklist:

Numeric ceiling and window documented for every trigger metric
Baselines stored and accessible to on-call during the window
Synthetic checks (curl) running against critical paths every 60 s
Crawl-error monitoring wired to Search Console or a scheduled crawler

Execution Steps

Set each threshold relative to baseline, with a window long enough to filter noise but short enough to bound damage.

1. Set the Error-Rate Ceiling

Define the 5xx and 4xx percentage that fires a rollback over a fixed window — a common default is 5xx above 2% sustained for 5 minutes. Separate the 4xx ceiling, which signals broken redirects rather than origin failure. The full derivation, alerting rules, and edge cases live in Defining Error-Rate Thresholds That Trigger Rollback.

Split the ceiling by route class rather than applying one number to the whole site. A 2% error rate spread evenly across static pages is a different situation from 2% concentrated entirely on checkout, and an aggregate figure hides exactly the case you most want to catch. At minimum, evaluate the ceiling separately for your revenue path, your authenticated area, and everything else, so a small volume of catastrophic failures cannot be averaged away by a large volume of healthy static traffic.

Measure the rate at the edge and at the origin, and alert on both. An origin returning 5xx while the CDN serves cached copies produces a healthy-looking edge metric and a deteriorating one underneath, and the moment the cache expires the failure arrives all at once. Conversely, an edge-only fault — a bad rule, an expired certificate at the edge — never touches origin metrics at all.

2. Set the Latency Ceiling

Pin a p95 latency ceiling — typically baseline plus 50% or a hard 1500 ms — over a 5-minute window. Latency regressions often precede error spikes as origin connection pools saturate. Watch p95 and p99 separately; a p99 blowout with a stable p95 usually points to a single slow path, not a systemic fault.

Latency deserves a second ceiling on the origin’s connection pool or queue depth, because that metric moves before the user-visible one does. By the time p95 crosses 1500 ms the pool has usually been saturated for a while, and the interval between “the pool is full” and “requests are failing” is precisely the warning you want. If your APM exposes queue depth or active connections, put a ceiling on it and treat a breach as an early escalation rather than an abort.

3. Set Traffic and Conversion Drop Triggers

Define a traffic-drop ceiling, such as organic sessions falling more than 20% versus baseline over 15 minutes, and a conversion-drop ceiling such as 15% over 30 minutes. These catch silent failures where pages return 200 but render broken. Cross-reference high-value paths from Traffic & Conversion Mapping so revenue-critical routes weight the decision.

Compare against the same hour of the previous week rather than against the previous hour. Traffic and conversion both have strong daily and weekly shapes, so an hour-on-hour comparison during a cutover scheduled at a natural trough will report a dramatic drop that is entirely seasonal. Week-on-week comparison removes that, and it also removes the temptation to cut over at an hour chosen to make the numbers look calm.

Be alert to the failure mode where the metric itself breaks. A migration that drops the analytics snippet, changes the domain in a tracking configuration, or trips consent handling produces an instant and total collapse in reported conversions while the site works perfectly. Confirm the tag is firing on the new origin as part of the pre-flight parity check, and when a conversion trigger breaches, verify the tag before treating the number as real.

4. Set the Crawl-Error Spike Trigger

Define a crawl-error multiplier — for example, more than 3x the baseline crawl errors within 60 minutes — to catch redirect and indexability regressions search engines see before users do. Feed this from Search Console Handover coverage reports. A crawl-error spike alone rarely justifies an immediate abort, but combined with a 4xx rise it confirms a routing fault.

Crawl errors are the one trigger that reports on damage you cannot see in your own traffic. Search engines fetch paths users have stopped visiting, follow links you have forgotten, and probe URL patterns that predate the current site — so a redirect rule that quietly collapses an entire legacy path segment often shows up in coverage reports days before it shows up anywhere else. Treat the crawl signal as an early-warning channel rather than an abort condition: it is slow, it is sampled, and it corroborates rather than decides.

5. Define Composite and Time-Boxed Rules

Combine signals so two correlated breaches escalate faster than one in isolation, and define a hard time box — keep all triggers armed for the full 72-hour soak. Document which single triggers auto-fire versus which require the owner’s confirmation, referencing the reversal steps in Migration Rollback Playbooks.

Laid out side by side, the five triggers form a deliberate ladder: fast, unambiguous signals fire on their own, while slower and noisier ones need corroboration before they cost you a migration.

The two automatic triggers are the ones with no benign explanation. Everything below them can be caused by a broken analytics tag, so they escalate rather than abort.

Composite rules are where most of the value sits, because correlated breaches are far more informative than isolated ones. A latency rise on its own may be a cold cache; a latency rise together with a session drop and a conversion drop is the same fault observed three ways, and it should escalate immediately rather than waiting for each window to complete independently. Write at least one composite rule that shortens the window when two signals breach together, and be explicit about which combinations bypass owner confirmation entirely.

Configs / Commands

Prometheus alert rules — error-rate and latency ceilings:

# Fire when 5xx ratio exceeds 2% sustained for 5 minutes
groups:
  - name: migration-rollback
    rules:
      - alert: High5xxRate
        expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
        for: 5m
      - alert: HighP95Latency
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 1.5
        for: 5m

Synthetic check — windowed status and latency sampling with curl:

# Sample status code and total time every 60s; log breaches for the on-call owner
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://www.example.com/checkout
  sleep 60
done | awk '$1 ~ /^5/ || $2 > 1.5 {print strftime(), "BREACH", $0}'

The synthetic check above is deliberately dumb — a fixed interval, a single path, one comparison — and that is a feature. During a cutover the monitoring you trust most should be the monitoring with the fewest dependencies, because a sophisticated pipeline that aggregates through the same infrastructure you are migrating can fail in the same way and at the same moment. Run at least one probe from outside your own network, hitting the real public URL, writing to a log nobody else touches. When the dashboards disagree with it, believe the probe.

Latency needs a more careful definition than error rate does, because it has to name a percentile, a measurement point, a comparison baseline and a sustained duration — Setting Latency Ceilings That Trigger Rollback works through all four.

Validation

Confirm each trigger fires on synthetic faults before relying on it during a live cutover. An untested alert rule is a hypothesis about your monitoring stack, and cutover night is a poor time to test it — a typo in a Prometheus expression produces silence, which is indistinguishable from health.

Inject a controlled 5xx burst in staging and confirm the High5xxRate alert fires within its window.
Throttle a backend and confirm the p95 latency alert pages the owner, not just a dashboard.
Drop a test path to 404 and confirm the crawl-error and 4xx signals rise as expected.
Verify dashboards show the metric against its baseline line, so the on-call call is unambiguous.

The window is the part most often set carelessly, and it is doing precise work: it is the knob that trades false positives against damage. A window too short converts every deploy blip into an abort; a window too long is a decision to let a real fault run for that duration before anyone acts.

The threshold decides what counts as bad; the window decides what counts as real. Tuning one without the other produces either false aborts or slow ones.

Rollback Triggers

Fire a rollback when any of these numeric ceilings is breached for its full window during the soak.

Error rate: 5xx above 2% sustained 5 minutes, or 4xx above baseline + 10 points sustained 10 minutes.
Latency: p95 above 1500 ms (or baseline + 50%) sustained 5 minutes.
Traffic: organic/total sessions down more than 20% versus baseline over 15 minutes with no external cause.
Conversion: key conversion rate down more than 15% over 30 minutes.
Crawl errors: crawl/indexing errors exceeding 3x baseline within 60 minutes, confirmed alongside a 4xx rise.

FAQ

Why attach a time window to every threshold instead of firing instantly? A single 5-second spike during a cache warm-up is noise; a 5-minute sustained breach is a regression. Windows filter transient blips so you do not roll back a healthy migration over a momentary anomaly, while still bounding how long a real fault runs.

Should any single metric auto-trigger a rollback without human confirmation? A severe error-rate breach (for example 5xx above 5% for 2 minutes) is a reasonable auto-trigger because it is unambiguous. Softer signals like conversion drop should require the named owner to confirm, since they have more benign causes such as analytics tagging breaking during the move.

How do I set thresholds for a site with no clean baseline? Use a representative staging load test plus industry-standard absolutes — 5xx above 2%, p95 above 1500 ms — as a floor, then tighten after the first stable week. Absolute ceilings protect you even when relative baselines are missing.

What if a threshold breaches for a reason unrelated to the migration? Roll back anyway, then investigate. This feels wrong and is almost always correct: during a soak window the migration is the largest recent change, and the cost of reverting a healthy migration is a repeated cutover, while the cost of attributing a real fault to an unrelated cause is an outage that runs until someone changes their mind. The exception is a cause you can positively confirm in seconds and that is visibly external — a cloud provider status page already reporting a regional incident, an upstream API returning errors to everyone. “It might be the marketing email that just went out” is a hypothesis, not a confirmation.

How do I avoid alert fatigue during a 72-hour soak? Route only the automatic triggers to a pager and everything else to a channel the owner is actively watching. The soak window is long enough that a stream of borderline notifications will train people to ignore it, which defeats the exercise. It also helps to suppress alerts that fire during known planned events — a scheduled batch import, a marketing send — by annotating those windows in advance rather than by loosening the thresholds that cover them.

Do these thresholds change as the soak window progresses? Yes. Keep them tight for the first few hours when regressions are most likely, then you may relax traffic/conversion windows once daily patterns are confirmed. Error-rate and latency ceilings should stay strict for the full 72 hours.

Should the thresholds differ for a rehearsal versus the real cutover? Keep the numbers identical and the response different. Using looser thresholds in staging defeats the point of the rehearsal, which is partly to discover whether your production ceilings are achievable at all — a staging run that trips the p95 ceiling on every attempt is telling you something important about the new origin before you have risked any real traffic. What should differ is what happens on a breach: in rehearsal, a breach is a finding to investigate; in production, it is an abort.

How many triggers is too many? Five to seven is a workable range, and the constraint is human rather than technical. Every trigger is something the on-call owner has to understand well enough to act on at speed, and a list of fifteen becomes a document nobody has internalised. Prefer a small number of well-calibrated triggers covering distinct failure classes — origin health, latency, user volume, revenue, crawlability — over exhaustive coverage of every metric you happen to collect. Anything that does not have a defined response is a dashboard panel, not a trigger.

Who should be watching these during a 72-hour soak, and how? One named owner at any given moment, with an explicit handover between shifts that transfers the baselines, the current readings, and any breaches that have occurred but not fired. The handover is the weak point: most missed regressions in a long soak happen in the hour after a shift change, when the incoming engineer has the dashboards but not the context that a metric has been drifting upward all night. Write the handover as three lines — what has moved, what has not, what would make you call it — and require it in the channel rather than in conversation, so the next person after that inherits it too.

← Back to Migration Rollback Playbooks