Scheduling Recurring Crawls to Detect Pre-Migration Drift

Problem Statement

A single pre-migration crawl is a photograph, but a site is a film: editors publish, developers ship, and parameters multiply during the weeks before cutover. If your redirect map is built against a crawl from three weeks ago, every URL added since is an unmapped 404 waiting to happen, and every page that quietly turned into a redirect or changed its canonical is a mismatch your map does not know about. You need recurring, scheduled crawls that diff against the last run so drift surfaces while there is still time to remap, rather than discovering it from a spike in 404s on launch night. The goal is not to crawl more, but to crawl on a cadence and compare automatically so a human only looks when something actually moved. This page is part of Crawl Baseline Generation; start there for the initial inventory.

Scheduled crawl diff loop A cron trigger runs a headless crawl, which exports a dated snapshot that is diffed against the previous snapshot, raising an alert on drift. Scheduled Crawl Drift Loop 1. Cron fires 2. Crawl 3. Diff 4. Alert Nightly 02:00 Dated CSV vs last run New/404/redirect Loop nightly until the freeze window opens
Each scheduled run exports a dated snapshot and diffs it against the prior one; drift raises an alert and the loop repeats until freeze.

When to Use This Approach

  • The site changes frequently (active editorial, e-commerce catalogue, or rapid release cadence) in the run-up to migration.
  • The migration freeze window is weeks away and the URL set is still moving.
  • You want to detect new URLs, status-code changes, and canonical changes automatically rather than re-auditing by hand.
  • You have a host that can run a headless crawler on a schedule (a build agent, a small VM, or a cron box).
  • You need a dated trail of snapshots to show stakeholders exactly when a URL appeared or broke.

Step-by-Step Instructions

1. Make the Crawl Fully Headless and Repeatable

A scheduled crawl cannot prompt for input, so run the Screaming Frog CLI in headless mode with a saved config and a dated output folder. Pinning the config guarantees every run is comparable.

# Screaming Frog CLI — headless nightly crawl into a dated folder
STAMP=$(date +%Y-%m-%d)
ScreamingFrogSEOSpider \
  --crawl https://oldshop.example.com \
  --headless \
  --save-crawl \
  --export-tabs "Internal:All" \
  --output-folder "/var/crawls/$STAMP" \
  --config /etc/crawls/pre_migration.seospiderconfig   # pinned config = comparable runs

2. Schedule It with cron

Wrap the crawl in a small script and schedule it nightly so drift is caught within 24 hours. Run it at a low-traffic hour and add a modest crawl delay in the config to avoid stressing the origin. Have the wrapper script set the date stamp, create the output folder, run the crawl, and then call the diff, so the entire pipeline is a single cron entry with one log file to watch. Pin the timezone explicitly in the cron environment, because a host that drifts between UTC and local time will run your “02:00” crawl in the middle of peak traffic after a daylight-saving change.

# /etc/cron.d/pre-migration-crawl — nightly at 02:00, log to a rotating file
0 2 * * * deploy /usr/local/bin/nightly_crawl.sh >> /var/log/crawl.log 2>&1

3. Diff the Latest Snapshot Against the Previous One

Compare the two most recent exports to surface added URLs, removed URLs, and status-code flips. A simple key of URL plus status code makes new and broken pages fall out of a set difference, and joining the two frames on URL exposes pages that changed status without changing address — the silent 200-to-301 flips that break redirect maps. Keep the diff dumb and deterministic so its output is trustworthy; resist the urge to “smartly” suppress changes, because the whole point is that a human reviews every real delta.

# Diff today's crawl against yesterday's on URL + status code
import pandas as pd, sys
prev, curr = sys.argv[1], sys.argv[2]
a = pd.read_csv(prev, usecols=['Address', 'Status Code'])
b = pd.read_csv(curr, usecols=['Address', 'Status Code'])
new_urls   = set(b.Address) - set(a.Address)               # appeared since last run
gone_urls  = set(a.Address) - set(b.Address)               # disappeared since last run
merged = a.merge(b, on='Address', suffixes=('_old', '_new'))
flips = merged[merged['Status Code_old'] != merged['Status Code_new']]  # status drift
print(f"new={len(new_urls)} gone={len(gone_urls)} status_changes={len(flips)}")

4. Alert on Drift Above a Threshold

Noise-free alerting means a non-zero drift count writes a report and exits non-zero so cron mail (or a CI step) flags it. Keep the threshold low during freeze so even small changes are reviewed, but allow a small tolerance early in the run-up when the site is still legitimately changing, so the team is not desensitised by nightly alerts that are all expected. The exit code is the integration point: cron mails on any non-zero exit, and a CI runner marks the job red, so the same script drives both a person and a pipeline without extra plumbing. Archive each diff report next to its dated crawl folder so the history of what changed, and when, is permanent.

# Fail the job when drift exceeds threshold so the run is flagged
DRIFT=$(python diff_crawl.py "$PREV" "$CURR" | grep -oP 'status_changes=\K\d+')
if [ "$DRIFT" -gt 0 ]; then
  echo "Drift detected: $DRIFT status changes — review /var/crawls" >&2
  exit 1   # non-zero exit triggers cron mail / CI failure
fi

A recurring crawl is worth setting up because the estate you audited is not the estate you will migrate. Editorial teams publish, catalogues rotate, campaigns land, and a URL set captured six weeks before the freeze will differ from the one that actually ships — usually in the sections that change most, which are frequently the ones carrying the most traffic. The scheduled crawl converts that drift from something you discover at the freeze into something you watch accumulate.

Kinds of drift a nightly crawl detects, and what each implies Five categories of change between consecutive snapshots — new URLs, removed URLs, status changes, canonical changes and redirect changes — with the typical cause of each and whether it should alert. Not all drift deserves an alert Change between snapshots Usually means Alert? New URLs appear normal publishing only above a threshold URLs disappear unplanned deletion or a crawl fault always 200 becomes 404 or 5xx a real regression on the live site always Canonical target changes a CMS or template edit nobody flagged always Redirect target changes someone editing rules ahead of the freeze always
Only the first row is expected. The other four describe a site changing underneath a migration that has already been planned against a different shape.

Worked Example

A publisher on oldnews.example.com plans to migrate to news.example.com in five weeks. Nightly Screaming Frog runs land in /var/crawls/2026-06-12 through /var/crawls/2026-06-19. On the night of the 17th the diff reports new=42 gone=3 status_changes=7: an editor relaunched a tag taxonomy, creating 42 new /topics/ URLs and turning 7 old tag pages into 301s.

Because the alert fired, those 42 URLs are added to the redirect map before freeze instead of becoming 404s on launch day, and the 7 status flips are traced to old tag pages that now 301 internally — chains that would have stacked on top of the migration redirects had they gone unnoticed. Two nights later the diff reports status_changes=0, confirming the taxonomy has settled and the map is back in sync.

The dated folders give the team an exact audit trail: they can point to /var/crawls/2026-06-17 as the night the taxonomy changed and show stakeholders that the drift was caught and remapped within 24 hours. The dated snapshots also feed the Core Web Vitals baseline work and the broader Pre-Migration Auditing & Risk Assessment record, giving a precise timeline of when the taxonomy changed.

Verification

Confirm the schedule actually runs and the diff catches known changes.

# Confirm cron executed and produced a dated folder in the last 24h
find /var/crawls -maxdepth 1 -type d -mtime -1

# Sanity-check the diff by planting a known new URL and re-running
python diff_crawl.py /var/crawls/2026-06-18/internal_all.csv \
                     /var/crawls/2026-06-19/internal_all.csv

# Verify the alert exits non-zero on drift
echo "exit code: $?"

Watch for these failures: a non-headless config that silently hangs waiting for a dialog; comparing folders out of order so “new” and “gone” are swapped; and crawling without a delay and tripping the origin’s rate limiting, which injects phantom 429 drift. Watch too for a crawl that quietly truncates — if a run hits a page cap or times out, it will report a flood of “gone” URLs that are not really gone, so assert a minimum expected URL count before trusting the diff. Finally, make sure the cron user has write access to the output and log paths, or the job will fail silently and you will believe the site is stable when nothing has actually run.

Threshold the alert on proportion rather than count, and suppress the known-noisy sections. A site publishing forty articles a day will generate forty new URLs every night, and an alert that fires on that is an alert somebody will mute within a week — after which the genuinely alarming night passes unnoticed. Express the rule as a share of the total inventory, exclude the paths you expect to churn, and reserve the paging tier for the categories that should never change at all.

Snapshot cadence tightening as the freeze approaches Crawl frequency increasing from weekly at eight weeks out, to nightly from three weeks out, to twice daily in the final week before the freeze, with the alert threshold tightening at each step. Crawl more often as the cost of surprise rises T−8 to T−4 weeks weekly crawl alert above 2% drift T−3 to T−1 weeks nightly crawl alert above 1% drift Freeze week twice daily any change alerts During the freeze the expected drift is zero, so the threshold can be zero — and a single unexpected change is worth waking somebody for, because it means the freeze is not actually holding. Run the crawl off-peak and cap concurrency: a nightly crawl that degrades the origin creates the very regressions it is watching for.
The final tier is the most valuable one — it is the only mechanism that reliably detects a content freeze being quietly broken.

FAQ

How often should the recurring crawl run before migration? Nightly is the practical default during the final month, dropping to weekly when the site is stable and freeze is distant. The cadence should be short enough that drift is caught within one editorial cycle so there is always time to remap before cutover.

Will nightly crawling overload the production origin? Not if you add a crawl delay and schedule for a low-traffic window. Set a delay of one to two seconds in the saved config and run at, say, 02:00 local time; the load is comparable to a single extra visitor and far lighter than search-engine crawling.

Can I run this without the paid Screaming Frog licence? Yes, the headless CLI and tab exports require a licence, but you can substitute an open-source crawler that emits CSV and feed the same diff script. The diff logic only needs a URL column and a status-code column, so any crawler that produces those will work.

Where should the snapshots be stored, and for how long? In the migration repository, dated, and kept until well past launch. Each snapshot is small once compressed, and their value is cumulative: a dated series answers “when did this URL first return 404?” instantly, which is a question that otherwise consumes an afternoon of log archaeology. Keep them through the soak window at minimum, since the most common use is post-launch, when somebody asks whether a page that is missing now was present before the migration.

Should the recurring crawl replace the frozen baseline? No — they do different jobs and conflating them destroys the baseline’s usefulness. The frozen baseline is a fixed reference point captured once at the freeze; the recurring snapshots are a moving record of how the estate got there. Keep both, and be explicit in the repository about which file is which, because the single most damaging mistake here is a nightly job that overwrites the baseline with the current state and leaves you comparing the new site against itself.

What if the site is behind authentication or a staging password? Supply credentials to the scheduled crawl the same way you would for the manual one, and store them as secrets rather than in the crawl configuration. The thing to watch for is a credential expiring mid-project: the crawl continues running, receives 401s for every URL, and produces a snapshot showing the entire site has vanished. Guard against it by alerting on an implausible drop as a separate rule from ordinary drift, and by treating a snapshot whose success rate collapses as a tooling failure rather than a site event.

Related

← Back to Crawl Baseline Generation