Crawl Baseline Generation

Context

A crawl baseline is the frozen, unaltered picture of the legacy estate that every post-launch validation is measured against. Webmasters, SEO engineers, and technical project managers capture it in the diagnostic window — after content freeze, before any DNS, CMS, or redirect change — so that “did the migration break anything?” becomes a diff against a known dataset rather than a guess. Without a baseline you cannot prove indexation parity, cannot find orphaned-but-trafficked URLs, and cannot tier redirects by value. This work sits at the front of the Pre-Migration Auditing & Risk Assessment sequence and feeds both the risk matrix and the redirect source map.

The baseline must be complete in two dimensions: every URL a crawler can reach by following links, and every URL real traffic actually requested. Link-following alone misses orphaned pages that still earn sessions; logs alone miss freshly published pages with no inbound traffic yet. You need both, joined.

Render crawl, logs, and sitemap merge into one deduplicated inventory, then join to analytics before export.

A baseline has one property that makes it different from every other artefact in a migration: its value comes entirely from being unchanged. The instinct when the numbers look untidy is to re-crawl with better settings and use the cleaner result, and that instinct destroys the thing you are building. If the first crawl missed a section, fix the settings and re-capture before the freeze, then commit that export and never touch it again. A baseline that has been quietly refreshed halfway through the project cannot answer the only question it exists for.

That is also why the capture window matters. Take it after content freeze, so the estate is not moving underneath you, and before any migration-related change, so nothing in it reflects work in progress. A baseline captured mid-remediation bakes the partially-fixed state into your reference point, and every later comparison silently measures against a site that never really existed.

Pre-flight Checks

Set crawl parameters, authentication, and exclusions before extraction so the baseline is complete and the origin survives the load.

Set the crawler user-agent to match the target bot (Googlebot smartphone) and crawl depth to unlimited.
Enable JavaScript rendering with a 3–5 s render wait; omitting it loses DOM links and metadata on client-rendered frameworks.
Cap concurrency at 5–10 threads with a 1 s delay to avoid origin overload; crawl the origin IP, not the CDN edge, to bypass cached responses.
Exclude faceted-navigation parameters, calendar queries, and session IDs to avoid infinite crawl traps.
Keep staging out of the index with X-Robots-Tag: noindex response headers (more reliable than robots.txt during DB syncs).
Supply authentication for gated conversion paths so member-only revenue funnels are captured.
Gather 30–90 days of raw access logs and the live sitemap.xml to merge with the link crawl.
Allow-list the crawl source address with whoever runs the WAF, and confirm on a small test crawl that the responses match what a browser receives before starting the full pass.

Execution Steps

1. Run the Render-Enabled Crawl

Execute a full crawl with JavaScript rendering against the legacy origin to capture the link-reachable URL set. Set unlimited depth and a 3–5 s render wait so hydrated DOM links and client-side metadata land in the dataset. Crawl the origin directly to avoid edge-cached responses masking real status codes.

Set the crawler’s identity deliberately and coordinate it with whoever runs the WAF. A crawl at ten threads announcing itself as Googlebot is a good way to be rate-limited, challenged, or silently served a different response set than a real user would get — all three of which corrupt the baseline in ways that are hard to spot afterwards, because the crawl completes and produces plausible numbers. Allow-list the crawl source address, confirm with a small test crawl that responses match what a browser receives, and only then run the full pass.

Watch the origin while it runs. A baseline crawl is often the heaviest sustained load the legacy estate has seen in months, and an origin that begins shedding requests under it will return 5xx codes that get recorded as though they were the site’s normal state. If error rates climb during the crawl, reduce concurrency and re-run rather than accepting the export — a baseline containing several hundred spurious 5xx entries will generate exactly that many false findings later.

2. Merge Logs and Sitemap for Completeness

Extend the link crawl with URLs that real traffic requested but no internal link exposes. Parse 30–90 days of access logs and the live sitemap, then union them with the crawl output and deduplicate. Anything present in logs but absent from the crawl is an orphaned candidate that needs an explicit redirect decision.

Normalise before deduplicating, or the union will not do what you expect. Logs record raw request paths, sitemaps record absolute URLs, and the crawler records whatever it resolved to — so the same page arrives in three formats, with and without a trailing slash, with mixed case, and carrying tracking parameters. A naive sort -u across the three treats each variant as a distinct URL and inflates the inventory substantially. Decide the canonical form first — protocol, host, case, trailing slash, which parameters are significant — apply it to all three sources, and record the rules alongside the export so a later reader can reproduce them.

3. Export the Full Dataset

Serialise the inventory with every field downstream work needs: status code, canonical, redirect target, meta robots, and indexability. Follow How to Export Full Crawl Data Before Migration for schema-compliant field mapping so the export drops straight into redirect tooling.

Store the export in a format that survives contact with other tools. CSV is the pragmatic default, but pin the encoding to UTF-8 and quote every field, because URLs routinely contain commas, non-ASCII characters, and — on international sites — percent-encoded sequences that a spreadsheet will helpfully mangle on open. The failure this avoids is subtle and common: an export that looks fine, is opened once in a spreadsheet application, is saved, and now has several thousand URLs silently altered. Treat the committed export as read-only and do analysis on copies.

4. Join to Analytics and Tier by Value

Attach business value to each URL so later prioritisation is objective. Join the inventory to GA4 sessions and conversions and to Search Console clicks, then segment by content type. Apply Traffic & Conversion Mapping to flag revenue-critical paths that must migrate as single-hop 301s.

Capture the fields you will need for comparison, not just the fields that are interesting now. The export should carry status code, final resolved URL, canonical, meta robots, X-Robots-Tag, title, h1, word count, and internal inbound link count for every URL — because the post-launch question is rarely “does this page exist” and usually “is this page the same page”. Word count and inbound link count in particular are cheap to record and are the two fields that most reliably reveal a template that migrated structurally but lost half its content.

5. Validate Integrity Against Logs

Confirm the baseline is trustworthy before it becomes the reference for every gate. Cross-check the export against raw logs for coverage gaps, recompute canonical-consistency and indexation scores, and circulate the discrepancy report through Stakeholder Communication Plans for sign-off.

Each of the five steps adds a distinct slice of coverage, and the reason for running all five is that no single source sees the whole estate.

The blind spots do not overlap, which is the entire argument for the merge step — each source covers precisely what another one cannot see.

Configs / Commands

The fragments below run in sequence and each writes an artefact the next one consumes, so the whole baseline is reproducible from a clean checkout. Commit the intermediate files as well as the final inventory — when a URL’s presence is questioned months later, being able to point at the specific source that contributed it is worth far more than the disk space.

# Screaming Frog CLI — unlimited depth crawl with JS rendering, full export
# Requires the Screaming Frog SEO Spider (headless mode on Linux)
java -jar ScreamingFrogSEOSpider.jar \
  --crawl https://production-origin.com \
  --headless --save-crawl \
  --export-tabs "Internal:All,Response Codes:All,Canonicals:All" \
  --output-folder /tmp/baseline/ \
  --config crawl.seospiderconfig   # render mode = Ajax, 5s wait, JS enabled

# Merge crawl URLs with log + sitemap URLs, then dedupe to one inventory
awk '{print $7}' access.log | sort -u > log_urls.txt        # paths from logs
grep -oP '(?<=<loc>)[^<]+' sitemap.xml | sort -u > sm_urls.txt
sort -u crawl_urls.txt log_urls.txt sm_urls.txt > inventory.txt
wc -l inventory.txt   # frozen baseline count

# cURL validation loop — record status code for every URL in the inventory
while IFS= read -r url; do
  code=$(curl -sI -o /dev/null -w '%{http_code}' "$url")   # headers only
  echo "$code $url"
done < inventory.txt > status_baseline.csv

# dig — capture the SOA TTL now so rollback can restore the exact value later
dig production-origin.com SOA +noall +answer
# Then lower the record TTL to 300 s exactly 48 h before cutover via the DNS API

Circulate the discrepancy report rather than filing it. The people who can explain an anomaly are usually not the people who found it — a content editor knows why four hundred URLs under one path were deliberately retired last year, a developer knows why a section returns 403 to anything without a session, and neither will volunteer that information unless asked. Publishing the list of unexplained findings and asking for explanations converts a large pile of ambiguous data into a small list of genuine problems, usually in a single afternoon.

Validation

Concrete pass/fail checks the baseline must clear before it is accepted.

Baseline URL count is within 5% of sitemap and log totals: grep -c '<loc>' sitemap.xml vs wc -l inventory.txt
No production path returns X-Robots-Tag: noindex: grep -i 'noindex' status_baseline_headers.txt returns nothing
Every 4xx/5xx in status_baseline.csv is explained (intentionally retired or flagged for fix)
Orphaned URLs (in logs, not in crawl) each have a recorded redirect decision
Render crawl captured client-side links — spot-check a JS-only page in the export
Export validated against raw logs for high-traffic paths with awk/pandas

The 5% deviation tolerance in that checklist is not arbitrary, and it is worth understanding what a breach of it actually indicates. Small differences between the three counts are normal: sitemaps lag publication, logs include URLs retired during the window, and crawls exclude parameters by design. A gap larger than a few per cent means one source is systematically missing a category rather than drifting at the edges — most commonly a whole section behind a render boundary, an authentication wall, or a robots.txt rule nobody remembered.

Compare all three counts as a diagnostic before accepting the baseline; a single number in isolation cannot tell you whether it is complete.

Rollback Triggers

Each condition below is a no-go at the gate rather than a problem to note and proceed past.

Halt the cutover and revert if any condition appears during pre-launch validation.

Baseline URL count deviates >5% from sitemap or 30-day log totals.
Any top-quintile-revenue URL is missing a single-hop 301 mapping.
Canonical discrepancy rate exceeds 2% across migrated templates.
Authentication walls unexpectedly block crawler access to revenue paths.
DNS propagation fails or CDN cache-purge schedules are unverified at the gate.

FAQ

How do I handle JavaScript-rendered URLs during baseline generation? Use a render-capable crawler (Screaming Frog with Ajax rendering, or Puppeteer/Playwright) and enforce a 3–5 s render wait so the DOM fully hydrates before link and metadata extraction; otherwise client-side routes never enter the inventory.

What crawl depth and thread count suit enterprise sites? Set depth to unlimited and run 5–10 concurrent threads with a 1 s delay. That keeps deep archives and paginated series in scope while staying under the origin’s overload threshold.

How do I validate baseline accuracy against server logs? Join the export to 30-day Nginx/Apache logs with awk or pandas, filter for 200/301 responses, and surface URLs that appear in logs but not the crawl — those orphaned, traffic-earning paths are the highest-risk omissions.

When should DNS TTL be lowered relative to baseline completion? Lower the authoritative TTL to 300 s exactly 48 h before cutover, and only after baseline validation confirms redirect mappings, canonical tags, and CMS config are production-ready.

How large a log window is actually needed? Thirty days is the practical minimum and ninety is materially better, for one specific reason: seasonality. A thirty-day window captured in a quiet month will miss every URL that only earns traffic around a seasonal peak — the gift-guide pages, the tax-year content, the event landing pages — and those are frequently high-value paths whose absence from the baseline is invisible until the season comes round again on a site that no longer serves them. If ninety days of raw logs are not retained, supplement with a longer Search Console export, which reaches back sixteen months at lower resolution.

Should the baseline crawl include the CDN edge as well as the origin? Run the origin crawl as the baseline and a smaller edge crawl as a separate comparison. They answer different questions: the origin tells you what the site is, the edge tells you what visitors currently receive, and a disagreement between them is itself a finding — a stale cached page, a rule rewriting status codes, an origin response the edge never serves. Do not merge the two into one inventory, because you will lose the ability to tell which layer any given anomaly came from.

What should happen to URLs in the logs that are obviously junk? Classify them rather than deleting them. Vulnerability scans, malformed paths, and requests for files that never existed on this site are noise, but the classification itself is worth recording — partly so the orphan list stays readable, and partly because a surprising volume of traffic to a nonsense path occasionally turns out to be a real integration calling a URL nobody documented. Filter them into a separate bucket, eyeball the top entries by volume, and only then discard.

How do I baseline a site that is too large to crawl exhaustively? Stratify rather than sample uniformly. Crawl every URL in the top value tiers exhaustively — those are the ones whose individual behaviour you will need to verify — and take a proportional sample within each remaining template class, sized so that a systematic template fault would show up. The log inventory should still be complete, because parsing logs costs nothing per URL and it is the log side that carries the orphan discovery. Record the sampling strategy in the manifest, since a later reader comparing counts needs to know that the crawl was deliberately partial rather than broken.

Can the baseline be reused for the next migration? Not as a baseline, but very much as a starting point. The frozen export belongs to the migration that produced it and should stay untouched, yet the tooling around it — the crawl configuration, the normalisation rules, the join scripts, the validation checks — is the genuinely expensive part and transfers directly. Teams that treat the baseline as a one-off spreadsheet rebuild all of it each time; teams that treat it as a small repository find the second migration’s audit takes a fraction of the first one’s.

← Back to Pre-Migration Auditing & Risk Assessment