Crawler policy

Access boundary

• Fetch public pages and public machine-readable responses only.
• Honor robots rules, site-owner blocks, access controls, and HTTP rate signals.
• Never solve CAPTCHAs, create accounts, use credentials, or bypass login, age, geographic, or security controls.
• Never submit a notice or website form during research.

Rate behavior

bounded single-domain diagnostics; no form submission; no access-control bypass

Default policy is one active request lane per host, bounded pages, bounded redirects, bounded retries, and backoff on rate limits or failure.

Policy version: rd4.crawler_etiquette.v1

Evidence retained

• Public URLs, response metadata, hashes, and bounded route-context snippets.
• Acquisition and extraction diagnostics needed to explain a route result.
• Provenance and history needed to reconcile new discovery with prior Route Truth.

Retention is minimized to the operational, evidentiary, security, and audit purpose. Correction requests are reviewed against provenance rather than silently rewriting history.

Data excluded

• Private user content, credentials, sessions, and CAPTCHA tokens.
• Account-only pages or private messages.
• Bulk media, media libraries, or content unrelated to public enforcement routing.
• Form submissions, notices, or other state-changing actions.