What a web crawler does
A web crawler is a program that finds pages by following links. It starts from one or more seed URLs, fetches each page, pulls out the links it contains and queues the new ones. It repeats that loop until the queue is empty or it reaches a limit.
Search engines crawl to find pages to index. Site audit tools crawl one site to map its pages and report problems such as broken links. Both use the same loop and differ mainly in scope and limits.
Fetch, parse, queue: the crawl loop
Fetching means sending an HTTP GET request and recording the response: status code, headers, response time and body. When the response is a redirect, the crawler reads the Location header and treats the target as another URL to fetch.
Parsing extracts links from HTML, mainly the href of each a element. Relative links are resolved against the page's URL and then normalized, for example by dropping the fragment after # and lowercasing the host. That way two spellings of one address count once.
The queue of URLs found but not yet fetched is called the frontier. A breadth-first crawl fetches every page one link away from the seed before any page two links away. The crawler also keeps a set of URLs it has seen, so it never fetches the same one twice.
Some sites generate endless URLs, such as a calendar with a next-month link or filters that combine into new query strings. Limits keep a crawler from following them forever.
Depth, page and scope limits
Every crawler bounds its work. The depth limit sets how many links away from the seed it will go. The page limit caps the total number of URLs fetched, and the scope keeps it on one host, domain or folder.
Timeouts and response size caps stop one slow or huge page from stalling the crawl.
Limits shape the report. A clean result covers only the pages the crawler reached, and anything past the depth or page limit was never checked. URL Intelligence Platform's crawler, for example, stops at 100 pages and 5 levels deep. On a larger site, its result is a sample of the pages nearest the starting URL.
robots.txt and polite crawling
Before crawling a host, a well-behaved crawler fetches /robots.txt and skips the paths it disallows for that crawler's user agent. The rules are standardized in RFC 9309. Following them is voluntary, so robots.txt asks crawlers to stay away but can't stop anyone.
Politeness keeps a crawler from overloading a site. That means few requests at a time to one host, a pause between requests, and backing off after 429 or 503 responses. A polite crawler also names itself in the User-Agent header so site owners can identify it.
Search engines adjust how much they crawl to how quickly and reliably a server responds. Google's documentation says crawl budget is mainly a concern for large sites and for sites whose pages change daily.
JavaScript and what a crawler sees
A basic crawler reads only the HTML the server returns. If your navigation or content is added by JavaScript after the page loads, that crawler won't see those links.
A rendering crawler loads each page in a headless browser and runs its scripts first. That finds more links but takes far more time and memory, so some audit crawlers render only on request or not at all. Google does render pages, though rendering can happen after the first crawl.
Google recommends making links a elements with an href attribute. Links that exist only as JavaScript click handlers may never be followed.
Reading a crawl report
A crawl report records what the crawler could reach from its seeds within its limits. These are the findings to act on.
- Broken pages are internal URLs that returned 4xx or 5xx. A good report lists the pages that link to each one.
- Redirected links point at a URL that redirects. Update each link to the final address.
- Click depth is how many links separate a page from the seed. Important pages buried deep are hard for crawlers and visitors to reach.
- Missing or duplicate metadata means pages without a title, description or main heading, or with a title another page also uses.
- Blocked URLs were skipped because robots.txt disallows them. Confirm that each block is intentional.
- Orphan pages are pages no crawled page links to. A crawl that follows links cannot find them, so compare its URL list with your sitemap or a CMS export.