Workflow
Domain Crawling
Use a crawl to see how a website is actually connected: which pages link where, how deep important pages sit, and which sections are isolated from the rest of the site.
A crawl answers a question your CMS can’t: what can a visitor or search engine reach by following links? Pages can exist, appear in the sitemap and still be nearly impossible to find because nothing links to them.
What a crawl reveals
- Click depth: how many links it takes to reach each page from the start
- Weakly linked pages that receive few internal links
- Sections of the site joined to the rest by a single link
- Broken internal links and redirects inside navigation
Reading the link map
The crawler draws the site as an interactive map of pages and the links between them. Clusters show sections; long thin branches show content buried several clicks deep. Select any page to see its status, depth, links in and out, and metadata.
Running a crawl
- Start from the homepage, or the top page of the section you want to map
- Set the depth (up to 5 links) and page limit (up to 100)
- Run the crawl; paths that robots.txt disallows are skipped, and Allow exceptions are honored
- Open the site map view and look for deep or isolated pages
- Export the page list as CSV for planning internal-link changes
Limits to keep in mind
Only pages reachable by links from the starting page are found.
Content that needs a login or is loaded only by scripts after the page renders may not be discovered.
Crawling needs a free account, and crawled pages count toward a daily allowance of 500 on the free plan.
Common questions
- Will crawling slow down my site?
- A crawl of up to 100 pages is light: pages are fetched one at a time rather than all at once. Even so, avoid running repeated crawls against a site you don’t own.
- Why are some pages missing from the crawl?
- Nothing links to them from the crawled pages, they sit deeper than the depth limit, robots.txt disallows them, or the page limit was reached first.