Feature
Site Crawler
Crawl up to 100 pages of a website from any starting page to find broken pages and missing titles, descriptions and H1s, then explore how they connect in a map.
The crawler starts at the URL you give it, follows links on the same site level by level and records every page it reaches: status code, response time, depth, title, description, H1 and the links in and out of it.
Limits you control
Set how many links deep to go (up to 5) and how many pages to collect (up to 100). The crawler reads robots.txt first and follows its rules for all crawlers, including Allow exceptions and wildcards. Pages are fetched one at a time.
The link map
Results can be viewed as a table of issues, a page list, or an interactive map of the site’s internal links that you can rotate, zoom and filter. Selecting a page shows its details. A list view and a flat 2D view are available for keyboard and screen-reader users or slower devices.
Issues it reports
- Broken pages: 4xx and 5xx responses, and pages that couldn’t be fetched
- Missing titles, meta descriptions or H1 headings
- Duplicate titles
Limits to keep in mind
Only pages reachable through links from the start page are crawled.
The crawler reads the HTML the server sends and doesn’t run JavaScript, so pages built entirely with JavaScript may show less than a browser does.
Common questions
- Can I export the crawl?
- Yes. The page list downloads as a CSV file.
- Does it need an account?
- Yes. Crawling needs a free account, and crawled pages count toward a daily allowance of 500 pages on the free plan.