robots.txt Rules and Crawling Directives

A robots.txt file is a plaintext file placed at the root of a website (e.g. `/robots.txt`) instructing search engines and other web robots which folders they are allowed to crawl.

The `Disallow` rule blocks crawlers from requesting pages in that folder. Note that robots.txt blocks crawling, not indexing. If a disallowed page is linked from elsewhere, it can still appear in search results. To prevent indexing, use `noindex` meta tags instead.