Understanding the Robots Exclusion Protocol (REP)
The Robots Exclusion Standard (RFC 9309) governs how automated web crawlers and search engine indexing bots interact with your web server. Placed at the site's root directory, a robots.txt file provides formal instructions specifying which paths crawlers may traverse and which should remain excluded to prevent server resource exhaustion and crawl budget dissipation.
Crawl Budget Optimization
Search engines allocate a finite crawling quota to each domain based on speed and popularity. Disallowing low-value, parameter-heavy search query pages, faceted filters, and private admin dashboards ensures crawlers focus strictly on high-priority transactional and informational pages.
Security vs. Obfuscation
A robots.txt file is publicly accessible to anyone navigating to /robots.txt. Never use Disallow directives to conceal proprietary or secret paths, as malicious bots systematically scrape robots.txt files to discover unprotected directories. Always enforce server-side authentication headers.
Standard Robots.txt Directives Reference
| Directive | Syntax Example | Crawler Support | Operational Purpose |
|---|---|---|---|
| User-agent | User-agent: Googlebot | All Spiders (Universal) | Names the specific robot the following block applies to |
| Disallow | Disallow: /admin/ | All Spiders (Universal) | Blocks crawler requests to matching directory paths |
| Allow | Allow: /admin/login | Google, Bing, Yandex | Overrides a broader Disallow rule for an explicit child URL |
| Crawl-delay | Crawl-delay: 10 | Bing, Yahoo, Yandex | Sets waiting period (in seconds) between bot requests |
| Sitemap | Sitemap: https://... | All Major Search Engines | Points scrapers to the XML sitemap index directly |
Frequently Asked Questions
Where must the robots.txt file be hosted on a web server?
The robots.txt file must always reside at the absolute root directory of your website domain (e.g., https://example.com/robots.txt). Placing it in a subfolder or subdirectory will cause web crawlers to ignore it completely.
Does a Disallow rule in robots.txt guarantee a page will not be indexed by Google?
No. A Disallow directive tells search engine crawlers not to fetch or crawl the URL's contents. However, if external websites link to that URL, Google may still index the link without content snippets. To completely prevent indexing, allow crawling and implement a noindex meta tag or X-Robots-Tag HTTP header instead.
Does Googlebot respect the Crawl-delay directive?
No. Standard Googlebot does not support the Crawl-delay directive in robots.txt. Crawl speed for Google must be managed through Google Search Console settings. Other search engines like Bing, Yahoo (Slurp), and Yandex do recognize and respect Crawl-delay.