Robots.txt

Robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which parts of the site they may or may not fetch. It uses a simple syntax of user-agent declarations and allow/disallow rules. Robots.txt is one of the oldest web standards, defined by the Robots Exclusion Protocol, and remains the primary mechanism for managing crawler access at scale.

What Robots.txt Means

Robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which parts of the site they may or may not fetch. It uses a simple syntax of user-agent declarations and allow/disallow rules. The file is fetched by crawlers as the first request when visiting a domain, before crawling any other pages. Path-based rules can block specific URL patterns: Disallow: /admin/ blocks anything under /admin/; Disallow: /*.pdf$ blocks all PDF files. Wildcards (*) and end-anchors ($) are supported by most major crawlers. The file must be served with a 200 response and plain-text content type.

How Robots.txt Works

Robots.txt works as an editorial signal, not a security control. Reputable search and AI crawlers — Googlebot, Bingbot, GPTBot, ClaudeBot, and most others — honor robots.txt directives, but compliance is voluntary. Less reputable bots and direct URL access ignore it. Anything genuinely confidential must be protected by authentication, not by a disallow rule. The right mental model is 'request,' not 'enforce.' Disallow prevents crawling — bots won't fetch the URL. Noindex (in meta tags or HTTP headers) prevents indexing — bots may crawl but won't add the URL to the index. They aren't interchangeable, and using disallow alone can leave URLs visible in search with sparse listings.

Common Pitfalls and Misconceptions

A common mistake is conflating disallow in robots.txt with noindex. Robots.txt prevents crawling; noindex (in meta tags or HTTP headers) prevents indexing. A page disallowed in robots.txt can still appear in search results if other signals (external links, sitemap entries) make Google aware of the URL — Google just won't have crawled the page, so it shows a sparse listing without snippet. To reliably keep a page out of search, allow crawl and use noindex. Another error is listing sensitive paths in robots.txt — the file is publicly readable, so disallowing /admin/ effectively advertises its existence to anyone curious enough to check.

Robots.txt in Practice

The practitioner pattern is to treat robots.txt as a deliberate, version-controlled policy file rather than ad hoc rules accumulated over years. Audit periodically against site architecture and crawler behavior: which user agents are addressed, which paths are blocked, whether any important sections have been accidentally disallowed, and whether the file accurately reflects current intent. AI crawler additions over the last few years have made many older robots.txt files incomplete — addressing GPTBot, ClaudeBot, Google-Extended, and the rest is now part of routine maintenance, not an exotic addition. Test changes in Google Search Console's robots.txt tester before deploying.