What is robots.txt?
The file always sits at the same address, for example https://www.example.co.uk/robots.txt. It only applies to exactly that host, protocol and port. A subdomain such as shop.example.co.uk therefore needs its own file.
Since 2022, the rules have been set out as the Robots Exclusion Protocol in the IETF specification RFC 9309. According to Google, robots.txt is mainly used to avoid overloading a website with requests. It is explicitly not access protection: reputable crawlers follow it, others ignore it. Confidential content therefore belongs behind a password.
Structure and rules
A robots.txt file consists of groups. Each group begins with one or more user-agent lines naming the crawler, followed by rules:
User-agent: *
Disallow: /basket/
Disallow: /*?sessionid=
Allow: /basket/help/
Sitemap: https://www.example.co.uk/sitemap.xml- User-agent: which crawler the group applies to; the asterisk stands for all.
- Disallow: paths that should not be crawled.
- Allow: exceptions within blocked areas.
- Sitemap: the address of your XML sitemap.
Paths begin with a slash and are case sensitive. The asterisk stands for any characters, the dollar sign for the end of the URL. If several rules match, Google applies the most specific one, meaning the one with the longest path. Google ignores the crawl-delay directive. The file must be UTF-8 encoded; Google reads at most 500 KiB and generally caches it for up to 24 hours.
Why robots.txt does not replace noindex
The most common misconception: a page blocked by robots.txt does not automatically disappear from Google. If other pages link to it, Google can still index the URL, but without a description in the search result.
If you want to keep a page out of the index, use the noindex rule in the meta robots data. It only works if Google is allowed to crawl the page. If you also block a noindex page in robots.txt, Google never sees the noindex.
Equally important: do not block CSS and JavaScript files that Google needs to render your pages.
AI crawlers in robots.txt
The major AI providers now distinguish between crawlers for training their models and crawlers for their search. This allows a targeted decision. According to the providers:
- GPTBot (OpenAI): collects content that may be used to train the models. Blocking it signals that your content should not be used for this.
- OAI-SearchBot (OpenAI): surfaces websites in ChatGPT’s search. According to OpenAI, blocked sites are not shown in its search answers. The two settings are independent of each other.
- Google-Extended (Google): not a crawler of its own but a token in robots.txt. It controls whether content is used to train Gemini models and for their answers (grounding). According to Google, it has no effect on inclusion or ranking in Google Search.
- ClaudeBot (Anthropic): collects content for training. Blocking it signals that future content should not be included in training data. For search, Anthropic uses Claude-SearchBot, and for retrieval on a user’s request, Claude-User.
If you do not want your content to be used for training, block the training crawlers. Search crawlers such as OAI-SearchBot and Claude-SearchBot remain allowed in this example:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /Note on Google-Extended: it controls training and grounding together. If you block it, you remain visible in Google Search, AI Overviews and AI Mode, but you no longer appear as a source in Gemini answers grounded in search. If Gemini should still be able to use your pages as a source, leave out this group.
For retrievals triggered directly by a person, such as via ChatGPT-User, OpenAI says the rules may not apply. And Anthropic points out that IP blocking is not a reliable substitute, because the crawler then cannot read robots.txt either. An overview of all bots is in the article on AI crawlers.
Checking your robots.txt
- Search Console: the robots.txt report in Google Search Console shows which file Google last fetched, with warnings and errors. After an important change, you can request a new fetch there.
- Status code: if the file returns a 4xx error (except 429), Google crawls without restrictions. With server errors (5xx) and status code 429, Google initially stops crawling altogether and then uses the last known version for up to 30 days.
- Relaunch: a “Disallow: /” from the test environment that moves to the live website along with everything else is a classic. Check the file after every go-live.
- Log files: a log file analysis shows whether crawlers follow your rules and which pages they actually fetch.
Would you like your robots.txt checked or to decide which AI crawlers get access? Then get in touch with us or read more about our technical SEO!
