What are AI crawlers?
A crawler is a program that fetches web pages automatically. Search engines such as Google have always used crawlers. Generative AI has added new ones, which fall into three groups according to their purpose:
- Training crawlers: collect content used to train future language models.
- Search crawlers: build an index from which AI search draws its answers and source links.
- User-initiated fetchers: fetch a page at the moment a user asks a question or shares a link.
The distinction matters because blocking has very different consequences depending on the group. If you want to appear in AI answers as part of generative engine optimisation, you must not block the search crawlers.
The main AI crawlers at a glance
OpenAI (according to the OpenAI documentation):
- GPTBot: training of foundation models. Blocking it signals that the content should not be used for training.
- OAI-SearchBot: search results in ChatGPT. If you block it, your site no longer appears in ChatGPT’s search answers, at most as a navigational link.
- ChatGPT-User: fetches triggered by a user in ChatGPT. Because the user initiates the fetch, robots.txt rules may not apply, according to OpenAI.
Google:
- Google-Extended: not a crawler of its own but a control token in robots.txt. It determines whether content Google crawls anyway may be used to train future Gemini models and for grounding, i.e. looking up content at the moment of answering. According to Google, it has no effect on Google Search or ranking.
- Googlebot: AI Overviews and AI Mode in Google Search draw on the normal search index. They can only be controlled via Googlebot and snippet settings such as nosnippet, not via Google-Extended.
Anthropic:
- ClaudeBot: training. Claude-SearchBot: search index. Claude-User: fetches for users’ questions. According to Anthropic, all three respect robots.txt.
Perplexity:
- PerplexityBot: search results in Perplexity, not used to train foundation models according to the provider. Perplexity-User: fetches for user queries, which according to Perplexity generally ignores robots.txt.
Controlling AI crawlers via robots.txt
Each crawler is addressed by name in robots.txt. The following example excludes the training crawlers of OpenAI and Anthropic as well as Google-Extended, and explicitly allows OpenAI’s search crawler. With Google-Extended, you block training and grounding in Gemini at the same time: you remain visible in Google Search, AI Overviews and AI Mode, but no longer as a source for Gemini answers grounded in search.
User-agent: GPTBot
Disallow: /User-agent: ClaudeBot
Disallow: /User-agent: Google-Extended
Disallow: /User-agent: OAI-SearchBot
Allow: /
Keep in mind:
- Each rule works on its own. According to OpenAI, the settings for GPTBot and OAI-SearchBot are independent of each other. If both are allowed, OpenAI may use one fetch for both purposes.
- Changes take time. For ChatGPT search, OpenAI states around 24 hours until a changed robots.txt is taken into account.
- robots.txt is a request, not protection. Reputable crawlers follow it, user-initiated fetchers sometimes do not, and fake bots ignore it. If you really need to protect content, you need access control on the server.
- Do not confuse: llms.txt is an overview file for language models and does not control access.
Training or search: what should you block?
There is no one-size-fits-all answer. It helps to split the question:
- Training: if you do not want your content to feed into future models, block GPTBot, ClaudeBot and Google-Extended. According to the providers, visibility in Google Search and ChatGPT search does not depend on them. However, Google-Extended also controls whether Gemini may use your content at the moment of answering. If you block it, you no longer appear as a source in Gemini answers grounded in search. AI Overviews and AI Mode are not affected.
- Search and user-initiated fetches: if you block OAI-SearchBot, Claude-SearchBot, PerplexityBot or the user-initiated fetchers, you reduce your chances of appearing as a linked source in these systems’ answers. According to OpenAI, sites that block OAI-SearchBot no longer appear in its search answers, at most as navigational links. For companies that want to be found via AI assistants, this is usually counterproductive.
- Individual sections: instead of blocking the whole website, you can also exclude individual directories, such as internal search results or filter pages.
Identifying AI crawlers in log files
Only the server log files show which AI crawlers actually visit your website. Web analytics tools generally do not record bots: many do not execute JavaScript, and known bots are filtered out automatically in Google Analytics, for example.
The name in the user agent alone proves nothing, as it can easily be faked. OpenAI, Anthropic and Perplexity therefore publish lists of their IP addresses, and Google offers IP lists and verification via reverse DNS. Only requests from these networks genuinely come from the named provider. Incidentally, Google-Extended never appears in the logs, because it has no user agent of its own.
Would you like to know which AI crawlers really read your website and set up your robots.txt accordingly? Then get in touch with us or read more about our log file analysis!
