● The short answer
Most businesses should not block AI search crawlers, because blocking them can keep you out of AI answers. The smart move is to separate the bots: you can block training crawlers like GPTBot, ClaudeBot or Google-Extended if you object to training use, while allowing search crawlers like OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot that decide whether you get cited.
Key takeaways
- AI companies run separate crawlers for training, search citation and user-requested fetches.
- OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers.
- Blocking training crawlers is a content-rights choice; blocking search crawlers is a visibility cost.
The question is really two questions
AI companies now run different crawlers for different jobs. Some collect content to train future models. Others index pages so an assistant can cite them in a live answer. A third type fetches a page only when a person asks about it. Blocking "AI crawlers" as one group mixes up a content-rights decision with a visibility decision.
| Company | Training | Search / citation | User-requested fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | Not used for foundation models, per Perplexity | PerplexityBot | Perplexity-User |
| Google-Extended (a control token, also covers Gemini app grounding) | Googlebot | n/a | |
| Microsoft | n/a | Bingbot (also grounds Copilot) | n/a |
OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that disallowing GPTBot signals content should not be used for training. Anthropic says blocking Claude-SearchBot may reduce visibility in Claude's search results. Google says Google-Extended does not affect inclusion or ranking in Google Search.
The how is where we come in.
What we walk through with you, for your business:
- When blocking training crawlers makes sense
- Check for blocks you did not choose
- Our take
Every business is different, so the plan is too. Tell us where you want to show up and we will show you what it takes.
Talk to usMore questions
If I block GPTBot, will ChatGPT stop mentioning me?
Not necessarily. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search. Blocking only GPTBot leaves you eligible for ChatGPT search citations. Your existing presence in older model training data also remains. Blocking OAI-SearchBot is what removes you from ChatGPT search answers.
Does robots.txt actually stop AI crawlers?
The major AI companies say their crawlers honor robots.txt. User-initiated fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. For a hard block, use firewall rules matched to published IP ranges, accepting that you lose citations.
Sources
- OpenAI: Overview of OpenAI crawlers
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central: Google's common crawlers
- Cloudflare: The crawl-to-click gap, Cloudflare data on AI bots, training, and referrals
- MIT Technology Review: Cloudflare will now, by default, block AI bots from crawling its clients' websites
Want us to check this for your business?
Send us your URL. We will show you where you are found, trusted and missing across AI answers, search, maps and reviews, in plain English.
Get a free visibility check