● The short answer
The main AI crawlers in 2026 fall into three groups. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot index pages that assistants cite. User fetchers such as ChatGPT-User, Claude-User and Perplexity-User open a page when a person asks. Each can be managed separately.
Key takeaways
- AI bots fall into training crawlers, search crawlers and user-requested fetchers.
- Blocking search crawlers like OAI-SearchBot, Claude-SearchBot or PerplexityBot can remove you from AI answers.
- Google-Extended and Applebot-Extended are control tokens, not separate crawlers.
Three kinds of AI bots
An AI crawler is any automated agent that fetches web pages for an AI system. They differ in why they visit, and that difference determines what blocking them costs you.
- Training crawlers gather content that may be used to train future models. Blocking them affects what future models learn, not today's search citations.
- Search crawlers build the index an assistant searches when answering. Blocking them can remove you from AI answers.
- User fetchers visit a specific page because a person asked the assistant something that needed it. Several vendors say robots.txt may not apply to these.
The how is where we come in.
What we walk through with you, for your business:
- The crawlers you are most likely to see
- How much crawling is going on
- How to see and manage them
- Our take
Every business is different, so the plan is too. Tell us where you want to show up and we will show you what it takes.
Talk to usMore questions
How do I know a bot is really GPTBot and not an imposter?
User-agent strings are easy to fake. OpenAI, Perplexity and other vendors publish the IP ranges their bots use, so you can verify requests by matching both the user agent and the source IP. Many CDNs do this automatically and label verified bots in their dashboards.
Do AI crawlers slow down my site?
They can, especially aggressive training crawlers hitting many pages quickly. Some respect a crawl-delay directive; Anthropic says ClaudeBot does. If load is a problem, rate-limit at the CDN or firewall rather than blocking search crawlers outright, so you protect performance without losing AI citations.
Sources
- OpenAI: Overview of OpenAI crawlers
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity Docs: Perplexity crawlers
- Google Search Central: Google's common crawlers
- Cloudflare: The crawl-to-click gap, Cloudflare data on AI bots, training, and referrals
Want us to check this for your business?
Send us your URL. We will show you where you are found, trusted and missing across AI answers, search, maps and reviews, in plain English.
Get a free visibility check