Every AI crawler user-agent (2026 reference)
Last updated: July 10, 2026
TL;DR
AI crawlers do three different jobs: train models (GPTBot, ClaudeBot, CCBot), ground live answers from a search index (OAI-SearchBot, PerplexityBot, Claude-SearchBot), and fetch a page in real time when a user asks (ChatGPT-User, Perplexity-User). The reference below lists them all. The one rule most people get backwards: blocking the training bots does not reduce your citations — only blocking the retrieval and agent bots does.
The three jobs a crawler can do
Before the table, internalise the categories — they decide what happens when you allow or block each bot:
- Training crawlers collect content into a model's training corpus. Allowing them lets your writing shape what the model “knows”; blocking them has no effect on whether you get cited in answers.
- Retrieval crawlers build and refresh the search index an AI engine draws from when it composes a cited answer. These are the bots that decide whether you can be a source. Block them and you disappear from that engine's citations.
- Agent crawlers fire when a specific user asks something and the assistant fetches a page live to answer. Block them and you can't be pulled into up-to-the-minute answers.
The 2026 AI crawler reference
| User-agent | Operator | Job | Blocking it affects citations? |
|---|---|---|---|
| GPTBot | OpenAI | Training | No — training only |
| OAI-SearchBot | OpenAI | Retrieval | Yes — feeds ChatGPT search |
| ChatGPT-User | OpenAI | Agent (live fetch) | Yes — blocks live answers |
| ClaudeBot | Anthropic | Training | No — training only |
| Claude-SearchBot | Anthropic | Retrieval | Yes — feeds Claude search |
| Claude-User | Anthropic | Agent (live fetch) | Yes — blocks live answers |
| PerplexityBot | Perplexity | Retrieval | Yes — feeds Perplexity |
| Perplexity-User | Perplexity | Agent (live fetch) | Yes — blocks live answers |
| Google-Extended | Training (Gemini) | No — does not control AI Overviews | |
| Googlebot | Search + AI Overviews/AI Mode | Yes — and removes you from Search | |
| Applebot-Extended | Apple | Training (Apple Intelligence) | No — training only |
| Amazonbot | Amazon | Training / Alexa | Mostly no — training/assistant |
| Meta-ExternalAgent | Meta | Training | No — training only |
| Bytespider | ByteDance | Training (feeds Doubao) | No — training only |
| CCBot | Common Crawl | Training corpus (many models) | No — but feeds many trainers |
| DuckAssistBot | DuckDuckGo | Retrieval | Yes — feeds DuckAssist |
Don't block the wrong bots
The trap: you want to stop feeding free training data, so you add a blanket AI block — and accidentally take out the retrieval bots that were your path into cited answers. The two decisions are separate. If your goal is AI visibility without contributing to model training, a defensible starting point is to allow the retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot) while blocking the training bots (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended). And never blanket-block Googlebot — that removes you from Google Search and its AI surfaces entirely.
Verify a bot is who it claims to be
User-agent strings are trivially spoofed, so don't make allow/block decisions on the name alone. The real operators publish either official IP ranges or support reverse-DNS verification. Confirm a request's IP resolves back to the operator's domain before you trust it, and prefer matching against the published ranges for anything security-sensitive. A scan handles this for you — it fetches as each verified agent and reports what really came back.
FAQ
If I block GPTBot, will ChatGPT stop citing me?
No. GPTBot only gathers training data. ChatGPT's cited answers come from OAI-SearchBot (index) and ChatGPT-User (live fetch). Block those and you lose citations; blocking GPTBot alone just opts you out of training.
Does blocking Google-Extended remove me from AI Overviews?
No — a common misconception. Google-Extended governs Gemini training. AI Overviews and AI Mode are built from Google's regular Search index via Googlebot, so blocking Google-Extended doesn't take you out of them (and blocking Googlebot takes you out of Google altogether). See ranking in AI Overviews.
How do I set all this in robots.txt?
One User-agent block per bot with your Allow/Disallow rules. The full walkthrough with copy-paste examples is in robots.txt for AI crawlers, and the Cloudflare guide covers the CDN layer that overrides it.
Not sure which bots your site currently lets in? Run a free per-agent scan — it fetches as each crawler and shows you exactly who's reaching you and who's blocked.
Related reading
- 8 min read
Why AI assistants see a blank page on your site
Most AI crawlers do not run JavaScript, so a client-side-rendered site serves them an empty page. Why the rendering gap happens and how to fix it.
- 8 min read
How to check if your website is visible to AI
Not sure if ChatGPT, Perplexity or Gemini can see your site? How to check if your website is visible to AI crawlers — the fast way and the manual checks.
- 8 min read
What is an AI Readiness Score (and how to check yours)
An AI Readiness Score is a free 0–100 grade for how well AI crawlers like GPTBot and Perplexity read your site — what it measures and how to check yours.