robots.txt for AI crawlers, done right

Why robots.txt is the first thing that breaks

Before an assistant can summarize or cite a page, its crawler has to be allowed to fetch it. robots.txt is the very first checkpoint, and it is the single most common reason a site is invisible to AI — usually not from a deliberate block, but from a default rule, a migrated config, or a CDN setting nobody revisited. It is also the cheapest thing to get right.

The user-agents that matter

These are distinct bots with distinct user-agents — a rule for one does not cover another:

• GPTBot — OpenAI training crawler. Blocking it removes you from future ChatGPT training but does not affect retrieval. See the full GPTBot reference.
• OAI-SearchBot— populates ChatGPT's search index/cache. Blocking it makes you invisible to ChatGPT Search even if GPTBot is allowed.
• ChatGPT-User — fetches a specific URL on behalf of a live chat. Blocking it stops grounding/verification. Roles in detail in ChatGPT Search SEO.
• ClaudeBot (and the legacy anthropic-ai, Claude-Web) — Anthropic.
• PerplexityBot — Perplexity's indexing crawler.
• Perplexity-User— Perplexity's live per-query fetch (analogous to ChatGPT-User). More on the split in Perplexity SEO.
• Google-Extended — controls Gemini/Vertex training use withoutaffecting normal Google Search ranking or AI Overviews inclusion.
• Bytespider — ByteDance/Doubao, mixed compliance reputation.

Directives that actually work

Keep it explicit. A permissive default plus per-bot groups that allow your public content and disallow private areas (dashboards, auth, internal endpoints) is the pattern that ages well. Two rules of thumb: Google-Extended is not Googlebot — disallowing it does not hurt your search ranking, it only opts you out of Gemini; and an empty Disallow:means “allow everything”, which is often what you want for marketing pages. Add a Sitemap: line so discovery is not guesswork.

A note on Content-Signal

The contentsignals.org Content-Signal:line lets you express purpose-based consent (search vs. AI input vs. AI training) alongside the bot-by-bot rules. Adoption is early and some strict validators flag it as an “unknown directive”, but compliant crawlers ignore lines they do not understand (RFC 9309), so it is safe to include if it matches your policy.

The silent trap: your CDN, not your file

Your robots.txt can be perfect and you can still be invisible. Cloudflare bot-fight mode, WAF rules, and “block AI scrapers” toggles return 403/401/429 to AI user-agents before the request ever reaches your origin — robots.txt never gets a say. Always test what the bots actually receive, post-CDN, not what your file says. If you would rather monetize than block, that is a different lever entirely: Bot Paywall charges specific bots per URL instead of refusing them.

Verify it the way a bot sees it

Reading your own robots.txt in a browser proves nothing — you are not GPTBot. Fetch the page as each AI user-agent and check the real status code. You can run a free audit that does exactly this (raw fetch, per-agent, post-CDN) and flags blocked bots by severity. Once crawlability is clean, the next layers are structure and an llms.txt — and the broader checklist is in getting cited by AI assistants.

FAQ

Check your robots.txt against every AI bot: run a free audit or see the plans.