Critical severity

How do I fix ClaudeBot being blocked by robots.txt?

ClaudeBot is blocked when robots.txt contains a Disallow rule that matches Anthropic crawlers — either under User-agent: ClaudeBot or under a catch-all User-agent: * group. Delete the Disallow line or add an Allow stanza for ClaudeBot, plus Claude-User and Claude-Web for live browsing, then redeploy and re-scan.

Agents affected

Claude (ClaudeBot)Claude browse (Claude-User)Claude browse (Claude-Web)

How to tell you have this problem

Our scanner reports "robots.txt blocks Anthropic ClaudeBot (training)" (check robots_txt_blocks_claudebot, critical).
Claude cannot fetch or summarize your pages when users paste your URLs.
robots.txt has "User-agent: ClaudeBot" with a Disallow, or a blocklist copied from a "block all AI bots" template.

How to fix it

1
Check the live file for Anthropic user-agents and catch-all blocks:
```
curl -s https://yoursite.com/robots.txt | grep -i -B 1 "disallow"
```
2
Remove the blocking rule or add explicit Allow groups. ClaudeBot is the training crawler; Claude-User and Claude-Web are the browse-time fetchers used when someone asks Claude about your site:
```
User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-Web
Allow: /
```
If you previously used the legacy "anthropic-ai" agent name in a Disallow, remove that too — our scanner flags it separately (robots_txt_blocks_anthropic_ai).
Redeploy, then re-run a free scan to confirm all Anthropic user-agents pass.

FAQ

What is the difference between ClaudeBot, Claude-User, and Claude-Web?

ClaudeBot crawls the web for training data. Claude-User fetches a page when an individual user asks Claude to read it. Claude-Web supports Claude web browsing features. Blocking only ClaudeBot still lets Claude cite you live; blocking all three makes you invisible to Claude entirely.

Can I allow Claude to cite me but opt out of training?

Yes. Keep Claude-User and Claude-Web allowed while disallowing ClaudeBot. You can also express this intent with a Content-Signal line in robots.txt (search=yes, ai-input=yes, ai-train=no) — an emerging convention our scanner also checks.

How fast does Anthropic pick up robots.txt changes?

Anthropic re-fetches robots.txt regularly, typically within a day. The change applies from the next crawl — it does not retroactively remove anything already crawled.

Not sure if your site has this issue?

Run a free scan — we test this check (and 60+ others) against your live site and give you the exact paste-ready fix. No signup needed.