Critical severity

How do I stop Cloudflare from blocking AI crawlers?

AI crawlers receive 401 or 403 from Cloudflare when Bot Fight Mode, Super Bot Fight Mode, or the "Block AI bots" setting rejects them before your application responds. Fix it by disabling the AI-bot block under Security, then adding a WAF custom rule that skips bot protection for the AI crawlers you want.

Agents affected

ChatGPT (GPTBot)ChatGPT search (OAI-SearchBot)Claude (ClaudeBot)Perplexity (PerplexityBot)Gemini (Google-Extended)CommonCrawl (CCBot)

How to tell you have this problem

Our scanner reports "blocked at the network/CDN layer" with a 401/403 status (check cloudflare_bot_protection_blocking, critical) — even though robots.txt allows the bot.
curl with a normal browser user-agent returns 200, but the same URL with an AI bot user-agent returns 403.
Cloudflare dashboard shows "Block AI Bots", Bot Fight Mode, or an "AI scrapers and crawlers" managed rule enabled.

How to fix it

In the Cloudflare dashboard go to Security → Bots (or Security → Settings on newer dashboards) and turn off "Block AI Bots" / set AI crawler handling to Allow. If Super Bot Fight Mode is on, make sure "Verified bots" is set to Allow.
2
Add a WAF custom rule (Security → WAF → Custom rules) with action "Skip" (skip Bot Fight Mode / Super Bot Fight Mode) so legitimate AI crawlers bypass bot protection. cf.client.bot matches Cloudflare-verified bots; the user-agent clauses cover AI crawlers not yet on the verified list:
```
(cf.client.bot) or
(http.user_agent contains "GPTBot") or
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "Google-Extended")
```

Verify each bot now gets a 200 (this is exactly what our scanner does):

curl -s -o /dev/null -w "GPTBot: %{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://yoursite.com/

curl -s -o /dev/null -w "ClaudeBot: %{http_code}\n" \
  -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])" \
  https://yoursite.com/

Re-run a free scan. If a bot still fails, check Cloudflare Security Events to see which rule fired and add it to the Skip rule.

FAQ

Is it safe to skip bot protection for these user-agents? Anyone can fake a user-agent.

Partially. cf.client.bot only matches bots Cloudflare has verified by IP, so it cannot be spoofed. The plain user-agent clauses can be, which is the trade-off for covering crawlers not yet verified. If that concerns you, keep only cf.client.bot and accept that some newer AI crawlers may still be blocked.

Why does robots.txt not fix this?

Cloudflare evaluates its firewall before the request ever reaches robots.txt semantics. A bot can be fully allowed in robots.txt and still get a 403 from the WAF. Both layers must pass — our scanner tests them separately.

Can I allow AI crawlers on some pages and block them on others?

Yes — scope the WAF skip rule with a path condition, e.g. add "and (http.request.uri.path ne \"/private\")". robots.txt Disallow rules for specific paths also work once the CDN lets the bot through.

Not sure if your site has this issue?

Run a free scan — we test this check (and 60+ others) against your live site and give you the exact paste-ready fix. No signup needed.