Is Cloudflare blocking your AI crawlers?

The silent block: how it happens

robots.txt is a polite request a crawler reads after it connects. A CDN or WAF block happens before that — the request gets a 403 (or a managed challenge) and never sees your content or your rules. So the two layers can disagree: your robots.txt says Allow, but Cloudflare says no. The result is a site that looks perfectly open to you and is completely invisible to ChatGPT, Claude and Perplexity. It is the single most common reason a technically healthy site gets no AI citations.

Three Cloudflare features cause this, often without anyone choosing them deliberately:

• Super Bot Fight Mode → “AI Scrapers and Crawlers”. A single toggle that blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others. On many plans it defaults to on.
• Bot Fight Mode.Challenges “automated” traffic broadly; legitimate AI crawlers that don't solve the challenge get dropped.
• Managed robots.txt. Cloudflare can prepend its own Disallow rules at the top of your robots.txt — so even if you added User-agent: GPTBot / Allow: / below, the earlier block wins.

How to check if you're blocked

Don't guess — verify. Three ways, fastest first:

1. Audit your URL as the AI bots. Run a free audit— we fetch your site with each AI crawler's user-agent and report exactly which ones get a 403, a challenge, or an empty page. That's the fastest way to see the gap the way the models do.
2. Check your Cloudflare dashboard.Security → Bots. Look at whether Bot Fight Mode is on, and (if you have Super Bot Fight Mode) whether “AI Scrapers and Crawlers” is set to Block.
3. Read your logs. Filter access logs for user-agents containing GPTBot, ClaudeBot, PerplexityBot and check the status codes. A wall of 403s is your answer. No hits at all can mean the block happens before logging.

Should you block or allow AI bots?

This is a real decision, not a default to leave on autopilot. Blocking the training and retrieval crawlers keeps your content out of AI answers entirely — which protects content you don't want reused, but also removes any chance of being the cited source when someone asks ChatGPT about your topic. For most marketing and product sites, visibility is the goal, so you want these bots allowed. For publishers and paywalled content, the calculus is different. The important thing is that the choice is intentional — most silent blocks are accidental, and accidental invisibility helps no one.

It's also not all-or-nothing. If you want AI bots to access some pages but pay for others — or be blocked from premium content while your marketing pages stay visible — you can enforce that per URL at the edge with Bot Paywall instead of a blunt site-wide block.

How to let the AI crawlers through

If you've decided to be visible, fix both layers — the edge and robots.txt:

1. In Cloudflare: Security → Bots → Configure Super Bot Fight Mode → set AI Scrapers and Crawlers to Allow (or create a WAF exception for the verified AI bots you want).
2. If you use Managed robots.txt, check what Cloudflare prepended and make sure it isn't disallowing the bots you want.
3. In your own /robots.txt, explicitly allow them — see our GPTBot reference and robots.txt for AI crawlers for the exact, copy-paste directives and the full crawler list.
4. Re-audit to confirm the 403s are gone and the bots now see your real content.

FAQ

Not sure whether your AI crawlers are getting through? Run a free audit — it checks every major AI bot in about 30 seconds — or see the plans.