Cloudflare blocks mixed-use AI crawlers on September 15 — is your site affected?

What exactly changes on September 15

Cloudflare gave AI companies a deadline: separate your crawlers by purpose — search, training, agents — or be treated as “mixed-use” and blocked by default on monetized pages across Cloudflare's network. The announcement (Cloudflare, July 1, 2026, TechCrunch coverage) is part of a bigger shift: Cloudflare says its customers already send more than one billion HTTP 402 Payment Required responses to AI crawlers per day, and its Pay Per Crawl program is evolving into “Pay Per Use” — compensating sites when content is actually used in an AI answer.

Who is affected

You are in the blast radius if all three are true:

• Your site is behind Cloudflare — free or paid plan.
• Your pages show ads — AdSense, programmatic display, an ads.txt file.
• You want AI visibility — you'd like ChatGPT, Perplexity or Gemini to read and cite your content.

If you'd rather block AI bots, September 15 is good news: the default moves in your favor, and the monetization path (402 + Pay Per Use) gets stronger. The problem is the silent middle: publishers who assume “my robots.txt allows GPTBot, so I'm fine” while the edge says otherwise. We have seen this failure mode before — Cloudflare's managed robots.txt and bot-fight settings have been quietly blocking AI crawlers on sites that never chose it.

Why robots.txt can't fix this

robots.txt is a request your server publishes; Cloudflare's block is enforced at the edge, before the bot ever reaches your origin or reads your robots.txt. A crawler classified as mixed-use gets a block response from Cloudflare itself. That means your allow rules become decorative — the only switch that matters lives in your Cloudflare dashboard (AI Crawl Control, formerly the AI Audit beta).

How to check your site before the deadline

• Run a free scan — our quick check fetches your site as GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Doubao and Grok and reports who gets blocked, including Cloudflare-level blocks.
• Check AI Crawl Control — in the Cloudflare dashboard, review which AI crawlers are allowed, blocked, or set to the new default.
• Check for ad signals — if your site serves ads (an ads.txt file is the clearest marker), assume the September 15 default applies to you.

What to do — both directions

If you want AI visibility: explicitly allow the crawlers you care about in AI Crawl Control before September 15, keep your robots.txt allows in place (they still matter for bots that do reach the origin), and re-scan after the deadline to confirm nothing regressed.

If you want control or compensation: the default block plus Pay Per Use is the strongest position independent publishers have had since AI crawling started. Decide per-bot, deliberately — blocking OAI-SearchBot removes you from ChatGPT search results, while blocking GPTBot only affects training. Blocking the wrong one is the most common self-inflicted wound we see in audits.

FAQ

My site has no ads — am I affected?

The September 15 default targets ad-monetized pages. Without ad signals you are outside the announced scope, but Cloudflare's broader direction (default-blocking unverified AI crawlers for new zones since 2025) still applies — worth a scan either way.

I'm not on Cloudflare — can I ignore this?

The deadline is Cloudflare-specific, but roughly a fifth of the web sits behind it, and other CDNs tend to follow its lead on bot policy. The habit that protects you everywhere: verify per-agent access after any CDN or security change.

Which crawlers count as “mixed-use”?

Any bot that doesn't cleanly separate search indexing from AI training or agent traffic. Cloudflare has pushed AI companies to declare purpose per user-agent; bots that don't comply get the default block on monetized pages. The precise list will firm up as vendors respond — we'll update this post as it does.