How do I fix GPTBot being blocked by robots.txt?

Agents affected

How to tell you have this problem

• •Our scanner reports "robots.txt blocks OpenAI GPTBot (training)" (check robots_txt_blocks_gptbot, critical).
• •Your content never shows up in ChatGPT answers, even for queries where you should be the obvious source.
• •robots.txt contains "User-agent: GPTBot" followed by "Disallow: /", or a catch-all "User-agent: *" Disallow that GPTBot inherits.

How to fix it

1. 1

   Open https://yoursite.com/robots.txt and look for a blocking rule like this (a catch-all "User-agent: *" with "Disallow: /" blocks GPTBot too, unless GPTBot has its own group):

   User-agent: GPTBot
   Disallow: /

2. 2

   Delete the Disallow rule, or add explicit Allow groups for the OpenAI crawlers you want. A crawler obeys its most specific matching group, so these override a catch-all block:

   User-agent: GPTBot
   Allow: /
   
   User-agent: OAI-SearchBot
   Allow: /
   
   User-agent: ChatGPT-User
   Allow: /

3. 3

   Redeploy and verify the live file now contains the Allow groups:

   curl -s https://yoursite.com/robots.txt | grep -i -A 1 "gptbot\|oai-searchbot\|chatgpt-user"

4. 4

   Re-run a free scan to confirm the robots_txt_blocks_gptbot finding is gone. Changes take effect the next time GPTBot re-fetches robots.txt (typically within about 24 hours).

FAQ