How do I remove an X-Robots-Tag noindex header that blocks AI crawlers?

Agents affected

How to tell you have this problem

• •Our scanner reports "X-Robots-Tag header sets noindex" (check x_robots_tag_noindex, critical) or "noai/noimageai" (x_robots_tag_blocks_ai, critical).
• •The page vanished from Google and AI answers even though robots.txt allows everything.
• •curl -I shows a header like "X-Robots-Tag: noindex" or "X-Robots-Tag: noai".

How to fix it

1. 1

   Confirm the header and its exact value:

   curl -sI https://yoursite.com/ | grep -i x-robots-tag

2. 2

   Find and remove the source. In nginx, look for and delete a line like this in your server/location blocks (then reload nginx):

   # nginx — DELETE this line if present, then: sudo nginx -s reload
   add_header X-Robots-Tag "noindex" always;

3. 3

   In Apache, either delete the matching "Header set X-Robots-Tag" line, or force-remove a header set upstream:

   # Apache (.htaccess or vhost)
   <IfModule mod_headers.c>
     Header always unset X-Robots-Tag
   </IfModule>

4. 4

   Also check layers that inject headers: CDN response-header transform rules (Cloudflare → Rules → Modify Response Header), your framework config (e.g. a headers() entry in next.config), hosting dashboards, and SEO plugins. Staging configs that set noindex often leak into production this way.

5. 5

   Re-run the curl check — the header should be gone (or contain only safe values like "index, follow") — then re-scan.

FAQ