Blocked by robots.txt
Cloudflare blocking AI crawlers
A managed robots.txt or a bot rule can stop AI crawlers even when Googlebot is allowed.
Cloudflare can publish a managed robots.txt that blocks GPTBot, ClaudeBot, and other AI crawlers, and it can challenge those user-agents at the edge. Googlebot may still be allowed. The two decisions are separate.
A block in the firewall matters more than a line in robots.txt. robots.txt is voluntary. A challenge page or 403 is a hard stop, including for AI search crawlers such as OAI-SearchBot and PerplexityBot.
Decide on purpose. Blocking training crawlers is a normal choice and does not by itself stop Google. Blocking Google-Extended is about Gemini training, not Google Search. Blocking OAI-SearchBot is about being cited in ChatGPT search.
What to check
- Read the live robots.txt, including any Cloudflare managed content.
- Fetch the URL with each crawler user-agent and note 403 or a challenge.
- Remember a fetch from IndexDoctor is not the crawler’s IP. Confirm the rule in Cloudflare if the result matters.
Fix prompt
Example prompt
Cloudflare or robots.txt is blocking one or more AI crawlers. If those crawlers should read the page, allow them in the bot rules and in robots.txt. Leave training bots blocked if that is intentional. Do not call an indexing API.
Check a URL · Crawlers · Glossary