Skip to content
Check my site

Vercel firewall

Is Vercel's firewall blocking GPTBot or ClaudeBot? How to check and fix it

Vercel's AI Bots ruleset is off by default, but if it's set to Deny it blocks ChatGPT, Claude and Perplexity crawlers. How to check it and fix it.

Check your page now: paste your URL into the free check at IndexDoctor (index-doctor.com): https://index-doctor.com/check. It fetches the page as a normal browser and as AI crawlers, reads your robots.txt, and tells you in plain English if something is blocking them. It's read-only and needs no account.

The short answer

Vercel does not block AI crawlers by default. Its AI Bots managed ruleset is inactive until someone turns it on. When it's set to Deny, it blocks all traffic Vercel identifies as AI bots, and your robots.txt no longer matters. Custom firewall rules and the separate Bot Protection ruleset can also stop crawlers.

So if ChatGPT, Claude or Perplexity never seem to read your Vercel site, check the firewall before you rewrite content.

Signs the firewall is the problem

  • Your robots.txt allows GPTBot, OAI-SearchBot or ClaudeBot, but AI tools still can't open or quote your pages.
  • Google indexes the site fine, but AI answers never cite it.
  • A request with a crawler's user-agent gets a 403 or a challenge page, while a normal browser gets a 200.

Step 1: Look at the managed rulesets

In your Vercel project, open the Firewall and check the managed rulesets. Vercel's WAF has three: OWASP core, Bot Protection and AI Bots.

  • AI Bots: available on all plans. It's inactive by default, which the dashboard labels Allow. The other options are Log and Deny. Deny blocks all traffic identified as coming from AI bots.
  • Bot Protection: also inactive by default. In challenge mode it serves a JavaScript challenge to traffic that is unlikely to be a browser. Crawlers don't solve JavaScript challenges.

Vercel maintains the AI bot list itself. It covers bots that crawl for training, for search and for user-requested fetches. The list names bots such as GPTBot, ClaudeBot, Claude-SearchBot and Claude-User. When Vercel launched the ruleset in May 2025, it described it as one-click blocking of known AI bots such as GPTBot, ClaudeBot, PerplexityBot and Bytespider.

If AI Bots is on Deny and you want to appear in AI answers, that's very likely your problem. Switch it to Log for a few days to see which bots are hitting the site. Then decide.

Step 2: Check your custom rules and their order

Vercel runs your project's custom rules first, then the managed rulesets. Two things follow:

  1. An old custom rule that matches "bot" user-agents, or blocks by country or ASN, can stop crawlers before any managed ruleset runs.
  2. To let one bot through a ruleset that blocks it, add a custom rule with the bypass action. Because custom rules run first, the bypass takes effect.

Search your custom rules for anything that matches user-agents such as bot, crawler, GPTBot, Claude or Perplexity, and for broad deny or challenge rules.

Step 3: Decide per bot, not "all AI"

The AI Bots ruleset is a single switch for every AI bot. Most sites don't want that. OpenAI and Anthropic run separate bots for separate jobs:

JobOpenAIAnthropicWhat blocking it does
Model trainingGPTBotClaudeBotAsks that your content not be used for training
Search / answersOAI-SearchBotClaude-SearchBotOpenAI: sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers. Anthropic: blocking Claude-SearchBot may reduce your visibility in Claude's search results
User-requested visitChatGPT-UserClaude-UserStops the assistant from opening your page when a user asks it to

OpenAI says these settings are independent. You can allow OAI-SearchBot to appear in ChatGPT search and still disallow GPTBot for training. Anthropic says disabling Claude-User stops Claude from fetching your content in reply to a user's question, which can reduce your visibility in Claude's answers.

A common setup: keep the AI Bots ruleset on Allow (or Log), then use robots.txt to opt out of training only:

Example

# Allow search and user-requested fetches
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training (optional; remove if you're happy to be used for training)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

If you'd rather block training at the edge, use custom rules aimed at the specific user-agents, not the all-AI switch.

Step 4: Verify the fix

  1. Compare two requests. Fetch the same page with a browser user-agent and with a crawler's, and compare the status codes:

    Example

    curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://www.example.com/

    A 200 for the browser and a 403 or challenge page for the bot means something at the edge is deciding.

  2. Check the firewall logs. With the AI Bots ruleset on Log, the firewall's traffic view shows which bots arrived and what happened to them.
  3. Know the limits. A user-agent test from your laptop is a strong hint, not proof. Real crawlers come from published IP ranges, and a firewall may treat a spoofed user-agent differently from the real bot. OpenAI publishes its IP lists in its crawler docs.
  4. Give it time. OpenAI says a robots.txt change can take about 24 hours to show up in its search systems.

Or run the free check at https://index-doctor.com/check. It does steps 1 and 3 for you and explains the result.

Fix prompt to paste into Cursor, v0 or your AI editor

Example

My site is deployed on Vercel. I want AI search and answer bots to read my public pages, and I want to opt out of model training only.
1. Create or update robots.txt (public/robots.txt, or app/robots.ts in Next.js) so it:
   - allows OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot on all public pages
   - disallows GPTBot and ClaudeBot
   - keeps "User-agent: *  Allow: /" and keeps my existing Disallow rules for private paths (/api, /admin, /dashboard)
   - lists my sitemap URL
2. Don't change any firewall settings in code. Tell me which Vercel Firewall settings to check by hand: the AI Bots managed ruleset (should be Allow or Log, not Deny), the Bot Protection ruleset, and any custom rule that matches bot user-agents.
3. Show me the final robots.txt.

FAQ

Does Vercel block AI bots by default?

No. Vercel's docs say the AI Bots managed ruleset is inactive by default. Someone on the team has to switch it to Log or Deny.

I blocked GPTBot. Am I out of ChatGPT search?

Not because of that. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent. Blocking OAI-SearchBot is what keeps you out of ChatGPT search answers.

Does Bot Protection affect Googlebot?

Vercel has a separate verified-bots system for well-known crawlers. Check Vercel's bot management docs for how your plan handles verified bots, and confirm in the firewall logs that Googlebot requests get through. Then use Search Console's URL Inspection to see what Google gets.

What about AI agents that browse for users?

Vercel's knowledge base says Deny mode also affects how AI agents can access your site, giving ChatGPT-Operator as an example.

How long until AI tools see the change?

For ChatGPT search, OpenAI says about 24 hours after a robots.txt update. Firewall changes apply on your side right away, but each crawler revisits on its own schedule.

Related guides

Sources (fetched 21 Oct 2026)

All six sources were re-fetched on 21 Oct 2026, including the Vercel KB line on ChatGPT-Operator and Anthropic's Claude-SearchBot description.