ClaudeBot
How to allow or block ClaudeBot
ClaudeBot is Anthropic's training crawler. Claude-User is a separate fetch a person asks for.
Anthropic documents ClaudeBot as the crawler that collects public pages. Site owners can name ClaudeBot in robots.txt to allow or block that crawl. Anthropic also documents Claude-User for fetches a person triggers inside Claude. Treat them as different tokens.
A block on ClaudeBot does not have to match your Googlebot rule. Many sites allow Google and block training crawlers. That is a normal split.
The examples in the ClaudeBot crawler page are rules for your robots.txt. Confirm the current token on Anthropic's crawler article before you copy it into a firewall, because a firewall matches the full user-agent string and the docs are the source of that string.
What to check
- Add a User-agent: ClaudeBot group if you want a rule that does not affect Googlebot.
- Decide separately about Claude-User if you want to limit user-triggered fetches.
- Check the live response, not only the file.
Fix prompt
Example prompt
Update robots.txt with a ClaudeBot group. Disallow / if Anthropic should not train on this host. Keep Googlebot allowed if the site should stay in Google. Do not call an indexing API.
Check a URL · Crawlers · Glossary
Questions
- Is Claude-SearchBot the same as ClaudeBot?
- No. ClaudeBot is the training crawler Anthropic documents for robots.txt. A search crawler, when one is documented, uses its own token.