Anthropic
ClaudeBot
Crawling public pages. Anthropic documents it so site owners can allow or block that crawl.
- Who runs it
- Anthropic
- robots.txt token
- ClaudeBot
- User-agent
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +https://claude.com/bot
Anthropic publishes ClaudeBot as the crawler token. Confirm the full user-agent on their article before a firewall rule, and use ClaudeBot in robots.txt.
ClaudeBot is the crawler Anthropic tells site owners how to allow or block. The robots.txt token is ClaudeBot.
It is separate from Claude-User, which Anthropic describes for fetches a person starts. A ClaudeBot rule does not automatically cover Claude-User.
It is also separate from Googlebot. You can allow Search and block this crawl.
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: ClaudeBot Allow: /
Block
User-agent: ClaudeBot Disallow: /
Related guides
Other crawlers
Questions
- Does ClaudeBot use the Googlebot group?
- No. It uses a group named ClaudeBot, or * if that group is missing.