Common Crawl
CCBot
Building the Common Crawl open corpus of the web.
- Who runs it
- Common Crawl
- robots.txt token
- CCBot
- User-agent
- CCBot/2.0 (+https://commoncrawl.org/faq/)
Common Crawl publishes CCBot and states that it follows robots.txt. Their CCBot page is the source if the version string changes.
CCBot collects pages for Common Crawl, a public dataset. Other organizations train models on that dataset, which is why sites that want to limit training often disallow CCBot even when they allow search crawlers.
Common Crawl documents that CCBot respects robots.txt. The token is CCBot.
Disallowing CCBot does not remove copies already published in an older crawl. It stops future fetches by this agent.
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: CCBot Allow: /
Block
User-agent: CCBot Disallow: /
Related guides
Other crawlers
Questions
- Is CCBot a search engine?
- No. It builds a corpus. Search ranking is a separate system.