Crawlers

Google. Crawling pages for Google Search.

Google. A robots.txt control for whether your pages are used to improve Gemini models. It is not the crawler for Google Search.

Microsoft. Crawling pages for Bing.

OpenAI. Fetching public pages that may be used to train OpenAI models.

OpenAI. Fetching pages that can be shown as results or citations in ChatGPT search.

OpenAI. Fetching a specific page when a person asks ChatGPT to look at it.

Anthropic. Crawling public pages. Anthropic documents it so site owners can allow or block that crawl.

Anthropic. Fetches that a person triggers from Claude, as distinct from the ClaudeBot crawl.

Perplexity. Fetching pages Perplexity uses for answers.

Common Crawl. Building the Common Crawl open corpus of the web.

Apple. A robots.txt control for Apple's use of web content in generative features. It is not the same token as Applebot for search.

ByteDance. A crawler identified as Bytespider in request user-agents. Public documentation of its policy is thin.

Meta. Fetching public pages for Meta's training, as documented separately from the crawler that loads link previews.