GPTBot
How to allow or block ChatGPT (GPTBot)
GPTBot is OpenAI's training crawler. It is not the crawler that fetches pages for ChatGPT search.
OpenAI documents three different crawlers. GPTBot collects pages that may be used to train models. OAI-SearchBot fetches pages that can be cited in ChatGPT search. ChatGPT-User fetches a page when a person asks ChatGPT to open it.
Blocking GPTBot does not by itself remove you from ChatGPT search, and allowing GPTBot does not put you into search. Use the matching group.
OpenAI says these crawlers respect robots.txt. A group named GPTBot applies to that crawler. The examples below are rules you can put in your own file. They are not a copy of OpenAI's robots.txt.
A firewall rule can still block the crawler even when robots.txt allows it. robots.txt is voluntary. A 403 is not.
What to check
- Decide whether you want training (GPTBot) and search citations (OAI-SearchBot) to be the same choice.
- Add a group for each token you want to treat differently.
- Fetch the URL and confirm the response is 200, not only that robots.txt says Allow.
Fix prompt
Example prompt
Add a robots.txt group for GPTBot. Disallow the whole site if training crawls should stop. Leave OAI-SearchBot allowed if ChatGPT search should still fetch the page. Do not call an indexing API.
Check a URL · Crawlers · Glossary
Questions
- Which token does ChatGPT search use?
- OAI-SearchBot. GPTBot is the training crawler. See OpenAI's crawler docs for the current user-agent strings.