OpenAI
OAI-SearchBot
Fetching pages that can be shown as results or citations in ChatGPT search.
- Who runs it
- OpenAI
- robots.txt token
- OAI-SearchBot
- User-agent
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
OpenAI publishes this token for search. Confirm the full string on their bots page before you match it in a firewall.
OAI-SearchBot is how OpenAI fetches pages for ChatGPT search. It is a different robots.txt group from GPTBot.
If you want pages to be fetchable for those answers, allow OAI-SearchBot and put the answer in the HTML. A shell page has nothing to cite.
If you want to stay out of that fetch, disallow OAI-SearchBot. That choice is independent of Googlebot and of GPTBot.
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: OAI-SearchBot Allow: /
Block
User-agent: OAI-SearchBot Disallow: /
Related guides
Other crawlers
Questions
- Is OAI-SearchBot a training crawler?
- OpenAI documents it for search, and documents GPTBot for training. Keep the groups separate.