Googlebot
Crawling pages for Google Search.
- Who runs it
- robots.txt token
- Googlebot
- User-agent
- Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Google publishes this token. Googlebot also uses other variants, including smartphone crawling. The robots.txt token stays Googlebot.
Googlebot is the crawler Google uses to fetch pages for Search. If it cannot fetch a URL, that URL is not a normal search result.
In robots.txt, the group named Googlebot applies to it. If that group is missing, Googlebot uses the * group. A rule for GPTBot or another bot does not apply to Googlebot.
Googlebot obeys robots.txt for crawling. A disallow stops the fetch. It does not, by itself, delete a URL that other sites already link to. noindex, on a page Google is allowed to fetch, is how you ask for the URL to stay out of the index.
A firewall can still return 403 or a challenge to Googlebot after robots.txt says Allow. The free check requests the URL with Google's published user-agent. That request does not come from Google's network, so confirm a block in your firewall if the result matters.
Google's Googlebot documentation
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: Googlebot Allow: /
Block
User-agent: Googlebot Disallow: /
Related guides
Other crawlers
Questions
- Does blocking Googlebot remove a page from Google?
- It stops future crawls. A URL that is already known can linger. noindex, on a page Google can fetch, is the direct opt-out.