robots.txt
robots.txt explained for beginners
How groups, Allow, and Disallow decide which crawler may fetch a path.
robots.txt sits at the root of the host, for example https://example.com/robots.txt. It is a list of groups. Each group starts with one or more User-agent lines and then Allow or Disallow rules.
A crawler uses the group that names it. Googlebot uses the Googlebot group if one exists, and otherwise the * group. GPTBot does not read the Googlebot group. Each bot you care about needs its own group, or it falls through to *.
The longest matching path wins. Allow: /blog and Disallow: / can leave /blog open. Disallow: / blocks the whole host for that agent. A missing file means allow. A 5xx on robots.txt can make Google slow down.
robots.txt controls crawling, not indexing by itself, and it is not a privacy lock. Anyone can read the file. Use noindex, or authentication, for pages that must stay private.
What to check
- Open /robots.txt on the same host as the page.
- Find the group for the crawler, then apply the longest match.
- Do not paste a disallow-all rule onto a site you want in Google.
Fix prompt
Example prompt
Explain robots.txt for this host. Show the group that matches Googlebot and the groups for GPTBot, ClaudeBot, and PerplexityBot. Say which paths are blocked. Do not call an indexing API.
Check a URL · Crawlers · Glossary
Questions
- Does * apply to every bot?
- It applies to bots that do not have a more specific group. A Googlebot group replaces * for Googlebot only.