Blocked by robots.txt
Blocked by robots.txt
Googlebot is not allowed to crawl the path. An index request cannot override that.
robots.txt is a crawl permission. If the group that matches Googlebot disallows the path, Google will not fetch it. A blocked URL can still appear as a bare link if other sites mention it, but it will not be indexed as a normal result.
The longest matching rule wins, and a specific Allow can beat a shorter Disallow. A wildcard group does not override a more specific Googlebot group.
Allow the path for Googlebot if you want it indexed. Leave training bots blocked if that is a separate choice. Those are different groups.
What to check
- Read the Googlebot group, then the * group, and apply the longest match.
- Do not disallow / or the whole site on a production host by accident.
- Remember that robots.txt is not a privacy control. noindex is, and it requires a crawl.
Fix prompt
Example prompt
Googlebot is blocked by robots.txt for this path. Allow the URL in the Googlebot group if it should be crawled. Leave other crawlers blocked only if that is intentional. Do not call an indexing API.
Check a URL · Crawlers · Glossary