noindex
noindex explained
What noindex does, where it hides, and why robots.txt cannot replace it.
noindex tells a crawler that fetched the page not to put that URL in its index. Google looks for it in a meta robots tag and in the X-Robots-Tag HTTP header. Other engines have their own names, but the Google tag is the one that changes Google Search.
The tag has to be in the response Google downloads. A tag added by JavaScript after the page loads can be missed. A tag in a shared layout can mark every page, including ones you wanted indexed.
robots.txt does not support noindex. If you disallow the URL, Google may not see the tag at all, and a URL that other sites link to can still be listed without a snippet. To keep a page out, allow the crawl and send noindex. To put a page in, remove noindex and allow the crawl.
Staging sites, thank-you pages, and internal search should keep noindex. Marketing pages should not.
What to check
- Search the raw HTML for name="robots" and content that includes noindex.
- Check response headers for X-Robots-Tag.
- Look at the layout or template, not only the page file.
Fix prompt
Example prompt
Remove noindex from the server HTML and from X-Robots-Tag on this URL if it should be indexed. Keep noindex on pages that should stay out. Do not call an indexing API.
Check a URL · Crawlers · Glossary
Questions
- I removed noindex. Why is the page still missing?
- Google has to crawl the URL again and see the tag gone. The old decision stays until that fetch.