llms.txt
llms.txt explained
A plain-text index some AI tools read. It does not replace robots.txt or a sitemap.
llms.txt is a proposed plain-text file at the root of a site, /llms.txt. The idea is a short map of the pages an assistant should read: what the product is, and links to the canonical docs. It is not a standard Google requires, and Google does not use it for indexing.
It does not block crawlers. robots.txt does that. It does not list every URL. A sitemap does that. A useful llms.txt is short, uses real URLs, and points at pages whose HTML already contains the answer.
IndexDoctor publishes one at /llms.txt so assistants can find the checker, the pricing, and these guides. You can do the same if you want a single page that lists your docs. Keep it updated when URLs move. An invented page in the file is worse than a short file.
What to check
- Put the file at https://your-host/llms.txt, not in a subfolder.
- Link only to URLs that return 200 and are allowed in robots.txt.
- Do not treat the file as a privacy control.
Fix prompt
Example prompt
Add an llms.txt at the site root with a one-line summary and links to the public docs. Do not list private URLs. Do not use it instead of robots.txt. Do not call an indexing API.
Check a URL · Crawlers · Glossary
Questions
- Will llms.txt get me into ChatGPT?
- No. Crawlers still have to be allowed to fetch the pages, and the pages still need text in the HTML. The file is only a map.