- Crawl
- A crawler fetches a URL and reads the response. Crawling is not the same as indexing. A page can be fetched and still left out of the index.
- Index
- The set of URLs a search engine keeps so it can show them as results. Google decides this after a crawl. You cannot push a URL in by calling an API from this product.
- Indexable
- The public signals allow indexing: the URL returns a successful page, Googlebot is allowed to fetch it, and the page does not say noindex. It does not mean the URL is already in Google.
- User-agent
- The name a client sends with a request. Crawlers publish a string, and robots.txt matches the token inside it, such as Googlebot or GPTBot.
- noindex
- An instruction, in a meta robots tag or the X-Robots-Tag header, that tells a crawler not to keep the URL in the index. The crawler has to be allowed to fetch the page to see it.guides/noindex explained
- Meta robots
- A tag in the HTML head, for example a robots tag whose content includes noindex or nofollow. It only counts if it is in the HTML the crawler downloads.
- X-Robots-Tag
- The same kind of instruction as meta robots, sent as an HTTP header instead of a tag. A header can noindex a PDF or another non-HTML response.
- Canonical
- The URL you declare as the preferred copy of a page, usually with a rel=canonical link. Google treats it as a hint. If it points at a different host or at a redirect, the URL you wanted can stay out.guides/duplicate without user selected canonical
- Sitemap
- A list of URLs you would like crawled, often linked from robots.txt. It helps discovery. It does not force indexing, and it should not list redirects, errors, or pages with noindex.
- H1
- The main heading in the HTML. One clear H1 that matches the page helps a crawler see what the URL is about. A heading drawn only after JavaScript runs may be missing from the first response.
- Server HTML
- The document returned by the first response, before client-side JavaScript fills it in. Crawlers that do not run your app only see this.
- Client-rendered shell
- A first response that is mostly an empty container. The words appear later in the browser. Google may render JavaScript, and it may still index nothing worth quoting.guides/is my lovable site visible to google
- Soft 404
- A URL that returns HTTP 200 but looks empty or like a not-found page. Google can drop it. A real missing URL should return 404.guides/soft 404
- URL Inspection
- A read-only Search Console method that returns Google's coverage state for one URL. It uses a weekly budget on paid plans. It does not ask Google to index the URL.
- Search Console
- Google's site for coverage, sitemaps, and search analytics. IndexDoctor can read it only after you connect it, and only with read-only access. Sign-in and paying do not require it.
- Crawl budget
- A loose name for how many URLs Google is willing to fetch on a host. Redirect chains, errors, and huge sitemaps of thin URLs spend it. It is not a number IndexDoctor sets.
- llms.txt
- A plain-text file some projects publish at /llms.txt as a map of pages for assistants. It does not block crawlers and Google does not use it as an indexing signal.guides/llms txt explainedllms.txt