What a Crawler Sees

Two questions that get run together

When a page is not showing up in search, almost everybody asks one question: why is Google ignoring my page. There are really two, and they have nothing to do with each other.

The first is permission. Is this page allowed in the index at all. That is decided by a handful of signals you can read in a few seconds once you know where they live, and if the answer is no, then nothing else about the page matters. You can write the best article in your industry on a page carrying a noindex tag and it will never appear.

The second is legibility. Assuming the page is allowed, is there anything on it to read. Not whether it looks right to you in a browser, but whether the words are present in the file your server sends before any script runs.

A page can pass one and fail the other, and from the outside the result looks identical. You search for your own business, you are not there, and the tool you reach for tells you something vague about authority.

Why it has to be view source and not inspect

This is the part worth getting right, because the two look similar and they answer different questions.

View source shows the HTML your server sent. Inspect shows the document as it stands now, after every script on the page has finished rearranging it. On a plain site they are nearly the same. On a site built with a modern front end framework they can be wildly different: an empty shell in one, a full page in the other.

Google will usually render the JavaScript and get to the full version in the end, on its own schedule and at a cost in crawl budget. Most of the other things reading the web right now will not. They take the HTML, read it, and move on. If your writing is not in there, those systems saw an empty page, and increasingly those systems are what people ask before they ever open a search engine.

That is the whole reason the instruction above is specific. Paste the inspector output and this tool will tell you everything is fine, because for the browser you are sitting in front of, everything is. What a model can see goes into what that gap costs.

A noindex tag is rarely a decision anybody made

When this tool reports that a page is asking to be left out of the index, the natural assumption is that somebody chose that. Occasionally somebody did. More often it arrived by accident, and there are a few well worn routes.

The most common is a staging flag that survived the move to production. Nearly every platform has a setting that hides a site while it is being built, and nearly every platform makes it easy to forget.

The more interesting one is a rule meant for part of a site that ends up applying to all of it. Somebody writes a condition to keep an empty section out of search results. The condition is true for the section, and it is also true everywhere else, because nothing in it says which page it is supposed to be about. The site keeps rendering perfectly. Every page quietly asks to be removed. Nothing in an admin screen shows it, nothing fails, and the first symptom is a traffic line going the wrong way weeks later. Site wide rule, page level job is about that class of mistake and how to catch it.

Which is the argument for checking the rendered output rather than the setting. A setting tells you what somebody intended. The tag in the HTML tells you what is actually being sent, and those two part company more often than anyone expects.

What this cannot see

Two things, and either can block a page whose HTML looks perfect.

The first is an X-Robots-Tag header. It does the same job as the robots meta tag and it travels in the HTTP response rather than the document, so it never appears in view source. If every check here is clear and the page still is not indexed, that header is the first place to look.

The second is your robots.txt, which can stop a crawler before it ever requests the page. A page blocked there can still get indexed on the strength of links alone, which produces the confusing result where your URL appears in search with no description under it.

There is also a third thing, which is not a limitation of the tool so much as of the whole category. None of this tells you whether a page deserves to rank. It tells you whether the page is eligible and legible. Those have to be true first, and when they are not, every hour spent on anything else is wasted.

When everything here comes back clean

Then the problem is somewhere else, and that is useful to know rather than disappointing. The usual next answer is that the page is simply new and has not been crawled yet, which looks like a technical fault and is not one. Discovered, currently not indexed covers what that status actually means and what moves it.

After that it is the ordinary work: whether anything links to the page, whether it answers a question somebody is asking, and whether the title earns the click. The search result preview handles the last of those.