Why Google Indexes Pages Blocked in Robots.txt

Because robots.txt controls crawling, not indexing. A disallow stops Google fetching the URL; it doesn't stop Google listing it. If other pages link to that URL, Google can index it from the links alone — showing the bare address with "No information is available for this page," because it was never allowed to look inside.

Search Console reports this as "Indexed, though blocked by robots.txt" in the Pages report. It's one of the few statuses that is simultaneously a warning, a non-issue, and a sign of a serious misconfiguration — depending on which URLs are in the list.

What Google is actually doing

A crawler discovering a link to example.com/private-thing/ learns three things without ever requesting the page: the URL exists, the anchor text people use for it, and how many places link to it. That's enough to create an index entry. The disallow only prevents the fourth step — fetching the content.

So the page enters the index as a shell: URL, sometimes a title derived from anchor text or external references, no description. It can rank for a navigational query, particularly a branded one, which is why the problem usually surfaces as "why is this ugly URL showing up when I search my own company name."

The corollary is the part that traps people: Google cannot see a noindex on a page it isn't allowed to crawl. Adding both directives at once is self-cancelling. The block prevents the fetch that would deliver the removal instruction, so the URL stays indexed indefinitely. This is far and away the most common reason a page refuses to disappear.

The three ways it happens

Scenario	What's in the report	Is it a problem?
Blocked after it was already indexed	Real content pages you deliberately disallowed	Yes — the block froze them in the index
Discovered only through external links	URLs you never linked to internally	Usually not, unless they're sensitive or embarrassing
Parameter, faceted and internal-search URLs	`?sort=`, `?s=`, `/filter/` variants	Usually not — cosmetic, not harmful

The second and third cases account for most of what you'll find, and most of it needs no action. A handful of parameter URLs listed without descriptions dilutes nothing and costs nothing. Clearing the report to zero is not the goal.

The first case is the one to act on. A path you blocked in order to get it out of Google has, by blocking it, guaranteed it stays in.

Decide whether it's actually a problem

Open the report, export the URL list, and sort into three buckets:

Must come out. Staging copies, internal tools, customer-facing URLs you're contractually or legally obliged to remove, anything returning content you'd be uncomfortable seeing in a SERP screenshot. These get the full fix below.
Should come out eventually. Thin duplicates, old campaign landing pages, print variants. Worth fixing on the next cleanup pass; not urgent.
Leave alone. Parameter noise, faceted variants, internal search results. These are exactly why the disallow exists — it's doing its job of saving crawl requests. Blocking them was the right call for crawl budget; a stray listing is the accepted side effect.

A useful sanity check on the "leave alone" bucket: search the exact URL with site: and look at what appears. If the result is a bare URL nobody would click, it is not costing you anything.

The fix, in the order that works

To genuinely remove a URL, Google has to be able to fetch it. That means temporarily giving up the block.

Remove the Disallow rule for that path in robots.txt. Counter-intuitive and non-negotiable — nothing else in this list works while the rule stands.
Add the removal directive to the URL itself. Either <meta name="robots" content="noindex"> in the <head>, or X-Robots-Tag: noindex in the HTTP response for non-HTML files. What is the X-Robots-Tag header covers the header version, which is the right choice for PDFs and bulk patterns.
Request a recrawl of the affected URLs via URL Inspection in Search Console. On a large batch, submit a sitemap containing just those URLs to speed discovery — a sitemap is a legitimate way to say "please look at these," even for pages you want removed.
Wait for the recrawl. Nothing changes until Googlebot actually fetches the page and reads the directive. Days for a well-crawled site, weeks for a neglected one.
Only then re-add the disallow, and only if you actually need to. Once the page is out of the index, re-blocking it stops future crawl requests. If it's a handful of URLs, leaving them crawlable-and-noindexed is simpler and less likely to be undone by a future you who doesn't remember the sequence.

When you need it gone today, use Search Console's Removals tool. It hides the URL from Google results for roughly six months, which buys time for steps 1–4 to take effect. It is a temporary suppression, not a removal — if the underlying directive isn't fixed, the URL returns when it expires.

If the content should not be public at all, none of this is the right answer. Put it behind authentication or return a 404/410. robots.txt is a publicly readable file, and publishing Disallow: /internal-pricing-2027/ advertises the path to anyone curious enough to look.

What it means for links pointing at the URL

This matters if you've earned or traded links to a page that's now blocked.

Google never fetches a disallowed URL, so it never sees the links on it — those pass nothing, and they pass less than the links on a merely noindexed page, which at least gets crawled for a while. Links pointing at the blocked URL are a different story: they're still counted as links to that address, but the address holds no content Google can evaluate and isn't a candidate for most queries, so the value largely sits idle.

The practical move is the same one you'd make for an orphan page: pick the indexable page the links should have pointed at, 301 the blocked URL to it once the block is lifted, and you consolidate the signal instead of stranding it. What is a 301 redirect covers how much carries across.

It's also an argument for controlling where your links land. When you trade links directly with another site owner, both placements sit on live, crawlable, in-content pages, and both stay verified over time — Backlinkster checks each swap by code rather than taking a screenshot at handover. Free accounts get five verified swaps a month; see the plans.

Frequently asked questions

Why is Google indexing a page I blocked in robots.txt? Because robots.txt blocks crawling, not indexing. Google can index a URL it has never fetched if other pages link to it, using the links and anchor text as its only information. The listing appears without a description, usually noted as "No information is available for this page."

How do I remove a page that's blocked by robots.txt? Remove the disallow rule, add a noindex meta tag or X-Robots-Tag header to the page, request a recrawl in Search Console, and wait for Google to fetch it. The block must come off first — Google cannot read a removal instruction on a URL it isn't permitted to request.

Is "Indexed, though blocked by robots.txt" bad? Only if the URLs in it are pages you actually wanted removed, or content that shouldn't be public. Parameter URLs and internal search results appearing there is normal and harmless. Check what's in the list before treating the status itself as an error.

Does a robots.txt block pass any link value? No. Google never requests the page, so it never sees the outbound links on it. A link placed on a disallowed page is inert — see do links on noindexed pages pass SEO value for the closely related noindex case.

How long does it take for a page to drop out of the index after adding noindex? Typically days to a few weeks, governed entirely by how often Google recrawls that URL. Requesting indexing on the URL speeds up the fetch. Search Console's Removals tool suppresses the listing immediately for about six months while the directive takes effect.

Should I use robots.txt or noindex? Use noindex when you want a page out of search results. Use Disallow when you want to stop crawlers spending requests on URLs you don't care about. Using both on the same URL achieves neither — it keeps the page indexed and hides the instruction that would have removed it.

The bottom line

A disallow is a request not to look, not a request to forget. If a URL is in the index and you want it out, the block has to come off long enough for Google to read a noindex — that's the whole fix, and the order is what people get wrong. Everything else in the report is usually noise: check which bucket each URL falls into before you start editing anything.