One of the easiest ways to create a fake SEO emergency is to block a page in robots.txt And assume that means Google will stop showing it. That is not how Google documents it. In Google’s own words, the file is about controlling crawling, not guaranteeing a URL stays out of search results. If other pages still link to that URL, Google can keep the address in search even without reading the page itself. So yes, the page is blocked. No, that does not mean the problem is solved.

The better move, when you actually want a page gone, is usually noindex. Google says in its guide that a noindex directive works only if Googlebot can crawl the page and see it. That’s the part people trip over. They block the URL in robots.txt, add noindex, and then wonder why the page hangs around like an uninvited guest. Google isn’t ignoring the instruction. Google can’t read an instruction you blocked it from seeing.

For practical cleanup work, this matters most on old promo pages, thin location pages, outdated tag archives, and random CMS junk that nobody has loved in years. Use a meta robots tag or X-Robots-Tag header, as Google outlines, to let the page stay crawlable long enough for Google to process the directive, and then decide whether that URL should stay noindexed, redirect somewhere useful, or return a proper 404. This is boring work, but boring technical SEO work is usually the stuff that keeps traffic drops from turning into month-long Slack therapy.

The bigger lesson is that index control and crawl control are not the same job, even if people keep mashing them together in audits. If a site has thousands of stale URLs, duplicate archives, and no ownership of cleanup, this is exactly the kind of mess where a technical SEO consultant or a done-for-you cleanup team can save a business from making the same mistake at scale. A fancy strategy is nice. Getting the basic instructions right is nicer.