Frequently asked questions
Does robots.txt stop a page being indexed?
Not on its own. robots.txt stops Google crawling a page, but the address can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag.
What does noindex do?
A noindex robots meta tag or X-Robots-Tag header tells Google to keep the page out of its results. Google has to be able to crawl the page to see the tag.
What is a canonical tag?
It tells search engines which address is the main version of a page. If it points somewhere else, Google will usually index that other address instead.
My page is indexable but not in Google. Why?
Being allowed in isn't the same as being chosen. New pages take time, and pages with few links or thin content may be crawled and left out. Links from other sites help Google find a page and decide it is worth keeping.