What Is a Robots.txt File?

A robots.txt file is a plain text file at the root of your domain that tells crawlers which parts of your site they're allowed to request. It controls crawling, not indexing — a page blocked in robots.txt can still show up in Google if other sites link to it. Every site should have one, even if it allows everything.

Where it lives and what it looks like

It must sit at exactly one location: https://example.com/robots.txt. Not in a subfolder, not renamed. One file per host and protocol — https://blog.example.com/robots.txt is a separate file governing that subdomain.

A minimal, sensible file:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

That's the whole format. Four directives cover almost every real use:

Directive	What it does
`User-agent:`	Names the crawler the following rules apply to. `*` means all crawlers that don't have their own block.
`Disallow:`	A path prefix the crawler must not request. `Disallow: /` blocks the entire site.
`Allow:`	Carves an exception out of a broader `Disallow`. Useful for `Disallow: /assets/` + `Allow: /assets/public/`.
`Sitemap:`	An absolute URL to your XML sitemap. Not tied to any user-agent — put it anywhere in the file.

Two rules that trip people up: paths are case-sensitive, and a crawler obeys only the single most specific matching User-agent block. If you write a User-agent: Googlebot block, Googlebot ignores your User-agent: * block entirely — including any rules you assumed were universal.

Crawling is not indexing (the mistake that costs the most)

This is the single most misunderstood thing about the file, and it causes real damage in both directions.

Blocking a URL does not remove it from search results. If other sites link to a blocked page, Google can index the URL without ever fetching it — producing the bare listing with "No information is available for this page." The page is in the index; Google just can't see what's on it.

Worse, because Google can't crawl the page, it can't see a noindex tag on it either. Adding noindex to a page and blocking it in robots.txt at the same time is self-defeating: the block prevents the very fetch that would deliver the removal instruction.

The rule is simple:

Want a page out of search results? Use noindex and allow crawling so Google can read it.
Want to stop a crawler wasting requests on a URL you don't care about? Use Disallow.
Want a page genuinely private? Neither. Use authentication — robots.txt is a public file that anyone can read, and it's a request that only well-behaved crawlers honour.

That last point is worth repeating: publishing Disallow: /internal-pricing-tool/ announces the existence of /internal-pricing-tool/ to the entire internet. Never use robots.txt as a security measure.

What to block, and what never to block

Genuinely worth blocking:

Internal search results pages (/?s=, /search)
Faceted-navigation parameter URLs on large ecommerce or listing sites
Cart, checkout, and account pages
Admin and login paths
Staging or duplicate environments on a separate host

Never block:

CSS and JavaScript files. Google renders pages before judging them. Block the assets and it sees a broken layout, which affects both quality assessment and mobile usability.
Your images, if you want them in image search.
Anything you're actively trying to get indexed — an obvious statement that a surprising number of sites get wrong after a migration.

On most small sites the correct robots.txt blocks almost nothing. Blocking is a large-site tool for controlling crawl budget, and applying large-site tactics to a 200-page blog usually creates problems rather than solving them.

Naming the AI crawlers

Robots.txt has quietly become the file that decides whether ChatGPT, Claude, Perplexity and Gemini can read your site at all. Each runs its own agent, and each respects robots.txt.

User-agent	Operator	What it's for
`GPTBot`	OpenAI	Crawling for OpenAI's models and search index
`OAI-SearchBot`	OpenAI	ChatGPT search results and citations
`ChatGPT-User`	OpenAI	Live fetches when a user's prompt triggers browsing
`ClaudeBot`	Anthropic	Crawling for Claude
`PerplexityBot`	Perplexity	Indexing for Perplexity answers
`Google-Extended`	Google	Gemini apps and Vertex AI grounding — not Google Search
`CCBot`	Common Crawl	Open crawl dataset used to train many models

Two things are worth knowing before you touch these.

Blocking them means you cannot be cited. If GPTBot is disallowed, ChatGPT has nothing of yours to quote. There is a legitimate business case for blocking — plenty of publishers have made it deliberately — but it should be a decision, not a default inherited from a template.

Google-Extended is not a Google Search control. Disallowing it opts your content out of training Gemini apps; it does not remove you from Google Search or from AI Overviews, which are governed by ordinary Googlebot access. Blocking Googlebot to stay out of AI Overviews would remove you from Google search results entirely.

A middle path some sites take: allow the search-and-cite agents so you remain quotable, and block CCBot, which feeds training datasets without ever sending a citation back. Whichever you pick, being explicit is better than relying on a wildcard, because the rules are easier to audit a year later when nobody remembers what was intended.

Getting crawler access right is table stakes, though — it's permission, not persuasion. What actually earns a citation is a page that answers the question cleanly and a web that references you elsewhere, which is the subject of how to do SEO for ChatGPT. That second half is why in-content links from real sites still matter in the AI era, and it's the problem Backlinkster solves — a verified 1-for-1 swap with a site owner in a related niche, each link placed inside genuine content rather than a footer.

Mistakes that deindex a site

Nearly every catastrophic robots.txt incident is one of these:

Disallow: / shipped from staging. The classic. A development environment's block-everything file goes live with the deploy. Check robots.txt immediately after any launch or migration.
Blocking a path that contains the pages you want indexed — Disallow: /blog also blocks /blog-archive and /blogging-tips, because it's a prefix match, not a folder match.
A crawler-specific block that silently cancels the wildcard rules. Add User-agent: Googlebot with one line under it and every User-agent: * rule stops applying to Googlebot.
Using robots.txt to fix duplicate content. It doesn't consolidate anything. Blocked duplicates keep whatever links point at them and pass nothing on. A canonical tag is the right tool.
Blocking a redirected URL, which stops Google fetching it and therefore stops it ever seeing the redirect.

How to test it

Never edit this file blind:

Search Console → Settings → robots.txt shows the fetched file, when it was last read, and any parse errors.
URL Inspection on a specific page tells you whether that exact URL is blocked, which is the answer you usually actually want.
Load yoursite.com/robots.txt in a browser. Confirm it returns 200 and plain text — a 404 means all crawlers assume full access, and a 500 can cause Google to pause crawling the site entirely.
Cross-check the syntax against Google's robots.txt specification, which documents exactly how matching, wildcards and precedence are resolved.

Frequently asked questions

Do I need a robots.txt file? Not strictly — a missing file returns 404 and crawlers proceed as if everything is allowed. But having one lets you declare your sitemap, set crawler-specific rules, and avoid the risk that a server misconfiguration returns a 5xx for that path, which Google may treat as a temporary instruction to stop crawling.

Does robots.txt remove a page from Google? No. It stops Google fetching the page, but a blocked URL can still be indexed from external links, showing with no description. To remove a page, allow crawling and use a noindex tag, or use the Removals tool in Search Console for something urgent.

Should I block AI crawlers like GPTBot? Only if you've decided you don't want to appear in AI answers. Blocking them prevents both training and citation. Most sites trying to grow visibility should allow the search-and-cite agents; blocking CCBot alone is the narrower option if training use is the concern.

Does robots.txt affect my rankings? Only through what it blocks. The file itself carries no ranking weight, but blocking CSS, JavaScript, or pages you want to rank has a direct and often severe effect.

Can I use wildcards in robots.txt? Google and Bing support * for any sequence of characters and $ to anchor the end of a URL — so Disallow: /*?sort= blocks any URL containing that parameter. These are extensions, not part of the original standard, so less common crawlers may ignore them.

The bottom line

Robots.txt is a small file with an outsized ability to break things. Keep it short, block only what genuinely wastes crawler requests, never use it to hide anything or to try to deindex a page, and check it after every deployment. The one modern addition worth making is a deliberate decision about the AI crawlers — because that file is now what decides whether an answer engine can quote you, and the default you inherited from a template probably wasn't chosen for that.