What Is an AI Crawler? (And Should You Block It?)

An AI crawler is an automated bot that fetches web pages for an AI company: to gather training data, to build a search index the assistant cites from, or to read one page a user asked about. Block the training-only crawlers if you object to training, but allow the search ones — blocking those means that assistant can never cite you.

The three types, and why the distinction matters

Almost every bad decision in this area comes from treating "AI bots" as one category. They do three different jobs, and the trade-off is different for each.

Type	What it does	Blocking costs you
Training crawler	Collects pages to train future models	Nothing in traffic terms — no citation, no referral
Search crawler	Builds the index the assistant retrieves and cites from	Your ability to be cited by that assistant, permanently
User-triggered fetcher	Fetches one page because a user pasted a link or asked about it	Your page failing to load for a user who explicitly asked for it

The headline is that the second and third categories send you visibility and sometimes clicks. The first sends nothing back. A blanket Disallow aimed at "AI" hits all three, which is why sites that panic-blocked in 2023 and 2024 later found themselves absent from ChatGPT and Perplexity answers while their competitors were being named.

Several companies split these across separate user-agents precisely so publishers can make the distinction. OpenAI, for example, runs GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for user-triggered fetches — three tokens, three decisions.

The user-agents worth knowing

These are the tokens you write in robots.txt. Names change and new ones appear, so treat this as a starting list rather than a permanent one, and check each vendor's published documentation before making a decision that matters.

User-agent	Operator	Job
`GPTBot`	OpenAI	Training data collection
`OAI-SearchBot`	OpenAI	Search indexing for ChatGPT citations
`ChatGPT-User`	OpenAI	User-triggered page fetch
`ClaudeBot`	Anthropic	Crawling for Claude
`Claude-User`	Anthropic	User-triggered page fetch
`PerplexityBot`	Perplexity	Search indexing for citations
`Google-Extended`	Google	Gemini app / Vertex grounding usage — not Search
`Googlebot`	Google	Core Search, including AI Overviews
`Bingbot`	Microsoft	Bing index, which feeds Copilot
`Applebot-Extended`	Apple	AI training usage of already-crawled data
`CCBot`	Common Crawl	Open dataset used to train many models
`Bytespider`	ByteDance	Training data collection
`Meta-ExternalAgent`	Meta	Training data collection

Two entries in that table deserve their own note, because they are the two most commonly misread.

Google-Extended does not control AI Overviews. It governs whether your content is used for Gemini app responses and Vertex AI grounding. AI Overviews are served through ordinary Google Search, crawled by Googlebot. Disallowing Google-Extended will not remove you from AI Overviews, and allowing it will not get you in. If you want out of AI Overviews specifically, the lever is the snippet controls — covered in how do I show up in Google AI Overviews.

Applebot-Extended is not a crawler at all. It is a usage-permission token: Applebot crawls once for Search, and Applebot-Extended tells Apple whether that already-collected content may be used for AI training. Blocking it changes nothing about crawl load.

Should you block them? A decision, not a default

Work through four questions in order.

1. Do you want to be cited by AI assistants? If yes, allow the search crawlers — OAI-SearchBot, PerplexityBot, ClaudeBot, and Googlebot and Bingbot as always. There is no way to be cited by a system that cannot read you. For most businesses this settles it: the assistants are a discovery channel, and opting out of a discovery channel to protect content you publish for free rarely nets out positive.

2. Do you object to training? This is a legitimate position, and it is the one the training-only tokens exist for. Blocking GPTBot, CCBot, Bytespider, Meta-ExternalAgent and Applebot-Extended while allowing the search bots is the common middle ground, and it is the configuration most publishers landed on.

3. Is crawl volume actually hurting you? For a small site on ordinary hosting, AI crawler traffic is usually a rounding error. For a large site, or one on metered bandwidth, it can be real. Check your logs before assuming either way — the answer varies enormously by site. If load is the problem, Crawl-delay and rate limiting at the edge are better instruments than a blanket block.

4. Is your content the product? A paywalled archive, a proprietary dataset, a subscription publication — these are the cases where blocking broadly is straightforwardly correct. If people pay for the content, do not give it away to a summariser.

Everyone else: allow search, block training, move on.

A worked robots.txt

# Search crawlers — allow, these can cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Training-only crawlers — block
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Everything else
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

One rule catches people out constantly: a crawler obeys only the single most specific User-agent block that matches it. If you write a User-agent: GPTBot block, GPTBot ignores your User-agent: * rules entirely — including the Disallow: /admin/ you assumed was universal. Repeat any rules that genuinely need to apply to everyone. The full syntax rules are in what is a robots.txt file.

What robots.txt can and cannot do

Be clear-eyed about the limits, because this file is a request, not a wall.

It is voluntary. Well-behaved crawlers honour it. Scrapers that ignore it do not, and there is no enforcement mechanism in the protocol. Blocking at the CDN or firewall level is the enforceable version.
It is public. Anyone can read yoursite.com/robots.txt, including the list of paths you would rather nobody visited.
It does not undo the past. Blocking a training crawler today does nothing about pages already collected for models that have already shipped.
It controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it.

How to see who is actually visiting

You do not have to guess. Filter your server access logs by user-agent string and you get the real picture — which bots visit, how often, and which pages they take.

If you are on Cloudflare, Fastly or a similar edge provider, the dashboard usually breaks out AI bot traffic directly, and most now offer one-click controls for the well-known tokens. That is a more reliable enforcement point than robots.txt, because it applies whether or not the bot chooses to cooperate.

Set a reminder to review this once or twice a year. New crawlers appear, tokens get renamed, and a list configured in 2024 is already out of date.

The part that matters more than the blocking

Allowing a crawler makes you eligible to be cited. It does not make you likely to be. Eligibility is a checkbox; citation is earned by having a page that answers the question cleanly and a brand the wider web already talks about.

The content half is covered in how to do SEO for ChatGPT — answer-first passages, real FAQ pairs, specifics rather than filler. The authority half is the one people skip: AI systems lean heavily on corroboration, and a site nobody links to or discusses does not get named regardless of what its robots.txt permits. Real editorial links from real sites are still the foundation. Backlinkster arranges them as one-for-one in-content swaps between site owners, verified live by code — five a month free, with plans from $19.

Frequently asked questions

What is an AI crawler? An automated bot that fetches web pages for an AI company. Some collect training data for future models, some build the search index an assistant cites from, and some fetch a single page on demand when a user asks about it.

Should I block AI crawlers in robots.txt? Usually only the training-only ones. Blocking GPTBot, CCBot and Bytespider while allowing OAI-SearchBot, PerplexityBot and ClaudeBot keeps you eligible for citations while opting out of model training.

Does blocking GPTBot stop ChatGPT from citing me? No. GPTBot is OpenAI's training crawler; citations in ChatGPT's search results come via OAI-SearchBot. They are separate tokens, and you can allow one while blocking the other.

Do AI crawlers hurt my server or SEO? They do not affect rankings. Crawl volume can matter on large or bandwidth-metered sites, so check your logs before acting. Rate limiting at the CDN handles load more precisely than a blanket block.

Does robots.txt actually stop AI crawlers? Only cooperative ones. The protocol is voluntary and has no enforcement. To genuinely block a bot that ignores it, block by user-agent or IP at your CDN or firewall.

Will blocking AI crawlers protect my content from being used? Partly and prospectively. It has no effect on content already collected for models that have shipped, and it does not prevent your content being described second-hand from other sites that quote you.

Is Google-Extended the same as blocking AI Overviews? No. Google-Extended covers Gemini app and Vertex AI grounding. AI Overviews run on ordinary Google Search crawling, so this token has no effect on them.

The bottom line

AI crawlers are not one thing, and the useful question is not "block or allow" but "block which." Training crawlers take without returning anything — block them if you object to the use. Search crawlers are how ChatGPT, Claude and Perplexity find something to cite — block them and you have removed yourself from a channel your competitors are still in. Split the difference on the user-agent line, check your logs once a year, and spend the saved effort on being worth citing.