How Do I Stop AI From Using My Content?

You can't stop it completely, but you can control most of it. Blocking AI crawlers in robots.txt stops cooperative bots, enforcing that block at your CDN stops the rest, and Google's nosnippet directive removes your text from AI Overviews. None of it removes content from models already trained, and none of it prevents someone else summarising you.

First: which "using" do you mean?

Three completely different things get filed under this question, and they have three different levers. Picking the wrong one is the single most common mistake here — people block a training crawler and are surprised they still appear in AI Overviews.

What you want stopped	The actual lever	Works?
Training — your text used to train future models	Block training crawlers (`robots.txt` + CDN)	Going forward only
Retrieval — assistants fetching and citing you live	Block search crawlers (`robots.txt` + CDN)	Yes, and you lose the citations
Snippet display — your words shown inside an AI answer	`nosnippet` / `max-snippet:0` meta directives	Yes, on Google

Decide which row you're in before touching anything. Row two in particular is a trade, not a fix — see the trade-off section below.

Layer 1 — robots.txt (free, easy, voluntary)

The baseline. Add a Disallow for the user-agents you object to. The full list of AI crawler user-agents and what each one does is worth reading before you write the file, because the tokens split by purpose: OpenAI runs GPTBot for training and OAI-SearchBot for search indexing, and blocking one doesn't block the other.

To block AI broadly while staying in ordinary search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

Two things this does not do. It does not remove you from Google AI Overviews — those are served through ordinary Google Search and crawled by Googlebot, so Google-Extended (which governs Gemini app and Vertex AI grounding) has no effect on them. And it does not stop anyone who ignores the file, because robots.txt is a published request with no enforcement mechanism whatsoever. It's a sign on the door, not a lock. The syntax rules and the precedence trap matter here — a crawler obeys only the single most specific User-agent block that matches it.

Layer 2 — snippet controls (the AI Overviews lever)

If your concern is specifically Google's AI features, this is the row you want. Google's snippet directives govern whether your text can be used in AI Overviews and AI Mode, and they work while you remain fully indexed and ranked.

Page-level, in your <head>:

<meta name="robots" content="nosnippet">

Or cap the length instead of removing it entirely:

<meta name="robots" content="max-snippet:0">

Or exclude just one section of a page, leaving the rest usable:

<p data-nosnippet>This paragraph won't be used in snippets.</p>

The catch is that this is genuinely a trade. nosnippet also removes your ordinary featured snippets and your search-result description text, which typically costs you click-through on results you're still ranking for. data-nosnippet is the precision instrument — use it to fence off the parts you care about rather than nuking the page.

Layer 3 — enforcement at the edge (the actual lock)

Layers 1 and 2 are requests. This is the one with teeth.

Blocking by user-agent or IP at your CDN, WAF or origin server means a non-cooperating bot gets a 403 rather than your content. Cloudflare, Fastly, AWS WAF and most managed hosts now ship one-click AI-bot rules — and since 2025 Cloudflare has blocked AI crawlers by default on new domains, with a marketplace letting publishers charge for access instead of simply refusing it.

What to know before enabling it:

User-agent strings are self-declared. A scraper can claim to be Chrome. Reputable AI companies publish verifiable IP ranges; serious blocking means checking those, not just the string.
Blocking is visible. If an assistant can't fetch you, users who explicitly paste your URL get an error. That's a real experience cost, not a theoretical one.
Rate limiting is often the better answer. If the actual problem is crawl load rather than principle, throttle instead of blocking — you keep the citations and lose the bandwidth spike.

Layer 4 — legal and licensing

Where technical measures end.

Terms of service. Stating that automated collection for model training is prohibited creates a contractual position. Whether it's enforceable varies by jurisdiction and is genuinely unsettled — treat it as a foundation for a claim, not as protection in itself.

Copyright and takedowns. DMCA notices work against reproduced content, not against a model that learned statistical patterns from it. If an AI product is outputting your text substantially verbatim, that's a stronger position than "it read my site."

Licensing. The pragmatic middle. Several publishers now sell access rather than blocking it, and edge providers have built payment rails for exactly this. If your content genuinely has commercial value to AI companies, being paid beats being scraped and beats being invisible.

Emerging standards. Machine-readable opt-out signals beyond robots.txt are being standardised — work on AI preference expression is active at the IETF, and the TDM Reservation Protocol exists for text-and-data-mining reservations. Adoption is early. Worth implementing alongside the layers above, not instead of them.

Nothing in this section is legal advice; if the stakes are commercial, take advice from someone qualified in your jurisdiction.

What none of this can do

Be clear-eyed, because there's a lot of overselling in this space.

It's not retroactive. Content already collected for a model that has already shipped stays in that model. Blocking today affects tomorrow's crawls, nothing before them.
It doesn't stop second-hand use. If another site quotes, summarises or reviews you, an assistant reads that page and describes you through it. Your robots.txt has no authority over anyone else's server.
It doesn't stop paraphrase of public facts. Ideas, prices and facts aren't protected by any of these mechanisms — only the specific expression is.
It doesn't stop bad actors. Every technical layer above is aimed at organisations that either cooperate or can be identified. Determined scrapers using residential proxies and browser automation defeat all of it.

The trade-off worth thinking about honestly

Blocking search crawlers has a price that people rarely price in: an assistant that can't read you can never cite you. Not now and not later.

Blocking is clearly right when your content is the product — a paywalled archive, proprietary research, a subscription publication. Giving that away to a summariser is straightforwardly losing money.

It's usually wrong when you publish content to be found. Marketing sites, documentation, blogs and support content exist to be discovered; opting out of a growing discovery channel to protect material you're already giving away free rarely nets out positive, and your competitors will still be in the answer.

The middle ground most publishers settled on: block the training-only crawlers, allow the search ones, and use data-nosnippet on anything you specifically don't want quoted. You opt out of training, keep the citations, and control the excerpts.

If you land there, the next question becomes how to actually earn those citations — which comes down to answering questions cleanly on the page and being a site the wider web talks about. Assistants lean heavily on corroboration, and a site nobody links to doesn't get named. Backlinkster covers the link half: one-for-one in-content swaps between real site owners, verified live by code, five a month free with plans from $19.

Frequently asked questions

How do I stop AI from using my content? Block the relevant crawlers in robots.txt, enforce that block at your CDN so non-cooperating bots are actually refused, and add nosnippet or data-nosnippet to stop your text appearing inside Google's AI answers. Each layer covers a different failure mode.

Does robots.txt actually stop AI from scraping my site? Only for crawlers that choose to obey it. The protocol is voluntary and has no enforcement. Major AI companies do honour it; scrapers don't. Blocking at your CDN or firewall is the enforceable version.

How do I stop my site appearing in Google AI Overviews? Use the nosnippet or max-snippet:0 robots meta directive, or data-nosnippet on specific passages. Blocking Google-Extended does not work — that token governs Gemini and Vertex AI, not AI Overviews, which run on ordinary Google Search.

Can I remove my content from an AI model that's already trained? No. Blocking a crawler prevents future collection; it does nothing about data already used to train a model that has shipped. There's no removal mechanism after the fact.

Will blocking AI crawlers hurt my SEO? Not directly — AI crawler directives don't affect Google or Bing rankings, provided you haven't accidentally blocked Googlebot or Bingbot. The indirect cost is losing citations in ChatGPT, Claude and Perplexity, which is a visibility loss rather than a ranking one.

Is there a way to get paid instead of just blocking? Increasingly, yes. Several CDNs now offer pay-per-crawl controls that let you charge AI companies for access rather than refusing it, and larger publishers negotiate licensing deals directly. It's a realistic option if your content has genuine commercial value.

The bottom line

Full prevention isn't available, and anyone promising it is wrong. What's available is layered control: a voluntary signal in robots.txt, an enforceable block at the edge, snippet directives for Google's answers, and a legal or licensing position underneath. Decide first whether you object to training, to retrieval, or to being quoted — they're separate problems with separate levers. Then accept the two hard limits: nothing is retroactive, and nothing stops other people describing you.