What Is llms.txt? (And Do You Actually Need One)

llms.txt is a plain Markdown file placed at the root of a site — yourdomain.com/llms.txt — that lists your most important pages with a one-line description of each, so a language model can find them without crawling the whole site. It's a proposed convention, not a standard, and no major AI company has committed to reading it.

Where it came from

The idea was proposed in September 2024 by Jeremy Howard of Answer.AI, and the format is documented at llmstxt.org. The reasoning is straightforward: HTML pages are full of navigation, cookie banners, scripts and layout markup, and models have finite context windows. A curated Markdown index gets a model to your actual content faster.

It borrows its shape from robots.txt — a well-known filename at a well-known location that machines can check first. That analogy is where most of the confusion starts, so it's worth being precise about the difference.

llms.txt vs robots.txt — not the same job

	`robots.txt`	`llms.txt`
Age	Standard since 1994	Proposed 2024
What it does	Tells crawlers what they may and may not fetch	Suggests what's worth reading
Enforced?	Respected by every major crawler	Advisory only
Read by AI bots?	Yes — GPTBot, ClaudeBot, PerplexityBot all honour it	Rarely, if at all
Effect if wrong	Can block AI systems entirely	Nothing happens

That last row matters more than anything else on this page. robots.txt is the file with real consequences: if it disallows GPTBot or ClaudeBot, those systems cannot read your pages and therefore cannot cite you. llms.txt has no such power in either direction.

What actually goes in the file

The format is deliberately simple Markdown:

# Backlinkster

> A peer-to-peer backlink exchange where site owners trade
> one-for-one in-content links, verified live by code.

## Guides
- [What is a backlink in SEO](https://backlinkster.com/resources/what-is-a-backlink-in-seo): Definition, examples, and why links still count.
- [What is anchor text](https://backlinkster.com/resources/what-is-anchor-text): The types, and the over-optimisation trap.

## Product
- [How it works](https://backlinkster.com/#how): The swap and verification process.
- [Pricing](https://backlinkster.com/#pricing): Plans and limits.

One H1 with the site name, a blockquote summary, then H2 sections containing linked lists where each entry has a short description. Some sites also publish /llms-full.txt, which concatenates the full text of every listed page into one file so a model can ingest everything in a single fetch.

Do AI systems actually read it?

Mostly no, and it's important to be honest about this rather than sell the file as an AI-visibility trick.

Google's John Mueller has publicly compared llms.txt to the old keywords meta tag — present, ignored — and said Google doesn't use it. OpenAI, Anthropic, Perplexity and Microsoft have not announced support for it either. Server-log analyses shared by site owners consistently show AI crawlers hitting HTML pages and robots.txt in volume while llms.txt requests sit near zero.

There is one genuine exception worth knowing: documentation platforms. Mintlify generates llms.txt and llms-full.txt automatically for the docs it hosts, and a number of developer-tool companies — Anthropic's own docs among them — publish one. Where these get used, it's usually a developer pasting the URL into a chat window on purpose, not a crawler discovering it. That's a real use case, just a much smaller one than "AI search optimisation."

So should you add one?

A reasonable position: add it if it's cheap, never treat it as a strategy.

It's worth it when:

You run developer docs or an API reference, where people genuinely paste the file into an assistant to give it context.
Your site generates the file automatically from existing content, so it costs nothing to maintain.
You want to be ready if adoption changes — the cost of being early is roughly zero.

It's not worth it when:

You'd be hand-maintaining it. A stale index is worse than none, and the moment it drifts from reality it's actively misleading.
You're adding it instead of fixing page content. That's the expensive mistake.

The thing to avoid is the trade most people accidentally make: an afternoon on llms.txt is an afternoon not spent putting a direct answer in the first fifty words of your top ten pages, which is the change that demonstrably affects whether you get quoted. Our own breakdown of whether backlinks help you get cited by ChatGPT walks through what the citation chain actually rewards.

What to do instead, in priority order

Check robots.txt allows the AI crawlers you want citations from — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended. This is the only file that can stop you outright.
Open each important page with a self-contained 40–60 word answer. Models lift passages, not pages.
Add real FAQ blocks — question-and-answer pairs are pre-formatted quotable passages.
Add Article and FAQPage schema so the structure is machine-readable without guesswork.
Publish a plain, indexable pricing page — agents comparing tools skip what they can't parse.
Get discussed off-site, on Reddit, forums, review sites and roundups. Assistants lean heavily on third-party mentions.
Then, if you like, add llms.txt.

Backlinkster publishes both /llms.txt and /llms-full.txt, generated automatically from the resource library on every deploy. They cost nothing to keep current, and they are not why anything here gets cited — the pages are. If you want the ranking half of that equation, see how the swaps work: one-for-one in-content links with owners in related niches, each verified live by code.

Frequently asked questions

What is llms.txt? A Markdown file at a site's root that lists its most important pages with short descriptions, so a language model can locate the useful content without crawling everything. It was proposed in September 2024 and is documented at llmstxt.org. It is a convention, not an adopted standard.

Do I need an llms.txt file? Only if it's free to maintain. No major AI provider has committed to reading it, so it won't affect citations on its own. Add it if your site can generate it automatically; skip it if you'd be hand-updating a file nobody fetches.

What's the difference between llms.txt and robots.txt? robots.txt is an enforced 1994 standard that controls whether crawlers — including GPTBot, ClaudeBot and PerplexityBot — may fetch your pages at all. llms.txt is an advisory 2024 proposal that suggests which pages are worth reading. Only one of them can block you.

Does Google use llms.txt? No. Google's John Mueller has publicly likened it to the keywords meta tag and stated Google doesn't use it. Google's own AI-features guidance says no special files or markup are required to appear in AI Overviews or AI Mode.

What is llms-full.txt? A companion file containing the complete text of every page listed in llms.txt, concatenated into one document, so an assistant can ingest a whole site in a single fetch. It's most common on documentation sites.

Will llms.txt help me rank in Google? No. It is not a ranking signal and Google does not read it. Ranking still comes from the usual inputs — matching search intent, content quality, internal links and external links.

Where do I put the llms.txt file? At the domain root, served as text/plain or text/markdown, at https://yourdomain.com/llms.txt. Nested locations aren't part of the convention.

The bottom line

llms.txt is a sensible idea that hasn't been adopted. The file is cheap, harmless, and occasionally useful for developer documentation — so publish one if your site can generate it automatically. Just don't let it stand in for the work that actually determines whether an AI names you: a clear answer near the top of the page, real Q&A blocks, valid schema, an unblocked robots.txt, and enough authority to be retrieved in the first place.