llms.txt Examples, Format, and llms-full.txt Explained

Updated October 2026. Rewritten against v2 of the spec, published August 2026 — which changed what ## Optional means, added link-relation discovery, and dropped the context-expansion tooling. Most llms.txt articles online still describe v1.

We run codersera.com/llms.txt and codersera.com/llms-full.txt in production — a 14.8 KB index and a 568 KB, 88,700-word corpus, rebuilt from a script. So this guide does two things other guides can't: it shows you a real file at real scale instead of an invented sample, and it reports what our own server logs say about who actually fetches it. The honest answer to that last question is uncomfortable, and it's in here.

What is llms.txt?

llms.txt is a plain Markdown file served at the root of a domain — https://example.com/llms.txt — that lists the pages you want a language model or AI agent to read, each with a one-line description. Jeremy Howard (Answer.AI, fast.ai) proposed it on 3 September 2024. The spec lives at llmstxt.org.

It is not robots.txt and it is not sitemap.xml. robots.txt restricts crawlers. sitemap.xml enumerates everything for indexing. llms.txt curates: it's the reading list you'd hand a smart new hire on day one, in priority order, with a sentence on each item explaining what it answers. Nothing in it is a rule, and it cannot block anything.

The problem it targets is real. Context windows are still smaller than most websites, HTML pages are mostly navigation and script, and every wasted token costs latency and money. The spec's own framing: the index "stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed."

What does a real llms.txt file look like?

Start with the smallest complete, fully compliant example in existence — the file the spec serves for itself, at llmstxt.org/llms.txt. It is 637 bytes and 10 lines, and it has every element the spec describes:

# llms.txt

> A proposal that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information.

## Docs

- [llms.txt proposal](https://llmstxt.org/index.md): The proposal for llms.txt
- [Python library docs](https://llmstxt.org/intro.html.md): Docs for `llms-txt` python lib
- [ed demo](https://llmstxt.org/ed.md): Tongue-in-cheek example of how llms.txt could be used in the classic `ed` editor, used to show how editors could incorporate llms.txt in general.

H1, blockquote, one ## section, three Markdown links each followed by : and a description, every link pointing at a .md file. That's the whole format. If you only copy one thing, copy that shape.

A real production file: ours

Here's the top of the file we actually serve. Full version: codersera.com/llms.txt — 14,821 bytes, 83 lines, 62 links across 5 sections, 97% of them described.

# Codersera

> Codersera matches companies with vetted remote developers across the modern web, mobile, AI, and infrastructure stack. We technically interview every developer, reference-check, and ship them with a risk-free trial period. We also publish long-form engineering and AI-tooling guides used by ~100k developers per month.

## About

- [About Codersera](https://codersera.com/about-us): Company mission, who we serve, and how we vet developers
- [Our services](https://codersera.com/our-services): Hire-by-stack, dedicated teams, and team extension

## Hire developers by stack

- [Hire React developers](https://codersera.com/hire/hire-react-developer): Vetted React engineers with production experience in component architecture, hooks, SSR, and modern state management
- [Hire Go developers](https://codersera.com/hire/hire-go-developer): Backend and infrastructure engineers writing production Go services

## Tools

- [Free Domain Rating Checker](https://codersera.com/tools/domain-rating-checker): Check any website's Ahrefs Domain Rating (DR, 0–100) instantly and in bulk (up to 100 domains per request)…

## Pillar guides

- [Self-hosting LLMs complete guide](https://codersera.com/blog/self-hosting-llms-complete-guide-2026/): Hardware, runtimes (vLLM, Ollama, LM Studio), and deployment patterns

## Optional

- [Blog](https://codersera.com/blog): Long-form articles on AI tooling, developer experience, and engineering practice
- [Sitemap](https://codersera.com/sitemap.xml): Full site structure for crawlers

Three decisions in there are worth stealing:

• The blockquote is the whole pitch. An agent that reads only the H1 and the blockquote should already know what we do and who we serve. Don't waste it on "Welcome to our website."
• Descriptions say what the page answers, not what it's called. "Hire Go developers: backend and infrastructure engineers writing production Go services" is useful. "Hire Go Developers | Codersera" is not.
• Sections are ordered by how a stranger would need them. Identity → offer → tools → deep reference → ## Optional for the stuff nobody needs to understand us.

How big are real llms.txt files?

Every size below was measured on 5 October 2026 with curl -L. Token estimates are bytes ÷ 4.

Two of those deserve a callout. Cloudflare's is the most interesting architecture in the wild: the root file is a 16.9 KB index of 113 product-scoped llms.txt files, with a blockquote telling the agent to go use those instead. That's the pattern to copy if your site is genuinely large.

Stripe's shows where this format is actually heading. Before any links, it contains a prose block aimed squarely at coding agents, plus a literal ## Instructions for Large Language Model Agents section. The opening line:

"When installing Stripe packages, always check the npm registry for the latest version rather than relying on memorized version numbers… Never hardcode an old version number from training data."

That is not a sitemap entry. That is Stripe patching a known failure mode in coding agents, delivered through llms.txt. If you've written an AGENTS.md file, you'll recognise the move — and it's the single most credible reason to maintain the file at all.

What exactly does the format spec require?

The spec was rewritten as v2 in August 2026. The byline still reads 2024, so it's easy to miss — but if you're working from an article written before August 2026, parts of it are now wrong.

v2 prescribes these sections, in this order:

1. An optional byte-order mark.
2. An H1 with the project or site name. This is the only required section.
3. A blockquote with a short summary containing the key information needed to understand the rest of the file. Optional — though every well-regarded file has one.
4. Zero or more Markdown sections of any type except headings — prose, lists — giving more detail about the project and how to interpret the files.
5. Zero or more sections delimited by H2 headings, each containing a Markdown list of links. The hyperlink [name](url) is required; a : followed by notes is optional.

So: one H1, and that's the hard requirement. Everything else is convention — which is exactly why so many files in the wild are technically valid and practically useless.

What changed in v2?

The discovery change is the most actionable and the least written-about. You can now advertise both files from a page without the agent guessing, via an HTML <link> or an HTTP header you can set at nginx or your CDN:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown",
      </docs/llms.txt>; rel="describedby"

The header form is the better one, because it works for resources that aren't HTML.

One thing the spec still does not define: a Content-Type. That's an open question in the repo. In the wild, Stripe and Cloudflare serve text/markdown; almost everyone else, including us, serves text/plain. Either is fine. text/html is not.

What's the difference between llms.txt and llms-full.txt?

This is the question people ask most, and the answer starts with a correction almost every article gets wrong:

llms-full.txt is not part of the spec. It appears nowhere in v1 and nowhere in v2. It is a community convention, spread largely by Mintlify, which auto-generates one for every documentation site it hosts. Because Mintlify hosts a very large share of developer docs, the convention looks universal when it isn't.

A related naming trap: in llms-ctx-full.txt, "full" meant includes the Optional section — not includes full page content. Different word, different meaning, and the two get conflated constantly.

How big do llms-full.txt files actually get?

Measured 5 October 2026. This is the part that changes how you should think about the file:

So the practical rule: once your llms-full.txt passes roughly 1.5 MB it stops being something a model can read and becomes something a pipeline has to chunk. Cloudflare's is seventeen times a one-million-token window. If you're building one in the hope that an agent will swallow it whole, keep it under about 500 KB — which is why ours concatenates 20 curated pillar guides and nothing else, rather than all 470-odd blog posts.

And it's a weakly-held convention even where people believe they've shipped one. Common Crawl's July 2026 analysis found 73.9% of llms-full.txt requests returned 404, with only 11.7% returning 200. Stripe's is one of those 404s.

How do you generate and maintain an llms.txt file?

There are three paths, and the one you pick should depend on how often your site changes.

Hand-written is correct for a marketing site or a small docs set. Half a day. The index is the part a human should curate, because the value is entirely in the judgement about what to include and how to describe it.

Generated by a tool is the fastest start and the worst ending if you stop there. Generator output is reliably too long, unordered, and missing the per-link descriptions that make the file worth reading. Treat it as a first draft. We cover which generators are worth using in our comparison of llms.txt generators.

Built in CI is the right answer for anything that changes weekly. This is what we do. The index at public/llms.txt is hand-maintained — it's 83 lines, and the descriptions are the product. The corpus is scripted: yarn build:llms runs scripts/build-llms-full.js, which pulls all 20 pillar guides from the CMS Admin API, converts each HTML body to clean Markdown, prepends a per-cluster table of contents, and writes a single public/llms-full.txt. It's idempotent — rerunning replaces the file cleanly — and it ships as a static asset on the next deploy.

What matters more than the script is the trigger policy, because a stale corpus is worse than none. Ours:

• Always regenerate when a new pillar is published, or when an existing one changes a section, a number, or a recommendation.
• Skip for typo and metadata fixes.
• Update the index by hand only when a page should join or leave the curated set — not every time you publish.

If you're on a hosted docs platform this is mostly done for you, with one limit worth knowing: Mintlify caps index files at 100,000 characters and auto-splits past that into separate /_llms/ files. The cap doesn't apply to llms-full.txt, which is how those 1.6 MB files happen.

Do AI crawlers actually read llms.txt?

Mostly no. Here is our own evidence, then everyone else's.

We pulled 15 days of nginx access logs for codersera.com — 21 September to 5 October 2026, 1,809,428 requests. Over that window /llms.txt and /llms-full.txt were successfully fetched 149 times. Here's who the named AI crawlers are, with their llms.txt fetches next to their total request volume on the same site over the same window:

Read that table carefully, because it says two different things.

The big labs don't look. Those crawlers made roughly 190,000 requests to our site in 15 days. ChatGPT-User, Claude-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot, DuckAssistBot and Meta-ExternalAgent fetched llms.txt zero times between them. GPTBot managed 3 out of 1,215 requests. They crawl us constantly. They do not read the file we wrote for them.

AI-native search startups do look, deliberately. ExaSearchBot and AndiAgent account for 48 of the 149 fetches — the single largest bucket. Andi's user-agent string is literally Andi-AgentSurfaceProbe, and it fetches llms.txt and llms-full.txt as a pair, every time. That's not incidental crawling; that's a product checking for an agent-facing surface.

Sorting all 149 fetches by what kind of client made them:

The second-biggest consumer of our llms.txt is the SEO tooling industry checking whether we have one. That is a genuinely funny result and it should calibrate your expectations.

Does the wider data agree?

Yes, and at much larger sample sizes.

Ahrefs analysed 137,210 domains against May 2026 server-log data. 28% published an llms.txt. Of those, 97% received zero traffic to the file in the entire month — nothing fetched them at all. Of the requests that did land, 96% came from bots, and 77% of those bots weren't AI tools: SEO audit tools were the largest single category at 21.7%, while AI retrieval bots — PerplexityBot, OAI-SearchBot, Claude's search crawler — accounted for 1.1%. Ahrefs also found that no AI bot ever speculatively probes for the file: zero requests arrived for llms.txt files that didn't exist.

Common Crawl examined 584,107 llms.txt files in its July 2026 archive and found the corpus is mostly junk: only 49.9% follow the spec's shape, only 32.9% include per-link descriptions, 40.4% are served as HTML rather than text, 22.6% contain no links at all, and 68.3% were produced by a template or plugin rather than curated — Wix alone accounts for 241,000 of them. It also found prompt injection in the wild: 3,793 files with mild steering language, 10 of top severity, 4 confirmed genuine attempts.

Google says so in its own documentation. Most articles quote a Reddit comment here. There's no need — Search Central's AI-features optimisation guide, last updated 10 July 2026, names the file under a heading reading "LLMS.txt files and other 'special' markup":

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

To Google's credit, the same page is fair about it: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files." Not useless — just not a Search input.

John Mueller's much-circulated comment says the same thing more bluntly. It's worth getting right, because the version that circulates has been quietly embellished. The original — u/johnmu, r/TechSEO, 16 April 2025, in a thread titled "LLM.txt – where are we at?" — reads in full:

"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag - this is what a site-owner claims their site is about ... (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)"

The widely-reproduced version adds a clause about AI services crawling "with a regular user-agent." He never wrote it. In a piece about checking your own logs, that's worth flagging: the most-quoted sentence in this debate has itself drifted from its source.

His position hardened rather than softened over the following eighteen months — "none of the search engines currently use the llms.txt file as anything other than a random text file" (February 2026), and on Google specifically: "Google doesn't use llms.txt or llms-author.txt. I don't know of any other crawler / llm confirming they're using these (other than SEO tools)" (July 2026).

That parenthetical matches our log table exactly — SEO tooling was our second-largest bucket at 46 of 149 fetches. He also offered the best available explanation for why any fetches show up: there are now directory sites that enumerate every domain's llms.txt and link to it, so ordinary crawlers find those links and follow them. We should assume some of our own 149 are exactly that, not interest in the file. One independent datapoint lines up unnervingly well: an operator hosting ~20,000 domains reported that nothing fetched these files except "some niche user agents (builtwith.com/biup is one)." BuiltWith accounts for 4 of our 149.

Astro deleted theirs. In PR #13538, merged 20 April 2026, the Astro docs team removed llms-full.txt and llms-small.txt entirely, writing: "we've seen little uptake recently in usage — these files get very little traffic. Instead we can focus efforts on the MCP server." Their numbers: under 0.1% of docs traffic, roughly 23 seconds of every build — about 10% of total build time — and around 2.5 hours of CI per month. Build time fell from 5m06s to 4m22s after removal.

A different mechanism reaches the same split. Cloudflare didn't bet on a root file at all — in February 2026 it shipped Markdown content negotiation at the edge, converting pages on the fly for clients that send Accept: text/markdown. Dries Buytaert, checking his own logs for exactly that header, reported: "No AI crawler uses content negotiation. Not one." Cloudflare's own write-up says the opposite about a different population: "we already see some of the most popular coding agents today — like Claude Code and OpenCode — send these accept headers."

Both are true, and together they draw the line more sharply than any llms.txt study does: open-web crawlers ignore every voluntary opt-in mechanism; interactive client agents use them. It isn't something about llms.txt specifically. It's a property of who's on the other end of the request.

The strongest case for the format comes from Mintlify, which benchmarked agent retrieval across thousands of runs and found that pointing an agent at an llms.txt reduced wasted fetches — a result Ahrefs acknowledged even while publishing the bearish traffic numbers above. Note what it measures: agent efficiency once directed at your docs, not discovery, and not citations.

So what is the honest verdict?

Four findings survive all of this, and they point the same way:

1. As an SEO or AI-citation tactic, llms.txt does not work. No major answer engine fetches it, and Google documents that Search ignores it. Nobody has produced evidence that having one improves citations. Don't let anyone sell you otherwise.
2. Supply is universal; documented demand is zero. OpenAI, Anthropic, Perplexity, Mistral, Cloudflare and Google all publish an llms.txt for their own developer docs. Not one of them documents reading anybody else's. Every crawler-control page they ship describes robots.txt and mentions llms.txt only to point at its own index; Anthropic's crawler-support article doesn't mention it at all, and Bing's webmaster blog has never mentioned it across its entire archive. That asymmetry is the honest headline, and it's more informative than any single study.
3. As an agent-facing surface, it does something. Ahrefs found Claude Code out-fetched every AI retrieval bot and every AI assistant in their dataset. Our logs show agent runtimes hitting it. The spec's v2 changelog states flatly that "coding agents use them reliably." If developers using coding agents are part of your audience, the file is read — just by the IDE, not by the search engine.
4. The content, not the file, is doing the work. Stripe's instructions block is valuable because it fixes a real agent failure. A generated list of 400 undescribed URLs is valuable to nobody. The format is a delivery mechanism; it does not add value on its own.

What are the most common llms.txt mistakes?

Ranked by how often they actually occur in the Common Crawl corpus, plus two verified live examples.

1. Serving it as HTML — 40.4% of successful responses. Usually a framework catch-all route swallowing the path. docs.cursor.com/llms.txt does this today: it 308-redirects to the docs homepage and serves 574 KB of HTML. The real file is at cursor.com/llms.txt. An agent following the obvious URL gets a web page.This failure has a second-order effect worth knowing. Google's Lighthouse added an llms.txt audit in April 2026 (core/audits/agentic/llms-txt.js, still flagged as under development). A clean 404 returns not applicable — no penalty. But a catch-all route returning HTTP 200 with HTML fails the H1, Markdown-link and length checks and scores zero. The same ambiguity inflates every adoption statistic that counts an HTTP 200 as "has an llms.txt" without validating the body — which is why the figures in this article come from studies that parsed the files, and why you should treat higher adoption claims sceptically.
2. No per-link descriptions — 67.1% omit them. The worst offender is prominent: docs.claude.com/llms.txt lists 753 links and describes none of them. A bare URL list tells a model nothing it couldn't get from your sitemap.
3. Listing everything instead of curating — 68.3% of files are plugin output, and 22.6% contain no links at all. If you wouldn't want a model quoting a page, leave it out.
4. Multiple H1s. The spec says one. cursor.com/llms.txt has 4; ai-sdk.dev/llms.txt has 42. Ours had 3 — two build notes at the top of the file, written as #-prefixed comments. Markdown has no comment syntax, so a parser reads them as headings and can bind your build note as the project title. The fix is to move notes into the prose block after the blockquote, which the spec explicitly permits as long as it isn't a heading. We'd shipped that bug for months and only caught it while writing this.
5. A llms-full.txt nobody can load — or that doesn't exist at all, which is 73.9% of requests.
6. Linking to HTML when a Markdown version exists. Cloudflare links 0 of 113 to .md; we link 0 of 62. It's not fatal — it just makes the agent pay the HTML parse cost you were trying to save it.
7. Treating it as access control. It isn't. llms.txt cannot block, restrict, or opt anything out. Only robots.txt does that.

Should you ship one anyway?

A decision rule rather than a cheerleading section.

Ship a curated llms.txt if your audience includes developers who use coding agents, or you have documentation an agent could get wrong in a way that costs you support tickets. That's Stripe's case, and it's a real return. Budget half a day, write real descriptions, and consider whether your highest-value content needs an instructions block more than a link list.

Ship an llms-full.txt only if you can keep it under about 500 KB and you can generate it from a script. Above ~1.5 MB you're producing a bulk-download artifact, not context — which is fine, but be clear that's what it is.

Don't bother if your site is a marketing site with no documentation and no developer audience, or if the plan is to point a plugin at your sitemap and call it AI optimisation. You'd be adding one more file to the 97% nothing ever fetches.

And whatever you decide, get robots.txt right first. Explicit User-Agent rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent are the only lever in this space that every major AI company actually honours today. llms.txt is optionality; robots.txt is control.

FAQ

What is llms.txt?

A Markdown file at your site root that lists the pages you want AI models and agents to read, each with a one-line description. Proposed by Jeremy Howard on 3 September 2024; the spec is at llmstxt.org and was revised to v2 in August 2026. It's a curated reading list, not a crawl rule.

What does an llms.txt file look like?

An H1 with your site name, a blockquote summary, then ## sections containing Markdown links in the form - [Page name](https://example.com/page.md): what this page answers, with an optional ## Optional section at the end for skippable content. The smallest complete real example is llmstxt.org's own 637-byte file, reproduced in full above.

What's the difference between llms.txt and llms-full.txt?

llms.txt is the specified file — a curated index of links, typically 1–90 KB. llms-full.txt is a community convention with no spec, popularised by Mintlify: the entire body text of every page concatenated into one file, typically 570 KB to 64 MB. The index is meant to be read in context; the full file is realistically a bulk-ingestion artifact.

Is llms.txt required?

No. It's a voluntary proposal, not a standard — no IETF or W3C status, no search engine requires it, and no AI provider has committed to reading it. Roughly 28% of domains Ahrefs surveyed publish one. The only thing that's "required" is the H1, and only if you choose to publish a file at all.

Do AI crawlers actually read llms.txt?

Overwhelmingly no. In 15 days of our own server logs, the named crawlers from OpenAI, Anthropic, Google, Perplexity, Meta and ByteDance made ~190,000 requests to our site and fetched llms.txt 3 times between them — all GPTBot. Ahrefs found 97% of llms.txt files received zero fetches in a month. The exceptions that do read it are AI-native search startups like Exa and Andi, and coding agents such as Claude Code.

How do I generate llms.txt?

Hand-write the index — it's the descriptions that carry the value, and 83 lines is a half-day. Generate the full-text corpus from a script in CI: ours pulls every pillar guide from the CMS API, converts to Markdown, and writes public/llms-full.txt idempotently via one command. Generator tools are fine for a first draft but reliably skip the descriptions and the priority ordering.

Where do I put the llms.txt file?

At the root of your domain: https://yoursite.com/llms.txt. In Next.js that's public/llms.txt; in Hugo, Docusaurus or Astro, static/ or the equivalent — the same place robots.txt lives. Since v2 you can also publish subpath files like /docs/llms.txt, which cover pages under that path, with the most specific file winning. The spec explicitly rejects /.well-known/.

How often should I regenerate llms.txt?

Regenerate whenever content changes a section, a number, or a recommendation — and always when you add or remove a page from the curated set. Skip it for typos and metadata edits. If the file is scripted this costs nothing, so the real risk is the opposite: a corpus that silently goes stale and starts feeding agents last quarter's pricing.

Does llms.txt help SEO or AI citations?

There is no evidence that it does, and reasonable evidence that it doesn't. Google treats it as an unverified site-owner claim, comparable to the old keywords meta tag. No study has shown a citation lift. Treat it as a developer-experience feature, not a ranking tactic.

Can llms.txt block AI crawlers?

No. It's an inclusion file with no enforcement semantics — it cannot restrict, rate-limit, or opt your content out of training. Use robots.txt with per-User-Agent rules for that, which is the mechanism the major AI companies do respect.

Should I use llms.txt or AGENTS.md?

Different jobs. llms.txt is published on your website for agents arriving from outside; AGENTS.md lives in a repository and steers an agent already working inside the code. If you maintain a public API or SDK, you plausibly want both — and Stripe's instructions block shows the two ideas converging.

What's the bottom line?

In October 2026, llms.txt is a well-designed format solving a problem the companies that would benefit most have declined to adopt. The spec got better in v2. Adoption got wider and much shallower. And the crawlers everyone writes these files for still don't request them.

Ship one if you have developer documentation and an agent-using audience, keep it curated, put real instructions in it where they'd help, and script the corpus so it never goes stale. Then go set your robots.txt properly, because that's where the actual control still lives.

There's a useful test for any claim you read about this file. In November 2025, John Mueller joked that if the value of llms.txt lies in enjoying the process, you may as well "call it cats.txt — who doesn't like cats?" Someone built it: catstxt.org now hosts a straight-faced parody standard, complete with an RFC draft and a mandatory ASCII cat. It is satire, and it is also a measuring instrument — any argument that proves llms.txt works will, applied unchanged, prove cats.txt works too. If the case for a file rests on "serious companies publish one" and "it can't hurt," the case is not yet evidence.

So measure it the only way that means anything: grep your own access logs. Ours took ten minutes and taught us more than every study we read.

If this is the kind of detail your stack needs someone to own, Codersera matches teams with vetted remote engineers who check the logs before believing the blog post.