What goes in an llms.txt file
The format is deliberately minimal so both humans and models can read it. The structure, in order: a single H1 with the project or site name; an optional blockquote giving a one-line summary; optional free-form Markdown with context; then any number of H2 sections, each containing a bullet list of links in the form [name](url): optional note. A section named Optional by convention marks links a model can skip if it is short on context.
The sibling file /llms-full.txt takes this further by inlining the actual Markdown content of the listed pages into one document, so a model can ingest an entire docs site in a single fetch instead of crawling page by page. The code example below shows a typical llms.txt.
llms.txt vs robots.txt vs sitemap.xml
All three are root-level files about how machines should treat your site, but they answer different questions and are consumed by different clients.
| File | Question it answers | Format | Consumed by |
|---|---|---|---|
llms.txt | Which content matters most, in clean form? | Markdown, human-readable | LLMs / AI tools (advisory) |
robots.txt | What may a crawler access? | Directives (User-agent / Allow / Disallow) | Search and AI crawlers |
sitemap.xml | What URLs exist and when did they change? | XML, machine-readable | Search engine crawlers |
The key distinction: robots.txt grants or denies access and AI-specific bot directives belong there, a sitemap is an exhaustive URL index for crawlers, and llms.txt is a curated, opinionated summary aimed at a model's context window. They are complementary, not substitutes.
Adoption in 2026 and how to generate one
An honest read of adoption: llms.txt is widely published by developer-tool and documentation sites, and a growing number of AI dev tools read it, but it is not a universally honored standard - many models and agents still fetch and parse live HTML rather than looking for /llms.txt first. Treat it as a low-cost, upside-only signal, not a guarantee that a model will use it. Its value also overlaps with retrieval and MCP-based access, where an agent fetches your content on demand.
Generating one means producing clean Markdown for your key pages - which is the same job as scraping a site for LLMs. For a static site you can template it from your own content; for a JavaScript-rendered site you first need each page rendered and converted to Markdown, which a web-data API that returns Markdown does in one call. Once you have the per-page Markdown, assembling llms.txt (the link index) and llms-full.txt (the inlined text) is straightforward.