llms.txt explained: the AI-era site map

What llms.txt actually is

llms.txt is a community proposal (llmstxt.org) for a Markdown file served at https://yourdomain.com/llms.txt. Where robots.txt says whichbots may crawl and sitemap.xml lists every URL for indexing, llms.txt does something neither does: it gives a language model a short, human-curated tour of the pages that actually matter, with one-line context on each. Think of it as the README you would hand an analyst who has thirty seconds to understand your site.

The format (it is strict, and small)

A valid file is plain Markdown with a specific shape. Parsers are unforgiving about the first two elements, so get them right:

• An H1 on the first non-blank line — your site or product name.
• A blockquote summary directly under it — one or two sentences on what the site is and who it is for.
• Optional free-text paragraphs giving extra context (no headings here).
• One or more H2 sections (e.g. ## Docs,## Product), each a bulleted list of- [Title](https://url): one-line description links.

Common mistakes our auditor flags: no H1 on line one, multiple H1s, an H2 section with no link bullets, or headings used before the first section. Keep it boring and exact.

Where it goes

Domain root, exactly /llms.txt, served as text/plain ortext/markdown, HTTP 200, no auth wall. Subpaths and meta tags do not count. Some sites also publish a fuller /llms-full.txt with expanded content — optional, and only if you genuinely have the depth to justify it.

How it differs from robots.txt and sitemap.xml

They solve different problems and you want all three. robots.txt is access control — see our companion post on robots.txt for AI crawlers. sitemap.xml is exhaustive discovery for indexers. llms.txt is editorial: a small, ranked, described shortlist for a model that will only ever skim. A page can be in your sitemap, allowed in robots.txt, and still be invisible to assistants because nothing told them it was the page that matters. That is the gap llms.txt fills.

Is it worth adding in 2026?

Honest answer: probably, but keep your expectations grounded. No major vendor confirms it consumes llms.txt, most AI crawlers don't appear to fetch it today, and independent analyses have found no measurable link between having one and being cited. A vocal part of the SEO community calls it busywork — and on the “does a crawler read it?” question, they have a point. So don't add it expecting a ranking bump.

Where it genuinely helps is agent context: when someone points ChatGPT or Claude directly at your site, a clean file gives the model a tidy map (Anthropic uses this pattern for its own docs). And it's a forcing function — writing the one-line descriptions makes you decide what your best pages actually are, which is the same discipline that gets you cited by AI assistants in the first place. The cost is a few minutes for a file that rarely changes, so it's a reasonable low-effort add — just not the thing that moves your visibility.

For the bigger picture of where llms.txt sits among the things that actually drive AI visibility, see our guide to Answer Engine Optimization.

Generate it, then verify it

You can write llms.txt by hand from your sitemap in fifteen minutes, or generate a spec-compliant draft and hand-tune it. Either way, verify the result the way an AI crawler sees it: run a free auditand it checks the file's presence, the H1/blockquote shape, and that every section has real links — the same rules above. Pair it with IndexNow so changes propagate fast, and see the plans if you want the generator and monitoring.

FAQ

Want this checked on your domain? Run a free audit or take the 2-minute readiness quiz.