What is llms.txt
llms.txt is a Markdown text file at the root of a website (yoursite.com/llms.txt) that gives large language models (LLMs) a short, curated map of the site's most useful pages.
Jeremy Howard of Answer.AI proposed it in a post dated September 3, 2024. llms.txt is a reading guide for AI with no ranking power of its own.
A plain definition for marketers
For marketers, llms.txt is a table of contents written for AI assistants. It names your brand and sums up what you do in a sentence or two. Below that it links the pages you'd want a model to read first, each with a one-line note. A sitemap lists every URL you have, while llms.txt lists only the pages that explain you best.
Why it matters for AI
The file matters because a model reads the web through a limited context window, and HyperText Markup Language (HTML) fills that window with menus and scripts. Howard's proposal suggests adding the file "to websites that are designed for reading by language models, not just humans." A clean Markdown summary lets a retrieval tool reach the right page in one fetch. For a brand, that can raise the chance that an assistant reads your product documentation before an old press release.
How LLMs read the file
An LLM reads llms.txt only when a tool fetches it on purpose. The proposal describes this happening while the model answers a question. After an assistant or coding agent requests /llms.txt, it reads the summary and follows the links that match the question. Markdown helps here because headings and lists parse cleanly. A companion file, llms-full.txt, puts the full text of the linked pages into one document, so an agent can take in everything with a single request.
Is llms.txt mandatory
No. llms.txt isn't required for search engine optimization (SEO), and Google doesn't use it to rank pages. John Mueller, Search Advocate at Google, put it bluntly on Reddit: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." He compared it to the old keywords meta tag. Your site works normally without the file.
Is llms.txt worth it
It is worth an hour of work, not more. Writing the file costs little and carries no risk, but the file won't change your rankings or your AI citations by itself.
When the file helps
Sites with deep documentation get the most from it. These include developer tools and API references, because coding assistants and agents fetch documentation on demand. Help centers benefit for the same reason. Writing the file also makes your team decide which 15 or 20 pages define the brand, and we've found that exercise useful in its own right.
When it adds little
A small brochure site with clean HTML gains almost nothing, because a model can already read those pages directly. The file can't rescue weak content either. If the linked pages don't answer what buyers ask, the assistant won't quote them, however neatly you list them.
A short file example
An llms.txt file has an H1 with the site name and a blockquote summary. Its annotated links appear in H2 sections.
```markdown # Example Co
> Example Co makes invoicing software for freelancers in the EU.
Product
Optional
- Blog: guides and updates
```
The spec allows a model short on context to skip links in the "Optional" section.
How to generate llms.txt
We recommend writing the file by hand. A raw crawl export lists everything, which defeats the purpose.
- 011Choose the pagesPick the pages that answer the questions buyers ask most often.
- 022Write the summaryDescribe in one or two plain sentences what you sell and who it's for.
- 033Group the linksSort them under H2 headings and give each one a one-line description.
- 044Publish at the rootSave the file as UTF-8 text at /llms.txt.
- 055Update it with the siteRevise the file whenever a key page moves or retires.
How to check your file
A working file loads with functional links and tells a model who you are.
- 011Open the URLConfirm /llms.txt returns a 200 status as plain text.
- 022Test every linkFix any link that returns a 404 or redirects.
- 033Read it as a model wouldPaste the file into an assistant and ask what your company does.
- 044Measure the outcomeWe track mentions and citations in ChatGPT and Google AI Overviews with weekly prompt sets. That shows whether the assistants' answers change after the file goes live.
Related terms to know
These terms often come up alongside llms.txt: