llms.txt Documentation Index Generator
A sitemap built for AI: llms.txt turns your site into a structured list models can consume, llms-full.txt compiles the entire site into one Markdown file for a single-shot context load, and a docs MCP server lets AI retrieve your own content in real time
Tool Interface
Interactive tool will be available soon
Features
- ✓ An index for AI: llms.txt helps large language models index content more efficiently, much like a sitemap helps search engines understand your structure
- ✓ Full context in one shot: llms-full.txt compiles your whole site's text into a single Markdown file, so one URL loads all your documentation into an AI tool
- ✓ Zero maintenance: modern documentation platforms generate and host llms.txt automatically so it stays in sync without manual updates after every release
- ✓ Docs as an MCP server: search MCP servers can be generated from your documentation so Claude, Cursor, Goose and ChatGPT can retrieve — and where relevant call — your content
- ✓ Discoverable by agents: Link response headers and X-Llms-Txt advertise llms.txt, your API catalog, MCP server card and agent card so the AI ecosystem finds you
How to Use
- Decide which layer AI needs: llms.txt when you only need a page inventory, llms-full.txt when a model must read all your text in one pass
- Turn on automatic generation: on a modern docs platform this is a setting; for a self-built site, write a Markdown manifest to the llms.txt standard and serve it at the root
- Enable live retrieval: generate a search MCP server for your docs and expose it to the AI clients your team uses, so answers come from your latest documentation rather than training data
- Monitor and iterate: check crawl logs to confirm AI tooling actually consumes it, watch index size limits, and move to a hierarchical index as the site grows so pages are not silently dropped
FAQ
What is llms.txt and why make a separate file for it?
llms.txt is an industry standard that helps large language models index content more efficiently. The official documentation draws the analogy that it is similar to how a sitemap helps search engines. Because model crawls and context budgets are limited, making them dig through HTML, scripts and ads is wasteful; a structured manifest simply tells them which pages exist. See https://www.mintlify.com/docs/ai/llmstxt and https://www.mintlify.com/blog/what-is-llms-txt
How do I choose between llms.txt and llms-full.txt?
They serve different jobs. llms.txt is a site index, a catalogue for AI that helps it discover and locate pages. llms-full.txt compiles all of your site's text into a single Markdown file, which the docs describe as letting developers paste a single URL to load full context into AI tools like ChatGPT or Claude. Per the official blog, llms-full.txt was developed by Mintlify in collaboration with Anthropic and is now part of the official llms.txt proposal. Small sites can start with the full file; large sites should index first and fetch full text on demand. See https://www.mintlify.com/docs/ai/llmstxt
Do AI systems actually use it?
Separate the two claims. Whether a given AI product crawls it depends on that vendor's crawling policy and is not yours to decide, so do not treat it as an SEO guarantee. The other side is a definite payoff: once you connect a docs MCP server, MCP clients such as Claude, Cursor, Goose and ChatGPT can search and retrieve your first-party documentation in real time, and on that path whether AI uses it depends entirely on the connection you and the client control. See https://www.mintlify.com/docs/ai/model-context-protocol
Does llms.txt break down for very large sites?
Yes, and it is a documented pitfall. An official engineering post says large documentation sites were hitting the 100,000-character llms.txt limit and omitting pages, so generation was rebuilt as a hierarchy of files that lets agents reach every page without loading the entire index. Once you are at scale, choose a hierarchy or keep only key entry points in the index. See https://www.mintlify.com/blog/what-is-llms-txt
Does this duplicate traditional SEO?
No, the audiences differ. Traditional SEO optimises crawling and ranking for search engines and the results page a human reads. llms.txt and docs MCP servers target AI agents, so that when they answer a user they can cite your accurate, current content instead of guessing from training data. Both draw on the same content assets but the delivery format and access path are different. See https://www.mintlify.com/docs/ai/llmstxt and https://www.mintlify.com/docs/ai/model-context-protocol