llms.txt
A plain-text file at the root of a site that lists its most important pages for large language models, in the way robots.txt lists crawl rules. It is a proposed convention, not a standard any engine is obliged to honour.
Updated August 21, 2026
The idea is to hand a model a short, curated map instead of making it infer one from your navigation. It costs almost nothing to publish and it cannot hurt, but be honest about the status: no major engine has committed to reading it, and it will not compensate for pages that are blocked, thin or contradictory. Treat it as a courtesy on top of a site that already works, never as a shortcut around one that does not.
Related terms
- robots.txt. A file at the root of your site telling automated visitors which paths they may fetch. Many AI crawlers respect it, so a careless rule here can quietly remove you from AI answers entirely.
- Crawler (bot). An automated program that fetches web pages. Search engines and AI systems use crawlers to read the web. Crawlers read the raw page and often do not run all of a site's scripts.
- Retrieval. The step where a system finds and fetches your page. If a page is blocked, unreachable, or clearly not about the topic asked, it is never retrieved, and nothing else matters.