robots.txt
A file at the root of your site telling automated visitors which paths they may fetch. Many AI crawlers respect it, so a careless rule here can quietly remove you from AI answers entirely.
Updated August 21, 2026
robots.txt is the bluntest instrument on your site: one wrong line can exclude everything. It is also where AI crawlers are handled specifically, since several announce their own user agents. Blocking them is a legitimate business decision, but it should be a decision, not an accident inherited from a template. If you want to be quoted, check that the agents you care about are allowed and that your sitemap is declared here.
Related terms
- Crawler (bot). An automated program that fetches web pages. Search engines and AI systems use crawlers to read the web. Crawlers read the raw page and often do not run all of a site's scripts.
- llms.txt. A plain-text file at the root of a site that lists its most important pages for large language models, in the way robots.txt lists crawl rules. It is a proposed convention, not a standard any engine is obliged to honour.
- Technical accessibility. Whether a machine can fetch and read your page at all: status codes, response speed, robots rules and content that exists without JavaScript. It is the precondition for every other signal.