Which AI crawlers can read your site?
13 documented AI user agents read the web for the major answer engines, and they do different jobs. This reads your live robots.txt and tells you which of them you are letting in — and, more usefully, which ones you have blocked without meaning to.
The mistake this is here to catch
Blocking GPTBot keeps your content out of OpenAI’s training data and costs you nothing in ChatGPT search. Blocking OAI-SearchBot removes you from ChatGPT’s answers entirely. They are different tokens with different consequences, and most “block AI crawlers” snippets circulating online block both. If you wanted the first and got the second, this is where you find out.
Every token checked here comes from the vendor’s own documentation, last verified September 5, 2026. The full reference, with what each one does and three copy-paste robots.txt policies, is on the AI crawler page.
Frequently asked questions
- What exactly does this check?
- It fetches https://yourdomain.com/robots.txt and works out, for each documented AI user agent, whether that file lets it crawl your site root. It reads one public file and nothing else.
- I have no robots.txt. Is that bad?
- No. With no file, everything is allowed, which is the right default for most businesses. It does mean you have no way to opt out of model training without adding one.
- It says allowed, but I still do not appear in AI answers.
- Access is the first hurdle, not the last. A crawler that reaches a page built entirely in JavaScript still leaves with nothing, and a page that never answers the question has nothing to quote. That is what a readiness scan measures.
- Two agents are marked as ignoring robots.txt. What do I do about those?
- OpenAI and Perplexity both state that their user-triggered agents do not follow robots.txt, because a person asked for that page. If you need to stop them, robots.txt is the wrong layer — block at the server, CDN or firewall instead.