Does blocking GPTBot remove me from ChatGPT?

No, and this is the most common and most expensive mistake in the whole area. GPTBot is the training crawler. The bot that decides whether you appear in ChatGPT's search results is OAI-SearchBot. Blocking GPTBot keeps your content out of training while leaving you fully visible in ChatGPT search; blocking OAI-SearchBot removes you from the answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google states plainly that Google-Extended has no effect on inclusion in Google Search and is not used as a ranking signal. It only governs whether your content trains Gemini and grounds its answers.

Some of these bots ignore robots.txt. Is that allowed?

OpenAI and Perplexity both state that their user-triggered agents do not follow robots.txt, on the grounds that a person asked for that specific page rather than a crawler deciding to visit. Whether you find that reasonable is your call; the practical point is that robots.txt will not stop them, so if you need to, block at the network or server level instead.

Is a robots.txt rule enough to keep my content out of a model?

It is the standard mechanism and the major vendors say they honour it, but it is a request, not a wall. It also does nothing retroactively: content already collected has already been collected. If exclusion genuinely matters to you, treat robots.txt as one layer and not the whole answer.

Do I need llms.txt as well?

It costs nothing and cannot hurt, but be clear-eyed about the status: no major engine has committed to reading it. It will not compensate for pages that are blocked, thin or contradictory. Fix robots.txt first.

Reference

AI crawlers: every user agent, and what blocking each one costs you

The AI bots that read your site, taken from each vendor's own documentation: what every user-agent token does, which ones ignore robots.txt, and why blocking the training bots is a different decision from blocking the ones that put you in answers.

Updated September 5, 2026

There is no single “AI bot”. Thirteen documented user agents read the web on behalf of the major answer engines, and they do four different jobs. Treating them as one thing is how site owners end up blocking themselves out of ChatGPT while believing they only opted out of training.

Every token below comes from the vendor’s own documentation, linked at the foot of the page, and was checked on September 5, 2026. Copy them exactly — a misspelled user-agent line is not an error, it is a rule that silently matches nothing.

Check your own domain against this list → The free checker reads your live robots.txt and tells you which of these you are letting in.

The one distinction that matters

Search bots decide whether you can appear in an answer. Training bots decide whether your work is used to build the model. They are separate tokens, separate decisions, and separate consequences. Most people want to be in the answers; many also want to stay out of the training. Those two wishes are compatible, and the recipe below does exactly that.

The mistake to avoid is pasting a “block AI crawlers” snippet from a blog post. Most of those block the search bots too, which quietly removes you from the results you were trying to win.

Search & answers

These decide whether you can appear in an AI answer at all. Blocking one removes you from that product's results.

User agent	Company	robots.txt	What it does
OAI-SearchBot	OpenAI	honoured	Surfaces websites in ChatGPT's search features.This is the one that decides whether you can show up in ChatGPT search results. Block it and you are out.
Claude-SearchBot	Anthropic	honoured	Analyses content to improve the relevance and accuracy of search results in Claude.
PerplexityBot	Perplexity	honoured	Surfaces and links websites in Perplexity's results.Perplexity states this is not used for model training.
Applebot	Apple	honoured	Powers search in Spotlight, Siri and Safari.

Model training

These collect content that may be used to train models. Blocking them is a rights and licensing decision; it does not remove you from search or answers.

User agent	Company	robots.txt	What it does
GPTBot	OpenAI	honoured	Crawls content that may be used to train OpenAI's foundation models.The bot most often blocked by mistake in place of OAI-SearchBot. Blocking GPTBot does not affect ChatGPT search.
ClaudeBot	Anthropic	honoured	Collects web content that may contribute to training Anthropic's models.
Google-Extended	Google	honoured	Controls whether your content trains Gemini models and grounds their answers. Not a crawler of its own.Google states this has no effect on inclusion in Google Search and is not a ranking signal.
Applebot-Extended	Apple	honoured	Opts your content out of training Apple's generative models.Apple states it does not crawl pages at all, and that disallowing it leaves you in Spotlight, Siri and Safari results.
CCBot	Common Crawl	honoured	Builds the open Common Crawl corpus, which many model builders use as a training source.Blocking it reaches further than one vendor, since the corpus is public and widely reused.

User-triggered fetch

These fetch a page because a person asked about it right now. Two of them state outright that they do not follow robots.txt, because a human initiated the request.

User agent	Company	robots.txt	What it does
ChatGPT-User	OpenAI	ignored	Fetches a page live when a person or a custom GPT asks about it.OpenAI states this does not follow robots.txt, because the action is initiated by a user rather than by a crawl.
Claude-User	Anthropic	honoured	Accesses a page when a Claude user asks a question that needs it.Unlike its OpenAI and Perplexity equivalents, Anthropic states this one does honour robots.txt.
Perplexity-User	Perplexity	ignored	Visits pages to answer a question a user just asked.Perplexity states this generally ignores robots.txt, because a user initiated the request.

Ad verification

These check pages submitted as advertisements. Only relevant if you advertise on that platform.

User agent	Company	robots.txt	What it does
OAI-AdsBot	OpenAI	honoured	Validates the safety of pages submitted as ads on ChatGPT.

Three policies, as copy-paste robots.txt

Which one is right depends on what you sell, not on how you feel about AI. Should you block AI crawlers? works through it case by case.

Pick the one that matches what you actually want. Put it in the robots.txt at the root of your domain, and keep whatever rules you already have for ordinary search engines.

Open to everything

You want maximum reach and have no licensing objection to your content being used in training. The default, and the right choice for most businesses selling something.

User-agent: *
Allow: /

In the answers, out of the training

You want to be found and cited, but do not want your work used to train models. The position most publishers land on, and the one people most often implement incorrectly.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Out of all of it

Your content is the product and you sell access to it. Understand what you are choosing: you will not appear in AI answers, and that traffic goes to whoever does.

User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

After you change it, check it

A robots.txt rule that does not do what you think is worse than none, because you stop looking. Fetch your own /robots.txt in a browser and read it back. Then check your server logs for the tokens above to see which bots actually reach you — that is the ground truth, and it is the one thing no tool can tell you from the outside.

For everything downstream of access — whether the page can be understood and quoted once a bot does reach it — a free readiness scan reads your site the way these crawlers do and scores what it finds. See also is ChatGPT reading my website and why a site does not show up in AI answers.

Sources

Each vendor’s own documentation. If a token here ever disagrees with these, they win.

OpenAI: developers.openai.com/api/docs/bots
Anthropic: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Perplexity: docs.perplexity.ai/guides/bots
Google: developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Apple: support.apple.com/en-us/119829
Common Crawl: commoncrawl.org/ccbot

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?: No, and this is the most common and most expensive mistake in the whole area. GPTBot is the training crawler. The bot that decides whether you appear in ChatGPT's search results is OAI-SearchBot. Blocking GPTBot keeps your content out of training while leaving you fully visible in ChatGPT search; blocking OAI-SearchBot removes you from the answers.
Does blocking Google-Extended hurt my Google rankings?: No. Google states plainly that Google-Extended has no effect on inclusion in Google Search and is not used as a ranking signal. It only governs whether your content trains Gemini and grounds its answers.
Some of these bots ignore robots.txt. Is that allowed?: OpenAI and Perplexity both state that their user-triggered agents do not follow robots.txt, on the grounds that a person asked for that specific page rather than a crawler deciding to visit. Whether you find that reasonable is your call; the practical point is that robots.txt will not stop them, so if you need to, block at the network or server level instead.
Is a robots.txt rule enough to keep my content out of a model?: It is the standard mechanism and the major vendors say they honour it, but it is a request, not a wall. It also does nothing retroactively: content already collected has already been collected. If exclusion genuinely matters to you, treat robots.txt as one layer and not the whole answer.
Do I need llms.txt as well?: It costs nothing and cannot hurt, but be clear-eyed about the status: no major engine has committed to reading it. It will not compensate for pages that are blocked, thin or contradictory. Fix robots.txt first.