Technical · June 20, 2026 · 7 min read

Should you block AI crawlers? GPTBot, ClaudeBot & PerplexityBot explained

The short answer: if you want to get cited in AI answers, don't block the crawlers that power those answers. The bots that fetch a page live to answer a user's question are the ones earning you citations, blocking them removes you from the answer entirely. Training crawlers are a separate, more defensible decision.

The two jobs AI crawlers do

Not all AI bots want the same thing. There are two distinct jobs, and conflating them is the most common mistake:

• Training crawlers collect content to train future models. Blocking these affects what a model "knows" months from now, but has no effect on today's answers.
• Retrieval / live crawlers fetch your page at the moment a user asks a question, so the engine can quote a current source. These are the ones that produce citations. Block them and you simply will not appear.

If your goal is getting cited by AI, the retrieval crawlers are the ones that matter most.

The major AI crawlers, and what each one does

• GPTBot (OpenAI) — primarily a training crawler. Controlled via User-agent: GPTBot in robots.txt.
• OAI-SearchBot (OpenAI) — indexes content for ChatGPT search results and citations. This is the one to allow if you want to show up in ChatGPT.
• ChatGPT-User (OpenAI) — fetches a page live when a user's prompt triggers a browse action. Allow it to be quotable in real time.
• ClaudeBot (Anthropic) — Anthropic's crawler. Claude-Web / ClaudeBot user agents fetch content used in responses.
• PerplexityBot (Perplexity) — indexes pages so Perplexity can cite them. Perplexity is citation-first by design, so this is a high-value bot to allow.
• Google-Extended — a robots.txt token that controls whether Google uses your content for Gemini and Vertex AI training. Critically, it does not affect classic Google Search indexing or AI Overviews — those use the normal Googlebot. Blocking Google-Extended will not remove you from AI Overviews.
• Bingbot — powers Bing's index, which feeds Microsoft Copilot. Blocking it costs you Copilot visibility.

What blocking actually costs you

Blocking a retrieval crawler is the AEO equivalent of telling Google not to index your site. You don't get privacy, you get invisibility. The engine still answers the buyer's question, it just uses your competitor as the source instead.

The asymmetry matters: a competitor who allows these crawlers becomes the cited answer for the exact queries you wanted to win. In a world where the AI answer often sits above the blue links, that's the whole game.

When blocking is reasonable

There are legitimate reasons to block specific bots:

1. Pure training crawlers, on principle. If you object to your content training models with no citation in return, blocking GPTBot (training) while allowing OAI-SearchBot and ChatGPT-User (retrieval) is a coherent stance, you stay out of training but remain quotable.
2. Server load. Aggressive crawling can strain small servers. Rate-limiting is usually a better answer than an outright block.
3. Gated or paid content. Content behind a paywall or login generally shouldn't be exposed to crawlers anyway.

A sensible default robots.txt

For most brands that want AI visibility, the goal is: allow retrieval, decide on training deliberately. A reasonable starting point:

# Allow the bots that produce live citations
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Optional: opt out of training-only crawling
User-agent: GPTBot
Disallow: /

# Sitemap helps every crawler find your best pages
Sitemap: https://example.com/sitemap.xml

A few caveats: robots.txt is a request, not enforcement, well-behaved bots honor it and others may not. Always test changes carefully, since a stray Disallow: / under the wrong user agent can quietly remove you from an entire engine.

Don't forget the flip side

Allowing the crawler is necessary but not sufficient. The bot still has to find content worth citing, direct answers, clean structure, and trustworthy claims. If AI isn't citing you even with crawlers allowed, the issue is usually one of these eight things, and a schema and content audit is the fix.

---

Not sure which crawlers can see your site, or whether your pages are even citable once they do? Revlift's AI Visibility Audit checks your crawler access, structured data, and citation share across ChatGPT, Perplexity, Gemini, and Copilot, and gives you a 90-day plan to fix what's holding you back. See our AEO services for the done-for-you version.