Which AI Crawlers Should You Allow in robots.txt? A Full Retrieval vs. Training Comparison and Practical Setup
Want to be cited by ChatGPT and Perplexity without feeding model training for free? The key is hidden in just a few lines of robots.txt. But many people never notice: you wrote "Allow," yet the traffic gets blocked by Cloudflare before it ever reaches your site. This piece helps you sort out every AI crawler, then teaches you to check that easily-overlooked invisible gap.
First, get the basics: AI crawlers come in three types—it's not "all on" or "all off"
Many people treat AI crawlers as one monolithic thing, and the result is either allowing everything (even your training data gets taken) or blocking everything (and getting kicked out of AI search results in the process). In reality, since 2025, OpenAI, Anthropic, and Perplexity have all split their crawlers into three roles, each with its own robots.txt identifier that you can control separately.
These three roles serve completely different purposes: training crawlers take your content to train models; retrieval crawlers index your content so it can be cited in AI answers; and user-triggered crawlers fetch your page in real time the moment a real person asks a question.
Once you understand this three-tier structure, you can make trade-offs that work in your favor—allow the ones that bring exposure, block the ones that would only take from you for free. Sorting out the roles first, then deciding whether to allow each one, is the foundation for everything that follows.
Retrieval vs. training: one comparison table to see who to allow
Once you classify the mainstream crawlers by role, the setup logic becomes clear. For most Taiwanese companies looking to do GEO (Generative Engine Optimization), the most common strategy is to allow retrieval and user-triggered crawlers, and decide on training crawlers based on brand strategy.
- Training type (feeds models, usually no direct exposure payoff): GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google Gemini training), CCBot (Common Crawl, a training source for many open-source models)
- Retrieval type (builds an index so you get cited in AI search): OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude search), PerplexityBot (Perplexity indexing)
- User-triggered type (fetches in real time when a real person asks, generally recommended to allow): ChatGPT-User, Claude-User, Perplexity-User
- Key distinction: Google-Extended only controls whether Gemini uses your content for training; it does not affect your regular Google search ranking (that's Googlebot). Don't confuse the two.
Allow or block? Three typical trade-off scenarios
There's no one right answer—it depends on your business model. The three scenarios below cover the choices most Taiwanese SMEs and content creators face.
If you make money from traffic and brand exposure (most corporate sites, blogs, SaaS): allow all retrieval and user-triggered crawlers, because being cited by ChatGPT and Perplexity is a free new exposure source. For training crawlers, it depends—if you mind your content being taken for free, block GPTBot/ClaudeBot/CCBot; if you don't mind trading it for broader model familiarity, allow them.
If you have a paywall or exclusive content (media, research reports, courses): typically block training crawlers (you don't want your content used for free training), but keep retrieval and user-triggered crawlers so AI can at least cite your summary and drive traffic back to your site. If you want nothing to do with AI at all (compliance-sensitive, purely internal sites): block everything—but be prepared, you'll gradually disappear from AI answers.
Curious how your site scores in AI's eyes?
Free scan — get your 0–100 AI-readability score and copy-paste fixes instantly.
Free GEO check →How to write robots.txt: a ready-to-use template
robots.txt sits in your site's root directory (e.g., https://yourdomain/robots.txt), and each crawler is controlled with its own User-agent block. Below is the common skeleton for "allow retrieval, block training," where each block is one User-agent line plus one Allow or Disallow line.
The rules are case-insensitive for directives, but the User-agent name must exactly match the official spelling (for example, it's OAI-SearchBot, not OpenAI-SearchBot). Once you're done, be sure to open https://yourdomain/robots.txt directly to confirm the file actually exists and isn't being redirected.
- Allow retrieval type: for OAI-SearchBot, Claude-SearchBot, and PerplexityBot, write one User-agent line and one Allow: / line each
- Allow user-triggered type: give ChatGPT-User, Claude-User, and Perplexity-User each an Allow: /
- Block training type: for GPTBot, ClaudeBot, and CCBot, write one User-agent line and one Disallow: / line each; to block Gemini training, add a Disallow: / for Google-Extended too
- Don't misuse the wildcard: the User-agent: * rule is just the default for "other unnamed crawlers"—AI crawlers already named individually use their own block and aren't affected by *
- robots.txt is "requested compliance," not technical enforcement: well-behaved big players will comply, but it can't stop crawlers that don't declare their identity (such as behavior flagged for rotating IP/UA to evade rules)
The invisible gap: robots says allow, but Cloudflare/WAF blocks it
This is the trap most people fall into yet find hardest to spot on their own. robots.txt is just a "polite preference sheet"; what really decides whether a request gets into your site is the layer further upstream—the CDN and WAF (web application firewall). If you use Cloudflare, its Bot Fight Mode is enabled by default on all plans (including the free tier) and will directly block traffic it judges to be a bot, including legitimate AI crawlers.
The result: the AI crawler never even reaches your server or reads that carefully configured robots.txt—it gets a 403 or a challenge page right at the Cloudflare layer. In recent years Cloudflare has also built in a managed "Block AI bots" rule, and this rule has higher priority than robots.txt—meaning an "Allow" in robots.txt simply can't override the WAF's block.
Rule evaluation has an order: Custom Rules run before the managed "Block AI bots" rule. So the correct fix is to create a custom Allow/Skip rule in the Cloudflare dashboard that matches User-Agents containing the crawlers you want to allow—such as OAI-SearchBot and PerplexityBot—so they're allowed through before the managed rule catches them. Sites sitting behind Cloudflare like WordPress and Webflow are especially prone to this.
How to check: a three-step verification that crawlers really get through
Just editing robots.txt isn't enough—you need to actually verify whether the request is being blocked by that upstream layer. The three steps below let you judge in a few minutes whether your setup is really taking effect.
- Step one, look at the file directly: open https://yourdomain/robots.txt in a browser and confirm the file exists, the content is correct, and it isn't redirected to a login page or homepage
- Step two, simulate a crawler fetch: use a command with the crawler's User-Agent to fetch your own homepage, e.g. curl -A "OAI-SearchBot" -I https://yourdomain/, and check whether the response is 200; if it's 403 or redirected to a challenge page, the WAF/CDN is blocking it
- Step three, compare different identities: run the same command once with a normal browser UA and once with a crawler UA—if the browser gets 200 but the crawler gets 403, you can be almost certain a bot management rule is the culprit, so add an allow rule in Cloudflare's WAF/Bot settings
- Advanced: check your server or CDN access logs to see whether OAI-SearchBot, PerplexityBot, Claude-SearchBot, etc. actually have successful 200 records; if not, it means they never got in at all
- To quickly measure your starting point, you can also first run a free GEO health check tool to see whether AI crawlers can read your pages, then work through the three steps above to rule things out one by one
FAQ
Q. I allowed GPTBot in robots.txt—why still can't ChatGPT read my site?
There are two most common causes. First, you allowed the wrong crawler: GPTBot is a training crawler, while ChatGPT's search and citations actually rely on OAI-SearchBot and ChatGPT-User—to be cited you should allow those two. Second, you're blocked at the upstream layer: even if robots.txt says allow, the bot management rules of CDN/WAF like Cloudflare have higher priority and will return a 403 before the request reaches your server. Use curl with OAI-SearchBot's User-Agent to fetch your homepage; if it isn't 200, add an allow rule in your WAF.
Q. What's the difference between retrieval and training AI crawlers, and which should I allow?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) take your content to train models and usually bring no direct exposure payoff. Retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your content so you get cited in AI answers, bringing exposure and traffic. Most sites doing GEO are advised to allow retrieval and user-triggered crawlers (ChatGPT-User, Claude-User, Perplexity-User), and decide whether to block training crawlers based on brand strategy.
Q. What is Google-Extended, and will blocking it affect my Google search ranking?
No. Google-Extended only controls whether Google can use your content to train generative AI like Gemini; writing it as Disallow simply opts you out of AI training. Indexing and ranking for regular Google search are handled by Googlebot, and the two are completely independent. So you can safely block Google-Extended to opt out of AI training while keeping Googlebot to maintain your original search ranking—neither affects the other.
Q. I use Cloudflare—robots.txt clearly allows AI crawlers but they're still blocked. What do I do?
Because Cloudflare's Bot Fight Mode is enabled by default on all plans, and the built-in managed "Block AI bots" rule has higher priority than robots.txt, the request gets blocked before reaching your server. The fix is to go to Cloudflare dashboard's Security → WAF → Custom Rules and create a custom rule that matches User-Agents containing the crawlers you want to allow (such as OAI-SearchBot, PerplexityBot), with the action set to Allow or Skip; custom rules run before the managed block rule, which lets them through.
Q. If robots.txt says Disallow, will AI companies definitely not crawl my content?
Not necessarily. robots.txt is an industry convention—it's "requested compliance," not technical enforcement. Well-behaved big players (OpenAI, Anthropic) will comply, but it can't prevent access at a technical level. There have been cases of crawlers flagged for rotating IPs and User-Agents to evade robots.txt. To truly block them, you need firewall rules at the WAF/CDN layer; a Disallow in robots.txt alone isn't enough to guarantee they won't crawl at all.
Q. Should I allow user-triggered crawlers like ChatGPT-User and Claude-User?
In most cases, yes, allow them. These crawlers aren't large-scale automated scraping—they fetch your page once in real time only when a real person asks a question in ChatGPT or Claude and needs to reference it, and they usually include a source link that drives traffic back to your site. Blocking them means giving up the chance to be cited and clicked through at the moment of a real person's query, which is usually not worth it for sites that want to build AI exposure.
Put what you learned to the test on your site in 10 seconds
Free scan — get your 0–100 AI-readability score and copy-paste fixes instantly.
Free GEO check →