AI Crawlers in robots.txt: What 8 Major Sites Block and Allow

We read the robots.txt files of NYT, Forbes, the Guardian, Wikipedia and four more. The smart ones block AI training crawlers and keep AI search crawlers open.

10 min read
Rankora cover: what 8 big sites tell AI crawlers, bar chart of 3, 2, 2 and 1 sites per robots.txt pattern

We read the robots.txt files of 8 major sites on 22 November 2026 to see how they treat AI crawlers in robots.txt. Only Wikipedia lets every AI bot in. The New York Times blocks every named AI bot. Forbes and Healthline block training crawlers but leave AI search crawlers open. Investopedia and the Guardian open the door to OpenAI and close it to Anthropic and Perplexity.

The useful finding is the split, not the blocking. The publishers with the most to lose have stopped treating "AI crawler" as one thing. They separate the bot that collects training data from the bot that builds a search index and the bot that fetches a page because a user asked. Your client sites should make the same split on purpose, instead of inheriting it from a plugin default.

Everything below comes from the live files, fetched on 22 November 2026. We parsed each one with Python's standard robots.txt parser and tested a normal article URL on each site against 10 user agents. These files change often, so treat this as a snapshot, not a permanent record.

The scoreboard: AI crawlers in robots.txt, site by site

"Yes" means the robots.txt rules allow that user agent to fetch a typical article URL. "No" means a rule disallows it. Googlebot is included as a control: all 8 sites allow it.

Heatmap of which AI crawlers 8 major sites allow in robots.txt: Wikipedia allows all, the New York Times blocks all
Fig. 01: AI crawler access by site, from robots.txt files fetched on 22 November 2026.
SiteGPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotClaude-UserPerplexityBotPerplexity-UserGoogle-Extended
nytimes.comNoNoNoNoNoNoNoNoNo
wikihow.comNoNoNoNoNoNoNoYesYes
bbc.comNoNoNoNoNoYesNoNoNo
theguardian.comYesYesYesNoNoNoNoYesYes
investopedia.comYesYesYesNoNoNoNoNoNo
healthline.comNoYesYesNoYesYesYesYesYes
forbes.comNoYesYesNoYesYesNoYesYes
en.wikipedia.orgYesYesYesYesYesYesYesYesYes

Several "Yes" cells exist only because the site never names that bot, so it falls back to the general User-agent: * rules. Under the robots.txt rules Google documents, a crawler follows the most specific group that matches its name and ignores the rest. A bot you forgot to name gets whatever your catch-all group says.

Why "AI crawler" means three different jobs

Every major AI company now runs separate user agents for training, search and user-triggered fetches, and each one reads its own robots.txt line. Blocking one does nothing to the others.

Table of training, search and user-fetch crawlers for OpenAI, Anthropic, Perplexity and Google
Fig. 02: each AI company runs separate crawlers for training, search and user-triggered fetches.
CompanyTrainingSearch indexUser-triggered fetch
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNone listedPerplexityBotPerplexity-User
GoogleGoogle-Extended (a control token, not a crawler)GooglebotNot applicable here

The vendors say this in their own docs. OpenAI's crawler documentation states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," while disallowing GPTBot signals content should not be used for training. Anthropic's help centre says blocking Claude-SearchBot or Claude-User "may reduce your site's visibility" in Claude's search results. Perplexity's crawler page says PerplexityBot surfaces sites in its search results and "is not used to crawl content for AI foundation models."

Google-Extended is the odd one out. Google's crawler documentation describes it as a product token covering Gemini training and grounding, and says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It also does not remove a site from AI Overviews; we covered that control in our breakdown of opting out of AI Overviews.

Pattern 1: lock every door (NYT, wikiHow, BBC)

The New York Times is the only site in the set that disallows all 9 AI user agents we tested. Each one gets its own group with Disallow: /, and Googlebot stays allowed. This is a deliberate trade: full Google visibility, zero AI search visibility through the front door.

wikiHow and the BBC look the same at a glance, but both files have gaps. wikiHow names ChatGPT-User, Claude-User and PerplexityBot but never names Perplexity-User or Google-Extended, so its catch-all rules apply to them. The BBC's file opens with comments forbidding retrieval-augmented generation and "AI-powered search" using its content, yet the file never names Claude-User, so that agent falls back to the general rules.

The lesson for agencies: a long block list is only as complete as its last update. Every few months a new user agent appears, and a list written once quietly goes stale.

Pattern 2: block training, keep search (Forbes, Healthline)

Forbes has the cleanest version of the split we found. Its file disallows GPTBot and ClaudeBot, then adds explicit Allow: / groups for OAI-SearchBot, Claude-SearchBot and Claude-User. Content stays out of training sets, and pages stay eligible to appear when ChatGPT or Claude search the web. You can read it yourself in Forbes's live robots.txt.

Forbes does block PerplexityBot, so the split is not applied evenly across vendors. That is a business decision the file does not explain, and we will not guess at it.

Healthline reaches a similar result by a different route. It names GPTBot, ClaudeBot, anthropic-ai, CCBot and Applebot-Extended and blocks them, with GPTBot allowed only on a /brand/ path and one drug page. It never names the search or user agents, so they inherit the open catch-all. Same outcome as Forbes, with more risk: a single tightening of the * group would shut the search bots out by accident.

Pattern 3: pick a partner (Investopedia, the Guardian)

Investopedia groups ChatGPT-User, OAI-SearchBot and GPTBot together and disallows only its /thmb/ image path, which means all three OpenAI agents can fetch articles. Claude-SearchBot, PerplexityBot and Google-Extended sit in a separate group that disallows everything apart from two narrow paths. ClaudeBot, Claude-User and Perplexity-User are blocked in a third group. The comments at the top of Investopedia's robots.txt say AI use, including retrieval-augmented generation, needs written permission. The file grants OpenAI's agents access anyway, which reads as an arrangement rather than an oversight. The file itself does not say what that arrangement is.

The Guardian reaches a similar shape through omission. It blocks PerplexityBot and all four Anthropic agents by name, and never names any OpenAI agent, so GPTBot, OAI-SearchBot and ChatGPT-User fall under its general rules and can fetch articles.

Your clients are unlikely to have content licensing deals. What this pattern shows is that publishers treat each AI engine as a separate channel with its own decision. That is the right mental model even for a 30-page service business.

Pattern 4: the open door (Wikipedia)

Wikipedia's robots.txt names no AI user agent at all. Its named groups target old scrapers and aggressive SEO crawlers, and its catch-all blocks only internal paths such as /w/ and special pages. Every AI bot we tested can fetch articles. For a site whose mission is to be read and reused, that is consistent.

What this teardown does not prove

Robots.txt is a request, not a lock. The files show intent; they do not show what any bot actually fetched. Two vendors say this directly: OpenAI's docs state that for ChatGPT-User "robots.txt rules may not apply" because a user started the request, and Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.

The files also cannot tell you how often each site gets cited in AI answers, and we did not measure that. Blocking a training crawler is forward-looking too: Anthropic says restricting ClaudeBot signals that a site's future materials should be excluded from training datasets. It says nothing about content collected earlier.

What to do with your client sites this week

  1. Read the live file, not the CMS setting. Open /robots.txt on each client domain and list every AI user agent it names. Plugins and hosting templates add AI blocks without anyone deciding to. A site crawler that reads robots.txt rules will show you which paths each group actually blocks.
  2. Decide per job, then per vendor. For most businesses that sell services, AI search visibility is a lead source, so the search and user agents should stay open. Blocking training is a separate, legitimate choice. Write both decisions down with the client.
  3. Name the search bots explicitly. Copy the Forbes approach and give OAI-SearchBot, Claude-SearchBot and PerplexityBot their own Allow: / groups. A tighter catch-all later will not lock them out by accident.
  4. Check the firewall too. A CDN or security rule that challenges bots can block an agent your robots.txt allows. If robots.txt says yes but your server logs show no hits from a vendor's published IP ranges, look there next.
  5. Measure the result. After any change, track whether the brand appears in ChatGPT, Perplexity and Gemini answers using an AI visibility tracker that monitors brand mentions, or the manual checks in our guide to spotting brand mentions in AI answers. OpenAI notes its search systems can take about 24 hours to reflect a robots.txt update.

A starting template for a site that wants AI search visibility but no training use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Adjust it to the client's actual decision. If the client wants to appear in Gemini answers grounded in Google Search, read Google's Google-Extended description first, because that token covers grounding in Gemini Apps as well as training.

FAQ

No. OpenAI documents GPTBot and OAI-SearchBot as independent settings. GPTBot controls whether crawled content may be used for training. OAI-SearchBot controls whether your pages can be shown in ChatGPT search answers. You can block the first and allow the second, which is what Forbes and Healthline do.

Which AI crawlers should a small business allow?

If AI answers could send you customers, allow the search agents: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Whether to allow training crawlers such as GPTBot and ClaudeBot is a separate choice about how your content is used. Neither choice touches your Googlebot rules. For the bigger picture, see our plain-English guide to GEO.

Does Google-Extended affect Google rankings or AI Overviews?

Google's documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It manages whether content can be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. AI Overviews are governed by Search's own controls, not by this token.

Do AI bots always obey robots.txt?

Not in every case. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person triggered the fetch, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Robots.txt reliably signals your preference to the automated crawlers that vendors say respect it. Enforcement needs firewall or server rules.

How often should I recheck a client's robots.txt for AI bots?

Quarterly at minimum, and after any CMS, plugin or hosting change. New user agents keep appearing: Anthropic and Perplexity both now run separate search and user agents alongside their main crawlers. The wikiHow and BBC files show how a block list misses agents it never names.

What to watch next

Google's crawler documentation now says it intends to support the AIPREF shared training-preference signal, written as user-agent: __usage_ai-training, in early 2027. If other vendors adopt the same signal, sites could state a training preference once instead of maintaining a list of named bots. Until then, the publishers in this teardown show the working method: name every bot you care about, and decide each one on purpose.

Tagged

AI crawlersrobots.txtGPTBotOAI-SearchBotClaudeBot

Found this useful? Share it.