This question comes up a lot, and it deserves a better answer than the two you usually get — “block everything, they’re stealing your content” or “block nothing, you need the visibility.”
Both are wrong for membership organisations, for reasons specific to how you work. Here’s the actual trade-off.
First: there isn’t one type of AI crawler
This is where most of the confusion starts. People talk about “blocking AI” as though it were one decision. It’s at least three, and they have completely different consequences.
Training crawlers collect content to build models. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Applebot-Extended, Bytespider and a dozen others. Blocking these means your content isn’t used to train future models. It does not affect whether you appear in AI answers today.
Retrieval bots fetch pages to answer a live user question. OAI-SearchBot, PerplexityBot, and Claude’s search crawler. These are the ones that determine whether you get cited in an AI answer. Blocking them is what actually makes you invisible.
User-triggered agents fetch a page because a specific person asked. ChatGPT-User, Claude-User. Someone has effectively clicked through to you via an assistant.
The critical point: these use separate user agents. Blocking GPTBot has zero effect on OAI-SearchBot or ChatGPT-User. You can decline to feed the training corpus while remaining fully available for citation. Most organisations don’t realise this is possible, and it’s usually the right answer.
Second: two Google-specific traps
Google-Extended is not a crawler. It’s a robots.txt token. Blocking it opts you out of Gemini training, and has no effect on Google Search whatsoever.
There is no Gemini crawler. Google grounds Gemini on content fetched by ordinary Googlebot. Which means the only way to keep your content out of Gemini’s live answers is to block Googlebot — and that would remove you from Google Search entirely.
So if your goal is “I don’t want Gemini quoting us”, the honest answer is that you can’t have that without destroying your search visibility. That’s not a satisfying answer, but it’s the true one.
The membership organisation problem
Here’s what makes this harder for you than for a commercial business.
Your public content is your influence. Position statements, guidelines, consensus documents, policy responses. You publish these precisely so they propagate. If a clinician somewhere acts on your guidance, that’s the mission working. An AI assistant relaying your position accurately, to someone who’d never have found your website, is arguably the cheapest reach you’ve ever had.
Your member content is your business model. CPD courses, the journal archive, member resources, the professional network. People pay for these. Feeding them to a model that then answers the question for free is a real problem, not a theoretical one.
Those two things want opposite policies. Which is exactly why “block everything” and “block nothing” are both wrong.
What we’d generally suggest
Allow retrieval bots everywhere. OAI-SearchBot, PerplexityBot, Claude’s search crawler. These are how you get cited. Blocking them is choosing to be absent from the answer while a commercial health site fills the gap with a summary of your own guidance.
Allow user-triggered agents. Someone asked for you specifically. That’s a visitor.
Make a deliberate decision on training crawlers. Reasonable people differ here. The case for allowing: your guidance ends up in the models clinicians will use for years, which is influence at enormous scale. The case for blocking: you get nothing back, no attribution, and no control. Neither is obviously right — but decide it on purpose rather than by leaving a plugin default in place.
Block everything from paywalled areas. All crawlers, AI and otherwise. If it’s behind a login it shouldn’t be reachable anyway, but check — misconfigured member areas are more common than you’d think, and a directory that’s supposed to require login but returns 200 to a crawler is a data protection issue before it’s an AI one.
Never block Googlebot. Whatever else you decide.
Three practical notes
robots.txt is a request, not a wall. Well-behaved crawlers respect it. Others don’t. If content genuinely must not be taken, it needs authentication, not a polite note. Treat robots.txt as a statement of intent rather than a security control.
Check what you’re already blocking. A study of the top 1,000 websites published in July 2026 found 40.9% were unreadable to GPTBot and 18.4% were dark to every AI crawler tested. A good share of that is accidental — inherited configs, plugin defaults, security rules nobody revisited. Look at your own file before you decide anything.
Your CDN may decide for you. From 15 September 2026, Cloudflare begins blocking multi-purpose crawlers by default on new signups and free-tier sites carrying ads. If you’re on Cloudflare, your effective policy may not be the one in your robots.txt. Worth checking.
The question to actually ask
Not “should we block AI?” but:
Which parts of what we publish exist to spread, and which exist to be paid for?
You almost certainly know the answer. Your crawler policy should simply reflect it — which for most membership organisations means open on guidance and policy, closed on member resources, and citable everywhere it’s free anyway.
If your current robots.txt doesn’t say that, it’s probably not saying anything you chose.
Sources: Ahrefs — llms.txt study, bot classification data · Google — crawlers and fetchers documentation
Not sure what your site currently allows? Our AI Visibility Check includes a review of your crawler policy — and lays out the trade-off for your organisation specifically, so you can make the decision rather than inherit it.
