Should WordPress Block AI Crawlers? GPTBot, Google-Extended, ClaudeBot and PerplexityBot Explained

• Szabi Kisded
• September 27, 2026
• WordPress
• 19 min read

The wrong question is whether WordPress should block AI crawlers. The useful question is what you want each crawler to do.

That distinction matters because the names currently turning up in access logs and robots.txt generators do very different jobs. GPTBot may collect public material for future OpenAI model training. OAI-SearchBot helps pages appear in ChatGPT Search. ClaudeBot and Claude-SearchBot are similarly separate. PerplexityBot exists to power search results rather than train foundation models. Google-Extended is stranger still: it is a control token, not a crawler you will see arriving under its own user-agent string.

Treating all of these as one category creates two predictable mistakes. Some publishers allow every bot because they want citations, without realizing that search visibility and model training can often be controlled independently. Others block every name containing “AI” and quietly remove their content from services that could have sent qualified visitors.

For most public WordPress sites, the sensible starting point is straightforward: allow the crawlers responsible for search and citations, decide separately whether to allow training crawlers, and protect private content with real access controls rather than robots.txt.

That is the short answer. The rest of the decision depends on the site you run, the value of its content and how much you expect AI discovery to matter to the business.

“AI crawler” is no longer a useful category

Traditional SEO gave site owners a relatively simple mental model. Googlebot crawled a page, Google indexed it and the page might appear in search. AI services have split that process into several activities.

A training crawler collects material that may contribute to the development or improvement of a generative model. A search crawler builds or refreshes an index used when an assistant answers a current question and links to sources. A user-triggered fetcher visits a URL because somebody explicitly asked an assistant to read it, summarize it or use it in an action.

The same company may operate all three. Blocking one does not necessarily block the others.

This is the current landscape, based on the providers’ published documentation as of September 27, 2026:

The table exposes why a blanket rule is usually too crude. A publisher may reasonably decline model-training access while still welcoming search crawlers that can display a link and send a visitor. The providers have created separate controls precisely so those two decisions do not have to be identical.

GPTBot is not the bot that puts you in ChatGPT Search

GPTBot receives the most attention, partly because its name is memorable and partly because it appeared early in the current wave of AI crawler controls. Its purpose is also widely misunderstood.

OpenAI describes GPTBot as a crawler for content that may be used to make its generative AI foundation models more useful and safe. A site-wide disallow rule tells OpenAI that the site’s content should not be used for that training purpose.

That rule does not opt the site out of ChatGPT Search. OpenAI uses a separate crawler, OAI-SearchBot, to surface websites in its search features. Its documentation is unusually direct: a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, although it can still appear as a navigational link in some circumstances.

This gives WordPress publishers a meaningful choice. If a documentation site, software company or online store wants current pages discoverable in ChatGPT but does not want future training crawls, it can allow OAI-SearchBot and block GPTBot. OpenAI says the settings are independent, even though it may reuse the result of a single crawl when both purposes are allowed.

ChatGPT-User sits in a third category. It is used for certain actions initiated by a person in ChatGPT or a Custom GPT. It is not OpenAI’s automatic search crawler, and OpenAI says robots.txt rules may not apply to these user-initiated requests. Blocking ChatGPT-User in the file is therefore not a dependable security boundary.

If a page must remain private, require authentication. If a paid download must not be retrieved without a valid purchase, validate the customer’s session or signed URL. A polite crawler instruction cannot replace either control.

Google-Extended does not control AI Overviews

Google’s terminology causes more mistakes than any other entry in this discussion.

Google-Extended sounds like an AI version of Googlebot. It is not. Google says it has no separate HTTP user-agent string. Existing Google crawlers fetch the content, while the Google-Extended token in robots.txt tells Google whether that content may be used for two sets of purposes: training future Gemini models, and grounding responses in Gemini Apps and Grounding with Google Search on Vertex AI.

Blocking it does not affect inclusion in Google Search and is not used as a Google Search ranking signal. You also will not find a Google-Extended request in an access log, because no such standalone request exists.

Most importantly, Google-Extended is not the switch for AI Overviews or AI Mode in Google Search. Google treats those as Search features. Google’s guidance for AI features says that Googlebot access controls crawling for Search, while ordinary preview directives control how much page content can be shown or used.

That leaves publishers with an awkward but important trade-off. Blocking Googlebot to escape AI Overviews also obstructs conventional Google Search. A site that wants to remain indexed but limit direct use in AI Overviews can use Google’s supported preview controls instead. nosnippet prevents page content from being used as a direct input for AI Overviews and AI Mode, but it also removes the page’s normal text snippet. max-snippet imposes a character limit across ordinary and AI search results. data-nosnippet can exclude a particular section of visible HTML.

Those controls have real SEO consequences. They should be tested page by page, not dropped across an entire WordPress site because an “AI blocker” plugin made the option sound harmless.

There is another useful clarification here: Google says a page needs to be indexed and eligible to appear with a snippet to be shown as a supporting link in AI Overviews or AI Mode. No special AI schema or machine-readable “AI text file” is required. Good crawlability, clear HTML, accurate structured data and conventional search quality signals still matter.

Anthropic separates training, search and user retrieval

Anthropic publishes three controls, and their names describe the jobs reasonably well.

ClaudeBot collects public web content that could potentially contribute to model training. Blocking it signals that future material should be excluded from Anthropic’s model-training datasets. The word “future” is important. A new robots.txt rule should not be presented as a deletion request for material already collected or incorporated into an earlier process.

Claude-SearchBot is used to improve the relevance and accuracy of Claude’s search responses. Anthropic warns that disabling it can reduce a site’s visibility in those results. A business publishing product documentation, original research or problem-solving articles should account for that cost before blocking it.

Claude-User retrieves web content in response to an individual user’s request. Unlike OpenAI and Perplexity, Anthropic says all three of its bots honor standard robots.txt directives. A site can therefore express a separate preference for user-directed Claude access, though this remains a crawler instruction rather than authentication.

Anthropic also supports the non-standard Crawl-delay directive. That can be useful if Claude’s crawling creates measurable load, but it should not become the foundation of a general traffic-control strategy. Support for Crawl-delay varies across bots. Caching, rate limiting and properly sized hosting are more predictable tools for protecting a busy WordPress origin.

PerplexityBot is a search bot, not a training bot

Perplexity’s crawler documentation is clear on its main crawler: PerplexityBot is designed to surface and link websites in Perplexity search results, and it is not used to crawl content for foundation-model training.

Blocking it is therefore a discovery decision. It may be the correct decision for a site whose pages should not appear in Perplexity, but it should not be sold as a model-training opt-out. It does not perform that job in the first place.

Perplexity-User handles requests initiated by users. It may fetch a page to answer a question and include a link. Perplexity says this fetcher generally ignores robots.txt, because the user requested the visit. Perplexity publishes separate IP ranges for both agents, which is useful when a firewall is supposed to allow the genuine service rather than anybody who types PerplexityBot into a user-agent header.

This last point applies across the industry: a user-agent string is only text supplied by the requester. It can be copied. If bot identity matters at the firewall, match the declared agent with a provider’s current published IP ranges, verified-bot data or another supported verification mechanism. Never assume that a string found in an access log proves who made the request.

The policy I would use on most public WordPress sites

For a public company site, plugin documentation portal, agency blog or WooCommerce store, I would normally preserve discovery while declining training crawls unless there were a clear reason to permit them.

That means allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot. It also means leaving Googlebot alone so the site remains eligible for ordinary Google Search, AI Overviews and AI Mode. GPTBot, ClaudeBot and Google-Extended can then be decided independently.

Here is a balanced starting policy:

# Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot
# remain governed by the site's normal public crawling rules.

# Decline automated model-training use.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The omission of explicit Allow: / groups for the search bots is intentional. On a normal public site, access is allowed unless a matching rule disallows it. Creating a bot-specific allow group can also stop that bot from inheriting useful wildcard restrictions already present in the file, such as WordPress’s /wp-admin/ rule.

This is not a universal recommendation. A publisher that actively wants broad model participation can allow the training uses too. An investigative newsroom, specialist database or subscription research business may take a more restrictive view. A private knowledge base should not rely on this file at all; it should sit behind authentication.

The decision becomes easier when it is tied to the business model.

Commerce and lead-generation sites generally benefit when assistants can discover a current product, service page or help article and cite the source. Search crawlers are likely worth allowing. Product prices, stock and availability should still come from authoritative on-site data or a controlled feed, not from a stale page copy.

Documentation sites and software vendors have a similar incentive. Being the cited answer to an implementation question can produce a high-intent visit. Clear version numbers, dated changelogs, stable URLs and visible technical content matter more than a generic promise of “AI optimization.”

Advertising-funded publishers face a harder calculation because an answer can satisfy the reader without a click. Blocking search crawlers prevents that particular discovery route, but it also makes the publisher less likely to receive citations and referral traffic from it. The right answer depends on measured value, licensing strategy and how much of the content remains useful only on the site.

Paid research, member content and private communities need actual authorization. Keep premium responses out of public HTML, feeds, sitemaps and caches. Make the application check the user’s entitlement on every protected request. Disallow: /members/ merely announces the path; it does not close the door.

How to add an AI crawler policy to WordPress

WordPress can generate a virtual robots.txt response at https://example.com/robots.txt. Developers can modify that response with the robots_txt filter.

The following small must-use plugin appends the balanced policy above. It leaves WordPress’s response untouched when the site is configured as non-public under Settings → Reading.

Create wp-content/mu-plugins/wpbay-ai-crawler-policy.php:

<?php
/**
 * Plugin Name: WPBay AI Crawler Policy
 * Description: Adds explicit AI crawler rules to WordPress robots.txt output.
 * Version: 1.0.0
 */

defined( 'ABSPATH' ) || exit;

function wpbay_add_ai_crawler_policy( $output, $public ) {
	// Preserve WordPress's site-wide privacy setting.
	if ( ! $public ) {
		return $output;
	}

	$policy = <<<'ROBOTS'
# AI search crawlers remain governed by the site's normal public rules.
# Model-training uses are declined.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
ROBOTS;

	return rtrim( $output ) . "\n\n" . $policy . "\n";
}
add_filter( 'robots_txt', 'wpbay_add_ai_crawler_policy', 20, 2 );

Do not paste this blindly into every site. It expresses one policy: search access is welcome and model-training access is declined. Change the groups to match the site owner’s actual decision.

There are two WordPress details worth checking before relying on it.

First, the robots_txt filter affects WordPress’s virtual response. If a physical robots.txt file exists in the site’s document root, the web server will normally serve that file before WordPress runs. Edit the physical file instead, or remove it after confirming that the virtual version contains everything required.

Second, an SEO plugin, security plugin, managed host or CDN may also alter or cache the response. The filter executing correctly in PHP is not proof that a crawler sees the intended result. Request the public URL:

curl -sS https://example.com/robots.txt

Check every hostname separately. www.example.com, docs.example.com and shop.example.com have separate robots.txt policies. Purge the CDN cache after a change, then request the file again from outside the origin network.

Do not confuse the robots_txt filter with WordPress’s wp_robots filter. The former controls the root text file. The latter controls robots meta directives placed in HTML. Both are useful, but they solve different problems.

When robots.txt is not enough

The Robots Exclusion Protocol is a standard way to publish crawl preferences. It is not an access-control system. RFC 9309, the formal specification, explicitly says it is not a substitute for security measures and recommends application-layer controls such as HTTP authentication when access must be restricted.

A well-behaved crawler will fetch the file and apply the matching group. A malicious scraper can ignore it. A requester can impersonate a familiar bot by copying its user-agent string. A user-triggered agent may operate under different rules from an automatic crawler. Content copied to another public site can also be discovered there even after the original source blocks crawling.

Use a WAF or CDN rule when the server must actively refuse a verified crawler. Cloudflare’s current AI Crawl Control, for example, can classify crawler activity and enforce allow or block decisions at the edge. Its own documentation also reveals a limitation: detection on the free plan is based on well-known user-agent strings, while stronger bot detection is available on higher plans. The presence of a convenient switch does not remove the need to understand how identity is established.

Hard blocking should be precise. Blocking every request containing bot, denying large cloud-provider IP ranges or challenging all non-browser traffic can interfere with search engines, uptime monitors, payment webhooks, accessibility tools and API clients. Start with logs, identify the exact traffic and test on a staging rule before applying it across production.

For genuinely sensitive material, use authentication, authorization and non-public storage. Keep drafts, customer records, license keys, private downloads and internal documentation away from anonymously accessible URLs. That advice was true before generative AI and remains true now.

What an allow or block decision can actually prove

A crawler request in a server log proves that a request reached the server with a particular user-agent string. Once the source has been verified, it can prove that the named service fetched a URL. It does not prove that the URL was indexed, quoted, cited, used for training or shown to a user.

Likewise, allowing a search bot does not guarantee a citation. The page still has to be discoverable, useful, current and suitable for the query. The service may choose another source. Blocking a training bot does not promise the removal of copies obtained before the rule changed, and it cannot govern copies hosted elsewhere.

Measure the layers separately. Edge or server logs reveal crawler requests. Manual prompt testing can reveal observed citations, although results vary by user, location and time. Analytics can identify referral visits when the service passes a referrer or campaign parameter. Conversions show whether any of those visits became commercially useful.

Combining those numbers into a single “AI visibility” total produces an impressive chart and a poor business metric. A thousand bot requests are not a thousand potential customers.

After changing a policy, record the date and retain the previous file. OpenAI says search-policy changes may take about 24 hours to propagate, and Perplexity gives a similar window. Google’s changes depend on recrawling and processing. Compare crawler volume, observed citations, referrals and conversions over several weeks rather than declaring success from the next day’s log.

Common mistakes to avoid

The most damaging configuration I see is User-agent: * followed by Disallow: /, added in an attempt to stop AI training. That blocks compliant search crawlers too. On a public commercial site, it can erase far more value than it protects.

Another common mistake is blocking Google-Extended and assuming Google AI Overviews are now disabled. Google-Extended affects specified Gemini and Vertex AI uses; Googlebot controls Search. If the goal concerns AI Overviews, use Google’s documented Search preview controls and accept their effect on ordinary snippets.

Some site owners install several crawler-control plugins at once. Each adds a different group, a CDN injects another version, and the origin retains an old physical file. The result looks correct in one WordPress settings screen but differs from the public response. Maintain one source of truth whenever possible.

Finally, avoid copying an enormous list of bot names from an undated blog post. Names, purposes and policies change. OpenAI, Anthropic and Perplexity publish bot documentation and identity information; Google maintains its crawler reference. Review those sources on a schedule and after any unexplained traffic change.

So, should WordPress block AI crawlers?

WordPress should not block AI crawlers as a single class because no coherent single class exists.

For most public sites, my default would be to keep search discovery open: allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, and preserve Googlebot access. I would then make a deliberate, documented decision about GPTBot, ClaudeBot and Google-Extended based on the owner’s position on model training and Gemini grounding.

That policy protects the option to be found without pretending that every form of machine use delivers the same value. It can be tightened for licensed or high-value editorial content and opened for projects that benefit from broad reuse.

Whatever you choose, publish the rule that matches the business decision, verify the public response, monitor what reaches the server and revisit it. AI discovery is changing too quickly for a copied robots.txt block to remain a permanent strategy.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot concerns crawling for potential model training. ChatGPT Search uses OAI-SearchBot, which has a separate robots.txt setting. OpenAI says a site can allow OAI-SearchBot while blocking GPTBot.

Will blocking Google-Extended hurt Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls specified training and grounding uses in Gemini Apps and Vertex AI. Do not block Googlebot unless you also intend to restrict ordinary Google Search crawling.

Does blocking PerplexityBot stop model training?

Perplexity says PerplexityBot is used to surface and link websites in search results and is not used to crawl content for foundation-model training. Blocking it is primarily a Perplexity search-visibility decision.

Can robots.txt protect paid WordPress content?

No. It publishes a crawl preference and is publicly readable. Protect paid or private content with authentication, authorization and server-side entitlement checks.

Can I allow only certain sections to AI search bots?

Yes. Replace the site-wide rule with path-specific directives. For example, a documentation site could allow /docs/ and disallow /account/. Test the group against the exact URLs, and remember that sensitive paths still need real access controls.

Should I block ChatGPT-User and Perplexity-User too?

You can state that preference, but do not assume it will be enforced. OpenAI says robots.txt may not apply to ChatGPT-User actions, and Perplexity says Perplexity-User generally ignores the file. Anthropic currently says its Claude bots, including Claude-User, honor the directives.

Is llms.txt a replacement for these rules?

No. llms.txt is an emerging convention for presenting a curated, machine-readable map of useful content. It is not part of the Robots Exclusion Protocol and is not an access-control or training opt-out mechanism. Use robots.txt for supported crawler preferences and real security controls for enforcement.

How can I tell whether a claimed AI bot is genuine?

Do not trust the user-agent string alone. Use the provider’s current published IP ranges, reverse-DNS procedure, signed-agent mechanism or a reputable verified-bot service where available. Verification methods differ by provider and can change.

Primary sources

• OpenAI crawler documentation
• Google common crawlers and Google-Extended
• Google AI features and website controls
• Google robots meta and snippet controls
• Anthropic crawler controls
• Perplexity crawler documentation
• WordPress robots_txt filter reference
• RFC 9309: Robots Exclusion Protocol
• Cloudflare AI Crawl Control