• CodeRevolution
• Web Development

You open a server log and find a new visitor:

GPTBot/1.4

It requested three articles, received three successful responses and downloaded several hundred kilobytes of content.

Did ChatGPT cite your site?

No. At least, that log does not prove it.

GPTBot is an OpenAI crawler associated with potential model training. ChatGPT Search uses OAI-SearchBot for search discovery, while ChatGPT-User may retrieve a page after a person asks ChatGPT to visit it. A citation is another event entirely. A referral click from that citation is yet another.

Most AI visibility reports fail because they count all four events as the same thing.

This guide builds a measurement system that keeps them separate. We will track AI crawler requests at the edge and origin, add a lightweight WordPress reporting layer, verify that claimed bots are genuine, monitor citations with a repeatable query set and connect AI referrals to conversions.

To track AI search bot traffic in WordPress accurately, combine CDN or server access logs with official IP verification, repeatable citation monitoring and referral conversion analytics. A WordPress plugin is a useful reporting layer, but it cannot be the only source of truth: the request may be answered by a CDN or page cache before the application loads, while the citation itself happens on a platform you do not control.

The short version: server logs prove requests, verified IPs prove crawler identity, citation monitoring proves observed visibility, referral data proves clicks and conversion tracking proves business value. No single tool proves all five.

Why normal WordPress analytics misses AI discovery

Google Analytics, Matomo and most WordPress analytics plugins are designed around a browser loading JavaScript. Crawlers usually do not behave like normal visitors. They request the HTML directly, may not execute the tracking script and may receive a cached response before WordPress or PHP runs.

That creates three blind spots.

First, an AI crawler can retrieve an article while your analytics dashboard records zero sessions. This is normal. Bot requests should not be treated as human pageviews.

Second, an AI engine can cite your page without anyone clicking the citation. The citation creates visibility, but no request reaches your site from the user. Your analytics platform has nothing to record.

Third, a person can read an AI answer, remember your brand and visit later through search, a bookmark or a typed URL. The AI answer influenced the visit, but last-click analytics assigns it elsewhere.

Traditional traffic reports therefore capture only one part of the new discovery process. The solution is not to invent one oversized “AI traffic” number. It is to measure several related events and keep their meanings intact.

The AI visibility evidence ladder

Before configuring a plugin or parsing a log, decide what your report will call a hit.

For this article, a citation hit means a monitored answer contains a clickable URL from your domain. It does not mean a bot crawled the page. A brand mention without a link should be stored separately as a mention.

This definition sounds strict because it needs to be. If the metric changes whenever a dashboard looks disappointing, the dashboard cannot guide a content strategy.

Know which AI bot you are tracking

The phrase “GPTBot tracking” gets search volume, but GPTBot is not the bot that matters most when your goal is current ChatGPT citations.

OpenAI’s crawler documentation separates its agents by purpose. Anthropic does the same, as does Perplexity.

Google does not expose a special “AI Overview bot” that lets you separate classic Google crawling from generative Search crawling in server logs. Google’s AI features use the normal Search foundation and Googlebot. A Googlebot request is therefore not evidence that an AI Overview displayed your page.

In June 2026, Google launched dedicated generative AI performance reports in Search Console for a subset of sites. Where available, those reports are much better evidence: they show URL impressions inside AI Overviews, AI Mode and generative features in Discover.

GPTBot tracking: what a request really means

A verified GPTBot request proves that OpenAI’s training crawler reached a resource. It can help answer operational questions:

• Which sections of the site receive GPTBot requests?
• Is the crawler receiving 200, 301, 403 or 5xx responses?
• How much content is being transferred?
• Is the bot repeatedly entering duplicate archives or parameter traps?
• Did a robots.txt policy change reduce or stop future requests?

It does not prove that the content entered a training dataset. It does not show which passage was extracted. It does not prove that a current ChatGPT answer knows, mentions or cites the page.

OpenAI also notes that when a site allows both GPTBot and OAI-SearchBot, it may reuse the result of one crawl for both purposes instead of fetching the same resource twice. Use access logs to measure network activity, not to reconstruct every downstream use. The independent robots.txt policies remain the clearest expression of what the site permits.

If your business goal is citation visibility, put OAI-SearchBot and ChatGPT-User on separate rows from GPTBot. Combining them destroys the meaning of the report.

What a ClaudeBot request really means

ClaudeBot is Anthropic’s model-development crawler. Claude-SearchBot supports web search, and Claude-User handles user-initiated retrieval.

This separation gives WordPress publishers a useful policy choice: you can block future training crawls from ClaudeBot while leaving Claude’s search and user retrieval agents unblocked. Later in this guide, we will implement that policy through WordPress without adding an overly broad User-agent: Claude rule.

Perplexity has no equivalent training claim for its documented agents

Perplexity says PerplexityBot is used to surface and link websites in search, not to train foundation models. Perplexity-User handles user actions and generally ignores robots.txt because the request was initiated by a person.

That means PerplexityBot activity belongs in your search-discovery report, while Perplexity-User activity belongs in the user-directed retrieval report. Neither should be mixed into a model-training total.

Start at the highest layer that sees the request

A typical WordPress request may pass through several layers:

1. The AI client connects to a CDN or web application firewall.
2. The edge serves a cached response or forwards the request.
3. Nginx or Apache receives the origin request.
4. A full-page cache may answer before WordPress loads.
5. WordPress and PHP finally process the request.

Every lower layer sees less traffic.

A request answered from Cloudflare’s cache appears in Cloudflare data but not in your origin access log. A request answered by a server-level page cache appears in the web server log but may never reach a WordPress tracking plugin. A static /llms.txt file can be logged by Nginx while WordPress knows nothing about it.

This leads to a simple priority:

Use the highest available layer for crawl counts. Use WordPress to add editorial context, not as the only source of truth.

Find AI bots in existing Nginx or Apache logs

Before changing server configuration, check whether your normal combined access log already contains the user agent, request path, response status, response bytes and referrer.

On a typical Linux server, a first pass might look like this:

sudo zgrep -Ehi \
  'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' \
  /var/log/nginx/access.log*

For Apache on Debian or Ubuntu:

sudo zgrep -Ehi \
  'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' \
  /var/log/apache2/access.log*

Adjust the path for your hosting stack. cPanel, Plesk, RunCloud, GridPane and managed WordPress hosts often store per-domain logs elsewhere or expose them through a control panel.

Do not stop after counting matching lines. Inspect the fields:

203.0.113.42 - - [08/Aug/2026:09:14:22 +0000]
"GET /guides/wordpress-geo/ HTTP/1.1" 200 48291 "-"
"Mozilla/5.0 ... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

The useful facts are the time, client IP, canonical path, final status, response size, referrer and full user agent. A request returning 403 means the bot knocked on the door. It does not mean the content was retrieved. A 200 response containing a Cloudflare challenge page is not a successful content retrieval either.

Create a dedicated Nginx AI crawler log

Separating AI-related requests makes retention and analysis easier. The following example uses Nginx’s map and conditional access_log support.

Place the map blocks and log_format inside the http context. Place the access_log line in the relevant server block or in http if it should cover every virtual host.

map $http_user_agent $ai_agent {
	default                  "";
	~*OAI-SearchBot          "OAI-SearchBot";
	~*GPTBot                 "GPTBot";
	~*ChatGPT-User           "ChatGPT-User";
	~*Claude-SearchBot       "Claude-SearchBot";
	~*ClaudeBot              "ClaudeBot";
	~*Claude-User            "Claude-User";
	~*PerplexityBot          "PerplexityBot";
	~*Perplexity-User        "Perplexity-User";
}

map $ai_agent $log_ai_agent {
	default 1;
	""      0;
}

log_format ai_crawler
	'$time_iso8601\t$remote_addr\t$host\t$request_method\t$uri\t'
	'$status\t$body_bytes_sent\t$request_time\t'
	'$http_referer\t$http_accept\t$sent_http_content_type\t'
	'$ai_agent\t$http_user_agent';

server {
	# Existing site configuration remains here.

	access_log /var/log/nginx/ai-crawlers.log ai_crawler if=$log_ai_agent;
}

This log deliberately records $uri rather than $request_uri. The normalized URI excludes query strings, which reduces accidental storage of campaign parameters, email addresses, search terms and other sensitive values. Keep the normal access log if you need the full request for separate operational reasons.

The Accept and response Content-Type fields are useful when investigating machine-readable delivery. You can see whether a client asked for HTML, Markdown or another representation and what the server returned. That still does not prove which text the system extracted.

After editing Nginx configuration, test before reloading:

sudo nginx -t
sudo systemctl reload nginx

Never paste a configuration block into production without checking the context used by your distribution or hosting panel. Some platforms regenerate virtual host files and require custom snippets in a designated directory.

Cloudflare users must restore the real client IP correctly

If Cloudflare proxies the domain, $remote_addr may be a Cloudflare edge address unless Nginx or Apache is configured to restore the original client IP.

Cloudflare documents the CF-Connecting-IP header and recommends using the Nginx real IP module or Apache mod_remoteip with Cloudflare’s published proxy ranges. Do not trust a client-supplied header on an origin that is also reachable directly. Restrict origin access to trusted proxies and keep the trusted IP list current.

Without this configuration, every bot appears to come from Cloudflare. IP verification becomes impossible at the origin even though the user-agent report still looks convincing.

Create a dedicated Apache AI crawler log

Apache can mark matching requests with SetEnvIfNoCase and write only those requests to a second file through conditional CustomLog:

SetEnvIfNoCase User-Agent \
	"(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User)" \
	ai_crawler

LogFormat "%{%Y-%m-%dT%H:%M:%S%z}t\t%a\t%v\t%r\t%>s\t%B\t%D\t%{Referer}i\t%{Accept}i\t%{Content-Type}o\t%{User-Agent}i" ai_crawler

CustomLog "/var/log/apache2/ai-crawlers.log" ai_crawler env=ai_crawler

Apache’s mod_log_config documentation confirms that multiple logs and conditional logging are supported. The exact file path and permitted directive context depend on the server. On shared hosting, you may not be allowed to define CustomLog in .htaccess; ask the host for raw access logs instead.

Test and reload on Debian or Ubuntu:

sudo apache2ctl configtest
sudo systemctl reload apache2

Use %a after correctly configuring mod_remoteip behind Cloudflare. Otherwise, the logged address may be the reverse proxy rather than the crawler.

Cloudflare AI Crawl Control is the fastest no-code view

Sites already proxied through Cloudflare have another option. AI Crawl Control is available across Cloudflare plans and provides a dashboard for AI crawler activity, requested paths, status codes, bandwidth and allow/block decisions.

Its AI traffic analytics can separate GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and other known agents. It also shows successful and unsuccessful requests, response formats, popular paths and data transfer. Referrer reporting is available on paid plans.

There are two practical limitations.

Non-Enterprise plans currently have a maximum 24-hour analytics window, so a monthly report requires regular CSV exports or another log destination. Detection on these plans relies on user-agent patterns, while Enterprise Bot Management customers can use Cloudflare detection IDs and configurable historical ranges.

Cloudflare also sits before your origin WAF and cache. This is an advantage for visibility, but it means Cloudflare totals will not match WordPress totals. That difference is expected, not a tracking bug.

If a crawler is marked “Allow” in AI Crawl Control but still receives 403, review earlier WAF, country, rate-limit and bot rules. Cloudflare’s own documentation notes that upstream rules can block an otherwise allowed crawler.

A user-agent string is not identity

Anyone can send this header:

User-Agent: GPTBot

A report based only on stripos( $user_agent, 'GPTBot' ) measures clients claiming to be GPTBot. It is useful for discovery and debugging, but it is not verified traffic.

For stronger evidence, combine the claimed user agent with the operator’s published network information:

Do not perform remote DNS lookups or download IP-range files during every WordPress page request. That turns bot tracking into a latency and availability dependency.

Verify offline while processing logs, use a scheduled job that caches the current CIDR lists or rely on a trusted edge classification. Store three states—verified, unverified and spoofed—instead of silently treating every name match as genuine.

IP lists change. A hard-coded range copied into a plugin and forgotten for two years will eventually reject legitimate crawlers or trust addresses that no longer belong to them.

Add a lightweight AI traffic logger to WordPress

Server and edge logs should remain the primary evidence. A WordPress-level logger can still be useful when editors need a dashboard tied to posts, categories and authors.

The dangerous implementation is a callback that writes one new post-meta row for every request. A busy crawler can create thousands of rows, invalidate caches and turn a visibility feature into a database problem.

A better design uses a dedicated table and aggregates repeated requests into daily buckets. The example below records the claimed user agent, purpose, normalized path, response status and count. It does not store query strings or IP addresses.

Create a normal plugin file such as wp-content/plugins/wpbay-ai-traffic/wpbay-ai-traffic.php:

<?php
/**
 * Plugin Name: WPBay AI Traffic Logger
 * Description: Aggregates claimed AI crawler requests that reach WordPress.
 * Version:     1.0.0
 */

defined( 'ABSPATH' ) || exit;

/**
 * Return supported user agents and their operational purpose.
 *
 * @return array<string, array{agent:string,purpose:string}>
 */
function wpbay_ai_traffic_agents(): array {
	return array(
		'OAI-SearchBot'    => array(
			'agent'   => 'OAI-SearchBot',
			'purpose' => 'search',
		),
		'GPTBot'           => array(
			'agent'   => 'GPTBot',
			'purpose' => 'training',
		),
		'ChatGPT-User'     => array(
			'agent'   => 'ChatGPT-User',
			'purpose' => 'user_fetch',
		),
		'Claude-SearchBot' => array(
			'agent'   => 'Claude-SearchBot',
			'purpose' => 'search',
		),
		'ClaudeBot'        => array(
			'agent'   => 'ClaudeBot',
			'purpose' => 'training',
		),
		'Claude-User'      => array(
			'agent'   => 'Claude-User',
			'purpose' => 'user_fetch',
		),
		'PerplexityBot'    => array(
			'agent'   => 'PerplexityBot',
			'purpose' => 'search',
		),
		'Perplexity-User'  => array(
			'agent'   => 'Perplexity-User',
			'purpose' => 'user_fetch',
		),
	);
}

/**
 * Create the aggregate table.
 */
function wpbay_ai_traffic_install(): void {
	global $wpdb;

	$table_name      = $wpdb->prefix . 'wpbay_ai_hits';
	$charset_collate = $wpdb->get_charset_collate();

	require_once ABSPATH . 'wp-admin/includes/upgrade.php';

	$sql = "CREATE TABLE {$table_name} (
		id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
		hit_date date NOT NULL,
		agent varchar(40) NOT NULL,
		purpose varchar(20) NOT NULL,
		path_hash char(64) NOT NULL,
		request_path text NOT NULL,
		status smallint(5) unsigned NOT NULL DEFAULT 0,
		hits bigint(20) unsigned NOT NULL DEFAULT 1,
		first_seen datetime NOT NULL,
		last_seen datetime NOT NULL,
		PRIMARY KEY  (id),
		UNIQUE KEY hit_bucket (hit_date, agent, path_hash, status),
		KEY agent_date (agent, hit_date),
		KEY purpose_date (purpose, hit_date)
	) {$charset_collate};";

	dbDelta( $sql );
}
register_activation_hook( __FILE__, 'wpbay_ai_traffic_install' );

/**
 * Identify a claimed AI crawler from its user agent.
 *
 * @param string $user_agent Full request user agent.
 * @return array{agent:string,purpose:string}|null
 */
function wpbay_ai_traffic_detect( string $user_agent ): ?array {
	foreach ( wpbay_ai_traffic_agents() as $needle => $details ) {
		if ( false !== stripos( $user_agent, $needle ) ) {
			return $details;
		}
	}

	return null;
}

/**
 * Aggregate a hit after WordPress finishes sending the response.
 */
add_action(
	'plugins_loaded',
	static function (): void {
		$user_agent = isset( $_SERVER['HTTP_USER_AGENT'] )
			? sanitize_text_field( wp_unslash( $_SERVER['HTTP_USER_AGENT'] ) )
			: '';

		$detected = wpbay_ai_traffic_detect( $user_agent );

		if ( null === $detected ) {
			return;
		}

		add_action(
			'shutdown',
			static function () use ( $detected ): void {
				global $wpdb;

				$request_uri = isset( $_SERVER['REQUEST_URI'] )
					? wp_unslash( $_SERVER['REQUEST_URI'] )
					: '/';

				$path = wp_parse_url( $request_uri, PHP_URL_PATH );
				$path = is_string( $path ) ? '/' . ltrim( $path, '/' ) : '/';
				$path = str_replace( array( "\0", "\r", "\n" ), '', $path );
				$path = substr( $path, 0, 2000 );

				$status = http_response_code();
				$status = is_int( $status ) && $status >= 100 ? $status : 200;

				$table_name = $wpdb->prefix . 'wpbay_ai_hits';
				$now_gmt    = current_time( 'mysql', true );
				$hit_date   = gmdate( 'Y-m-d' );
				$path_hash  = hash( 'sha256', $path );

				$sql = "INSERT INTO {$table_name}
					(hit_date, agent, purpose, path_hash, request_path, status, hits, first_seen, last_seen)
					VALUES (%s, %s, %s, %s, %s, %d, 1, %s, %s)
					ON DUPLICATE KEY UPDATE
						hits = hits + 1,
						last_seen = %s";

				$wpdb->query(
					$wpdb->prepare(
						$sql,
						$hit_date,
						$detected['agent'],
						$detected['purpose'],
						$path_hash,
						$path,
						$status,
						$now_gmt,
						$now_gmt,
						$now_gmt
					)
				);
			},
			0
		);
	},
	0
);

The table’s unique key collapses repeated requests for the same day, agent, path and status into one row. That is much easier to query than raw event rows.

A simple report in phpMyAdmin or another database client might use:

SELECT
	hit_date,
	agent,
	purpose,
	status,
	SUM(hits) AS total_hits
FROM wp_wpbay_ai_hits
GROUP BY hit_date, agent, purpose, status
ORDER BY hit_date DESC, total_hits DESC;

Replace wp_ with the site’s actual table prefix.

This example has deliberate limits. It performs one small database increment for every matching request that reaches WordPress. It does not verify the source IP. It does not run for responses served by Cloudflare, a host-level cache or a static web server before PHP loads. High-traffic sites should aggregate from CDN or server logs instead of writing during requests.

A production plugin should also add a retention task, capability-protected reports, CSV escaping, multisite handling and schema versioning. Raw user agents and IP data should never be exposed through a public REST route.

Map requested paths back to WordPress content

The server knows the path. WordPress knows the post.

When preparing a report, resolve the normalized URL to a post ID with url_to_postid(), then join that post to its type, author, category, publish date and update date. Do this during reporting or a background import, not during every crawler request.

This allows questions that raw access logs cannot answer:

• Are AI search crawlers concentrated on documentation, news or product pages?
• Do recently updated posts receive a new retrieval wave?
• Which authors or topic clusters attract user-triggered fetches?
• Are bots wasting requests on pagination, feeds and thin archives?
• Which priority pages have never received a verified search-crawler request?

Keep the raw path in the report even when WordPress cannot resolve it. Sitemaps, robots.txt, llms.txt, Markdown feeds, REST endpoints and deleted URLs may be among the most informative requests.

How to block ClaudeBot in WordPress without blocking Claude Search

Some publishers want Claude visibility but do not want future content collected by the dedicated model-development crawler.

Anthropic supports that choice because it exposes different tokens:

• ClaudeBot for model development;
• Claude-SearchBot for search quality;
• Claude-User for user-directed retrieval.

To block only ClaudeBot, add the following to a small site plugin or an existing functionality plugin:

<?php
/**
 * Exclude the site from ClaudeBot model-development crawls.
 *
 * Claude-SearchBot and Claude-User are intentionally not blocked.
 */
add_filter(
	'robots_txt',
	static function ( string $output, bool $public ): string {
		if ( ! $public ) {
			return $output;
		}

		$output .= "\n# Anthropic model-development crawler.\n";
		$output .= "User-agent: ClaudeBot\n";
		$output .= "Disallow: /\n";

		return $output;
	},
	20,
	2
);

WordPress documents this output filter in the robots_txt hook reference.

After activation, open the public URL directly:

https://example.com/robots.txt

Confirm that the ClaudeBot group appears once and that the response is plain text with a successful status.

Do not add a broad group such as User-agent: Claude. Robots matching is based on user-agent tokens, and an imprecise rule may not express the policy you intend.

Do not add a separate Claude-SearchBot Allow: / group merely for reassurance. Specific groups can stop inheriting restrictions from the wildcard group. If your User-agent: * rules correctly block private utility paths and leave public content crawlable, allow Claude-SearchBot by simply not disallowing it.

Why the WordPress filter may appear to do nothing

The filter changes WordPress’s virtual robots.txt response. It cannot modify a physical robots.txt file in the web root. An SEO plugin, security plugin, host or Cloudflare managed robots feature may also replace or append to the final response.

Inspect the public file, not the settings screen.

Anthropic says robots.txt is the supported opt-out signal. Blocking crawler IP addresses alone may prevent ClaudeBot from reading the policy and is not the same as declaring an opt-out. Allow it to retrieve robots.txt while disallowing the rest of the site.

The same pattern works for OpenAI training:

User-agent: GPTBot
Disallow: /

That rule does not block OAI-SearchBot. Keep the training and search decisions separate.

Track whether the robots policy worked

A policy is not complete until the response data agrees with it.

Record a deployment timestamp, then watch:

OpenAI and Perplexity say crawler-policy changes can take about 24 hours to propagate. Do not call a rule broken five minutes after publishing it.

If you enforce robots through a WAF, retain the difference between “blocked by policy” and “crawler respected robots without requesting the page.” Both reduce successful retrievals, but only the second shows voluntary compliance.

Track AI referral clicks in analytics

Crawler logs measure machines requesting content. Referral analytics measures people arriving from AI products.

OpenAI provides the cleanest documented signal. Its publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. In GA4, inspect Session source rather than relying only on the default channel group.

Create an exploration or report filter that retains the raw source and referrer. A practical domain pattern can include:

chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.(ai|com)|gemini\.google\.com|copilot\.microsoft\.com

Platform domains and attribution behavior change. Store the full raw source in your data warehouse or report, then group it through a mapping table you can update. Do not hard-code “AI traffic” as one permanent regex inside business logic.

For ChatGPT, also create a specific segment where:

Session source exactly matches chatgpt.com

Compare engaged sessions, landing pages, newsletter signups, purchases and revenue with the site average. AI referrals are often smaller in volume than search, but the visitor may arrive after reading a detailed answer and have unusually strong intent.

Server-side referral counting

Browser analytics can be blocked by consent settings, extensions or script failures. Your access log normally records the Referer header when the browser sends one.

A first-pass search can identify known AI product domains:

sudo zgrep -Ehi \
  'https?://(chatgpt\.com|chat\.openai\.com|www\.perplexity\.ai|perplexity\.ai|claude\.ai|claude\.com|gemini\.google\.com|copilot\.microsoft\.com)' \
  /var/log/nginx/access.log*

Referrers are not guaranteed. Browser privacy rules, app webviews and redirect chains can remove them. A missing referrer does not prove the session was not AI-assisted.

Also keep referral visits out of the crawler table. One is a human click; the other is a machine retrieval.

Google AI visibility needs Search Console, not a Googlebot regex

Googlebot crawls for the whole Google Search system. There is no reliable Nginx expression that identifies “this request was for AI Mode.”

Where the new Search Generative AI report is available, Search Console can show:

• impressions inside generative AI features;
• the URLs that appeared;
• countries;
• devices for Search;
• performance over time.

Google announced the report as a limited rollout in June 2026, so not every property will have it immediately. Its data is included in the broader performance totals while the dedicated view separates generative visibility.

This is one of the few first-party impression datasets for AI answers. Export it regularly and keep it distinct from ordinary Google Search clicks. It solves a problem that server logs cannot solve: the page may appear in an AI feature without receiving a click or a new crawl at that moment.

Can you monitor LLM extraction?

Not directly from a public WordPress site.

The origin can observe a request and the response it sent. It can record the URL, headers, status, content type, bytes and timing. It cannot see which paragraphs a remote system retained, how it split the page into chunks, which embeddings it created, how it reranked those chunks or whether the final model used them.

Calling every 200 response an “LLM extraction” overstates the evidence.

Use more precise terms:

Tracking Accept and Content-Type headers can reveal format preferences. Repeated retrievals after an update can suggest that a system is refreshing its copy. Neither signal proves extraction.

Avoid hidden text, invisible tracking pixels and bot-specific content. Many crawlers will not load a pixel, and serving materially different content to machines creates trust, quality and potential search-policy problems. Make the visible page easier to retrieve and cite, then measure the outcomes you can actually observe.

Build a repeatable citation monitoring set

Crawler data answers “Did an operator request this page?” Citation monitoring answers “Did the engine cite us for the questions that matter?”

Create a fixed set of commercial and informational prompts around your site. For a WordPress plugin business, that may include category comparisons, troubleshooting questions, “best tool for” queries, compatibility questions and direct product questions.

Each test row should store:

Run the same core set on a schedule, but include a few natural wording variants. Generative answers can change between runs, locations, accounts and modes. A single successful screenshot is a useful example, not a trend.

Do not mix API results with consumer-interface results without labeling them. An API search product may use a different index, model, prompt, ranking process or citation renderer from the public application.

Respect platform terms. Automated browser scraping at scale is not a sound measurement foundation. Prefer official APIs where they represent the product you need to study, and use controlled manual checks for the consumer experience.

Citation metrics that remain understandable

Use a small set of formulas:

Citation rate =
prompts with at least one citation to your domain
÷ eligible prompts tested

Brand mention rate =
prompts that mention your brand
÷ eligible prompts tested

Successful retrieval rate =
verified crawler requests returning expected 2xx content
÷ all verified crawler requests

Crawl coverage =
priority URLs with at least one verified AI search-crawler retrieval
÷ all priority URLs

AI referral conversion rate =
conversions from AI referral sessions
÷ AI referral sessions

Report the numerator and denominator beside every rate. “Citation rate increased to 40%” means little if the query set dropped from 100 prompts to five.

Connect the full discovery funnel

The most useful report is not a giant crawler counter. It connects several datasets by date and canonical URL:

This makes diagnosis possible.

If OAI-SearchBot never requests a priority section, investigate discovery, internal links, sitemaps, crawler access and WAF rules.

If it retrieves the pages successfully but the site never appears in monitored answers, the problem is more likely relevance, evidence, extractability, authority or query selection.

If citations grow but referral clicks do not, the answer may already satisfy the user, the citation may be visually weak or the query may have low commercial intent.

If AI referrals arrive but do not convert, inspect the landing page and offer. That is a conversion problem, not a crawler problem.

The dashboard I would build for a WordPress publisher

The executive view should fit on one screen:

The technical view should add agent, purpose, verification state, path, status, response type, bytes, first seen and last seen.

The editorial view should add WordPress post ID, title, type, author, taxonomy, publish date, modified date and priority group.

Never rank authors by raw crawler hits. Bots may repeatedly request one sitemap, feed or popular old article. Normalize by eligible URLs and focus on verified search crawlers, citation outcomes and qualified visits.

Common reporting mistakes

Data retention, privacy and security

Access logs can contain IP addresses, query parameters, referrers and user-agent data. Depending on the site and jurisdiction, some of that data may be personal data.

Collect only what the report needs. Normalize paths and avoid query strings in the dedicated AI log. Restrict file permissions. Do not expose log downloads in an unauthenticated WordPress endpoint. Keep raw data for a short operational period and retain daily aggregates for trend analysis.

A practical policy might keep raw AI request logs for 30 to 90 days, verified daily aggregates for a year and citation-test records for as long as the query methodology remains comparable. The correct period depends on legal obligations, traffic volume and business needs.

Sanitize log content before rendering it in wp-admin. User agents, paths and referrers are client-controlled strings. Escape them as data and never concatenate them into HTML, SQL or shell commands.

Using CiteLure Pro for WordPress-level visibility

If you do not want to build the WordPress reporting layer yourself, CiteLure Pro includes AI crawler activity and AI referral reporting inside WordPress, along with CSV exports and crawler controls. Its current product specification covers common OpenAI, Anthropic, Perplexity, Google, Bing and other AI-related agents.

That convenience does not change the measurement hierarchy. A WordPress plugin can report requests that reach WordPress and referrals visible to the application. It cannot see a response served entirely at the CDN edge, and a user-agent match alone is not the same as verified network identity.

Use the plugin dashboard for editorial workflow and convenient trends. Use Cloudflare or server logs when you need complete request coverage and stronger technical evidence. Then use controlled citation monitoring for the outcome neither layer can observe.

Disclosure: CiteLure Pro is sold through WPBay. The tracking model in this guide applies whether you use CiteLure Pro, a custom plugin or an external log platform.

A practical 30-day implementation plan

Days 1–3: establish definitions

Write down what your team means by crawl, successful retrieval, citation, mention, referral and conversion. Build the bot-purpose map before importing data.

Choose 20 to 50 priority URLs and 30 to 100 queries that represent real discovery opportunities. Freeze the initial query set so the baseline can be repeated.

Days 4–7: capture edge and origin activity

Enable Cloudflare AI Crawl Control if the site uses Cloudflare. Confirm the server’s combined log contains user agent, path, status, bytes and referrer. Add a dedicated Nginx or Apache AI log if you control the server.

Fix real-client-IP handling behind the proxy before trying to verify bots.

Week 2: verify and classify

Separate training, search and user-triggered agents. Import the official IP ranges into an offline verification process. Review 403, 429, 5xx, redirect chains and challenge pages.

Publish the intended robots policy and record its deployment date.

Week 3: measure citations and referrals

Run the baseline query set across the chosen AI products. Store exact cited URLs, mentions and answer accuracy. Configure analytics segments for ChatGPT and other known AI referrers.

If the Google generative AI report is available in Search Console, export its baseline.

Week 4: connect the evidence

Join crawler paths, monitored citations, referral landing pages and conversions through canonical URLs. Identify one technical problem, one content gap and one conversion gap.

Do not optimize the entire archive. Fix a small priority group, repeat the same tests and compare the resulting funnel.

Frequently asked questions

How do I track GPTBot in WordPress?

Use CDN or server access logs to find the GPTBot user agent, then validate source IPs against OpenAI’s current published ranges. A WordPress plugin can record requests that reach PHP, but it will miss pages served from a CDN or server cache. Keep GPTBot in the training-crawl category rather than treating it as ChatGPT citation traffic.

Does GPTBot traffic mean ChatGPT cited my page?

No. GPTBot is associated with content that may be used for model development. ChatGPT Search uses OAI-SearchBot, while ChatGPT-User handles some user-triggered fetches. Even an OAI-SearchBot request proves retrieval, not a citation. Citation monitoring or first-party impression data is still required.

How can I block ClaudeBot in WordPress?

Add a User-agent: ClaudeBot and Disallow: / group through WordPress’s robots_txt filter, then inspect the public /robots.txt response. This targets Anthropic’s model-development crawler without automatically blocking Claude-SearchBot or Claude-User.

Can I block ClaudeBot but still appear in Claude search?

Yes, Anthropic documents separate agents for model development, search and user-directed retrieval. Block only the exact ClaudeBot token. Leave Claude-SearchBot and Claude-User unblocked if Claude discovery is wanted.

Why does GA4 show no GPTBot or ClaudeBot visits?

Most crawlers do not execute the browser JavaScript used by GA4. This is expected. Use access logs or CDN analytics for bots and reserve GA4 for human referral sessions and conversions.

What is the difference between an AI crawl and a citation hit?

An AI crawl is a machine request to your server. A citation hit is an observed AI answer linking to your domain for a defined test query. A crawler can retrieve a page without citing it, and an engine can cite an indexed page without fetching it again at the moment the answer is generated.

Can WordPress tell which paragraph an LLM extracted?

No. WordPress can record the request and the response it served. It cannot see the remote system’s chunking, embeddings, reranking or final prompt context. Treat a successful response as retrieval evidence, not extraction proof.

How do I track ChatGPT referral traffic?

OpenAI says ChatGPT appends utm_source=chatgpt.com to referral URLs. Segment Session source = chatgpt.com in analytics and retain raw referrer data in server logs. Connect those sessions to landing-page engagement, leads, orders and revenue.

Should AI bots be counted as website visitors?

No. Report machine requests and human sessions separately. Adding bot requests to pageview totals inflates traffic, distorts engagement and makes conversion rates meaningless.

Is Cloudflare enough for AI crawler tracking?

Cloudflare provides broad edge visibility, including requests served from cache, and its AI Crawl Control dashboard is an excellent starting point. Long-term history, referrers and advanced verification vary by plan. Keep regular exports or server-side aggregates when you need durable reporting.

How often should I check AI citations?

Weekly or monthly is enough for most publishers. Use the same core query set, platform, language and market each time. Run more frequent checks only around a significant content release or technical migration.

Final takeaway

AI visibility cannot be reduced to one visitor counter.

GPTBot may crawl a page for potential model development. OAI-SearchBot may retrieve it for search discovery. ChatGPT may cite it in an answer. A person may click that citation. The same person may later buy through another device or channel.

Those events are related, but they are not interchangeable.

Start with edge and server logs. Verify who actually made the request. Separate training crawlers from search crawlers and user-directed fetchers. Confirm the status and content delivered. Monitor a stable set of queries for real citations. Segment human referrals and connect them to outcomes.

When the numbers are kept honest, the new discovery funnel becomes diagnosable. You can see whether a visibility problem begins at access, crawling, retrieval, citation selection, click-through or conversion.

That is far more useful than celebrating a thousand GPTBot requests and hoping one of them means ChatGPT noticed.