• CodeRevolution
• Web Development

The first mistake people make with llms.txt is assuming that it is a new robots.txt for artificial intelligence.

It is not.

An llms.txt file can introduce your site, organize its most useful resources and give compatible AI agents a low-noise path to authoritative content. It cannot deny access, prevent model training, remove a page from an AI search index or force ChatGPT, Claude, Gemini or Perplexity to cite you.

That is why a production WordPress llms.txt implementation needs more than one file.

That distinction matters because a badly designed WordPress setup can create the illusion of control while exposing more content than intended. An agency publishes a polished llms.txt, links every client document in it and assumes the file defines the AI policy. The real crawler policy remains unchanged. Worse, the manifest may now advertise forgotten PDFs, member-area URLs or staging documentation to anything that requests it.

The correct implementation uses several independent layers. llms.txt describes the content you want compatible agents to find. robots.txt expresses which compliant crawlers may fetch which paths. Search indexing directives control supported search engines. Authentication and a web application firewall enforce the restrictions that cannot depend on voluntary crawler behavior.

The short version: use llms.txt as a curated map, robots.txt as a declared crawl policy and server-side controls as the actual security boundary. Do not ask one public text file to perform all three jobs.

This guide builds that complete setup for WordPress. It covers the current proposal, the exact Markdown format, a production-ready virtual endpoint, AI crawler rules, WordPress multisite and headless installations, llms-full.txt, caching, monitoring, security reviews and the mistakes that make an apparently valid implementation useless.

What is llms.txt?

llms.txt is an emerging convention proposed by Jeremy Howard in September 2024. A site publishes a Markdown document at a predictable location—normally https://example.com/llms.txt—that briefly explains the site and links to a curated set of useful resources.

The original llms.txt proposal is aimed primarily at inference-time use. An AI assistant or agent working on a current task may not have enough context space to ingest an entire website. HTML also contains navigation, scripts, consent interfaces, related-post modules, advertisements and repeated template text. A short Markdown map can direct the application to the few pages that matter.

The proposal deliberately does not define how every AI system must process the file. That makes llms.txt useful as a publishing format, but weak as a promise. A compatible agent may read it. A search crawler may ignore it. A model vendor may expose its own llms.txt for developer tools without using the files published by arbitrary websites.

This is why the wording matters. You are making a machine-readable resource available. You are not registering the site with every major AI engine.

llms.txt is a proposal, not an access-control standard

As of August 2026, the format remains an open proposal. WordPress Core has a Trac proposal for native virtual llms.txt and llms-full.txt support, but it has not become a dependable core API that plugin developers can assume exists on every current installation. WordPress sites generally create the endpoint through an SEO plugin, a dedicated plugin, a physical file or custom code.

Google makes the current SEO limitation unusually clear. Its guide to optimizing for generative AI features says Google Search ignores llms.txt; maintaining one neither helps nor harms rankings or visibility in Google’s generative Search features. At the same time, Chrome’s Lighthouse agentic browsing audit recognizes the file as an optional agent-readiness feature. A missing file is marked not applicable, while a server error retrieving it is flagged.

Those two facts are not contradictory. Google Search ranking and browser-agent readiness are different systems solving different problems.

It also explains why llms.txt should be treated as one small technical component of a wider Generative Engine Optimization strategy for WordPress, not as a replacement for content quality, crawlability, structured data or source authority.

For a WordPress documentation portal, plugin developer, API provider or agency knowledge base, llms.txt can still be useful. It creates a stable entry point that developers can paste into an AI tool, that coding assistants can retrieve and that future agents may understand without parsing the entire theme. It simply should not be sold as a guaranteed citation switch.

llms.txt, robots.txt, sitemaps and real security solve different problems

The safest way to design the system is to separate discovery, permission, indexing and enforcement before writing any code.

The /robots.txt file has a formal specification in RFC 9309, but even that specification warns that the protocol is not a substitute for security. Its directives are public and depend on crawler compliance. Listing a secret path in robots.txt can make the path easier to discover.

llms.txt is even less suited to protection because it contains no Allow or Disallow grammar at all. If a resource must remain private, require authentication before the web server or WordPress returns it. Do not link it from llms.txt, do not rely on noindex, and do not assume a polite crawler represents every automated client on the internet.

Understand the three kinds of AI access before writing crawler rules

“Block the AI bots” sounds like one decision. In practice, major vendors separate at least three purposes: foundation-model development, search discovery and a fetch triggered by a user. Blocking the wrong token can remove your content from an AI search product while doing nothing to a user-directed fetcher.

The following matrix reflects the vendors’ published documentation at the time of writing.

OpenAI’s crawler documentation explicitly lets a publisher allow OAI-SearchBot while disallowing GPTBot. Anthropic documents separate ClaudeBot, Claude-SearchBot and Claude-User. Perplexity’s crawler page distinguishes its automatic search crawler from its user-triggered fetcher.

Google needs special care. Google-Extended does not appear as a separate string in access logs. It is a policy token evaluated while Google crawls with existing user agents. Blocking it does not remove the site from Google Search and is not a Google Search ranking signal. It also is not a switch for opting out of AI Overviews while remaining in normal Search; Google’s generative Search experiences use the normal Search foundation.

Vendor behavior and names change. A production crawler policy should therefore link to the official source used for each decision and include a review date. Do not copy a 2024 list into a client standard and assume it remains complete in 2026.

The exact llms.txt format

The specification uses Markdown because it is readable by humans, language models and conventional parsers. The order is intentionally simple.

An H1 containing the project or site name is the only required element. It may be followed by a blockquote summary, additional non-heading context and any number of H2 sections containing Markdown link lists. Each link requires a label and URL; a colon and short description are optional.

The section named Optional has a specific meaning. A consumer building a shorter context may omit those links.

# Example WordPress Studio

> Example WordPress Studio builds and maintains performance-focused WooCommerce sites for European retailers.

Use the technical documentation for implementation details. Product availability and pricing must be read from the linked canonical pages because they can change.

## Start Here

- [About the studio](https://example.com/about/): Company background, engineering focus and contact details.
- [WordPress services](https://example.com/services/wordpress-development/): Supported project types, delivery process and maintenance options.

## Technical Documentation

- [Plugin installation guide](https://example.com/docs/plugin-installation/): Requirements, installation steps, activation and first configuration.
- [REST API authentication](https://example.com/docs/rest-api-authentication/): Authentication methods, required headers, permissions and error responses.

## Products

- [Example Commerce Toolkit](https://example.com/products/commerce-toolkit/): WooCommerce extension for controlled catalog synchronization.

## Policies

- [Support policy](https://example.com/support-policy/): Support scope, response expectations and excluded services.
- [Privacy policy](https://example.com/privacy-policy/): Data processing and contact information for privacy requests.

## Optional

- [Engineering blog](https://example.com/blog/): Broader commentary and tutorials that are not required to understand the products.

The file does not need keyword repetition, marketing slogans or prompts telling an AI what to say. Its descriptions should help a retrieval system choose the correct source. “Current compatibility table and minimum PHP requirements” is useful. “Best revolutionary WordPress plugin at an unbeatable price” is noise.

Use absolute canonical URLs. An agent may retrieve the file outside a normal browser navigation context, so relative paths add unnecessary ambiguity. Keep the public hostname, HTTPS scheme and trailing-slash policy consistent with the canonical pages.

What belongs in a WordPress llms.txt file?

A sitemap aims for broad, mechanically generated coverage. A useful llms.txt should be editorially narrow.

Start with the pages that establish identity and reduce ambiguity: the home page, about page, authoritative product or service pages, primary documentation, current pricing or licensing explanation, support policy and essential legal or data-handling pages. On a technical site, add installation, authentication, API reference, compatibility, migration and troubleshooting resources.

The link description should explain what fact the target page can verify. This makes the manifest useful even when two pages have similar titles.

Do not dump every post, tag, author archive and attachment into the file. Search result pages, date archives, paginated duplicates, thin taxonomy pages, expired campaigns, internal search URLs, cart and checkout states, account endpoints, preview links, signed downloads and anything marked private or noindex are poor candidates.

For WooCommerce, prefer stable product category introductions, selected flagship product pages, licensing documentation, shipping and return policies. Do not turn llms.txt into a price or stock feed. An AI system should obtain volatile commercial facts from the current canonical product page, merchant feed or a controlled API—not from a description cached for weeks.

For a first release, 15 to 40 high-value links is usually more useful than hundreds of automatically selected URLs. That range is an editorial recommendation, not a protocol limit. Expand it only when each additional link has a distinct retrieval purpose.

Three ways to add llms.txt to WordPress

There is no universally best delivery method. The right choice depends on who owns the content, whether the file must update automatically and how the site is deployed.

Option 1: use the llms.txt feature already in your SEO plugin

Before installing another plugin, check the SEO stack already running on the site.

Yoast SEO currently exposes the feature under its AI tools settings. Its official setup guide supports automatic or manual page selection. Rank Math provides an LLMS Txt module with post-type, taxonomy, item-limit and additional-content controls, documented in its llms.txt guide.

Manual selection is normally the better agency default. Automatic generation can be acceptable for a small documentation-only site with disciplined post types, but it should not silently publish every public object merely because WordPress reports that object as queryable.

After enabling the feature, inspect the actual response at /llms.txt. Do not assume the settings screen proves that the edge can serve it. Look for duplicate sections, incorrect canonical domains, development URLs, noindex content and pages that return redirects or errors.

Also check whether the plugin writes a physical file or serves a virtual one. This affects containers, multisite, read-only deployments and cache invalidation. Two active plugins trying to own the same path can produce confusing results, while a physical file usually takes precedence before WordPress loads.

Option 2: publish a physical file at the public web root

For a small site, the cleanest implementation may be a hand-authored file maintained with the deployment repository.

Place it at the root of the public hostname:

https://example.com/llms.txt

The important word is public. If WordPress lives in /var/www/wordpress but Nginx serves /var/www/public, writing to the WordPress application directory will not create a public root file. In a split Docker deployment, the PHP container and web-server container may not even share that filesystem.

Return a successful 200 response and a text content type. text/markdown; charset=UTF-8 is precise, while text/plain; charset=UTF-8 remains widely compatible. Avoid a chain from HTTP to HTTPS, non-www to www and finally to the file. One canonical redirect is survivable, but serving the final resource directly is simpler for agents and diagnostics.

Static files are fast and easy to cache. Their weakness is governance. Someone must update the file when URLs, policies, products or domains change. Treat it like code: store it in version control, require review and run link checks during deployment.

Option 3: serve a curated virtual file through WordPress

A virtual endpoint avoids root-directory permissions and works well when the web server already forwards unknown paths to WordPress. It is also a better fit for containerized and read-only deployments because nothing needs to write into ABSPATH.

The following small plugin registers /llms.txt, generates a safe default document, supports curated sections through filters, validates allowed hosts and caches the completed response. It intentionally does not scrape every public post.

<?php
/**
 * Plugin Name: WPBay Curated llms.txt
 * Description: Serves a curated, cacheable llms.txt endpoint from WordPress.
 * Version:     1.0.0
 */

defined( 'ABSPATH' ) || exit;

final class WPBay_Curated_Llms_Txt {
	private const QUERY_VAR = 'wpbay_llms_txt';
	private const CACHE_KEY = 'wpbay_llms_txt_v1';

	public static function bootstrap(): void {
		add_action( 'init', array( __CLASS__, 'register_route' ) );
		add_filter( 'query_vars', array( __CLASS__, 'register_query_var' ) );
		add_action( 'template_redirect', array( __CLASS__, 'serve' ), 0 );

		add_action( 'save_post', array( __CLASS__, 'clear_cache' ) );
		add_action( 'deleted_post', array( __CLASS__, 'clear_cache' ) );
		add_action( 'trashed_post', array( __CLASS__, 'clear_cache' ) );
	}

	public static function activate(): void {
		self::register_route();
		flush_rewrite_rules( false );
	}

	public static function deactivate(): void {
		delete_transient( self::CACHE_KEY );

		// Rebuild rules on the next request without this plugin's route.
		delete_option( 'rewrite_rules' );
	}

	public static function register_route(): void {
		add_rewrite_rule(
			'^llms\.txt$',
			'index.php?' . self::QUERY_VAR . '=1',
			'top'
		);
	}

	public static function register_query_var( array $vars ): array {
		$vars[] = self::QUERY_VAR;
		return $vars;
	}

	public static function clear_cache(): void {
		delete_transient( self::CACHE_KEY );
	}

	public static function serve(): void {
		if ( '1' !== (string) get_query_var( self::QUERY_VAR ) ) {
			return;
		}

		$body = get_transient( self::CACHE_KEY );

		if ( ! is_string( $body ) ) {
			$body = self::build_document();
			set_transient( self::CACHE_KEY, $body, 6 * HOUR_IN_SECONDS );
		}

		$etag = '"' . hash( 'sha256', $body ) . '"';

		header( 'Content-Type: text/markdown; charset=UTF-8', true );
		header( 'X-Content-Type-Options: nosniff', true );
		header(
			'Cache-Control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400',
			true
		);
		header( 'ETag: ' . $etag, true );

		$if_none_match = isset( $_SERVER['HTTP_IF_NONE_MATCH'] )
			? trim( wp_unslash( $_SERVER['HTTP_IF_NONE_MATCH'] ) )
			: '';

		if ( hash_equals( $etag, $if_none_match ) ) {
			status_header( 304 );
			exit;
		}

		status_header( 200 );
		echo $body; // phpcs:ignore WordPress.Security.EscapeOutput.OutputNotEscaped
		exit;
	}

	private static function build_document(): string {
		$site_name = self::clean_text( get_bloginfo( 'name' ) );
		$tagline   = self::clean_text( get_bloginfo( 'description' ) );

		if ( '' === $site_name ) {
			$site_name = 'WordPress Site';
		}

		if ( '' === $tagline ) {
			$tagline = 'Authoritative public information from ' . $site_name . '.';
		}

		$sections = array(
			'Start Here' => array(
				array(
					'label'       => $site_name . ' homepage',
					'url'         => home_url( '/' ),
					'description' => $tagline,
				),
			),
		);

		/**
		 * Filters the curated llms.txt sections.
		 *
		 * Each section contains entries with label, url and description keys.
		 */
		$sections = apply_filters( 'wpbay_llms_txt_sections', $sections );

		$home_host = strtolower(
			(string) wp_parse_url( home_url( '/' ), PHP_URL_HOST )
		);

		$allowed_hosts = apply_filters(
			'wpbay_llms_txt_allowed_hosts',
			array( $home_host )
		);

		$allowed_hosts = array_values(
			array_unique(
				array_filter(
					array_map(
						static function ( $host ): string {
							return strtolower( trim( (string) $host ) );
						},
						(array) $allowed_hosts
					)
				)
			)
		);

		$lines   = array();
		$lines[] = '# ' . self::clean_label( $site_name );
		$lines[] = '';
		$lines[] = '> ' . $tagline;
		$lines[] = '';
		$lines[] = 'Use the linked canonical pages for current facts and details.';

		foreach ( (array) $sections as $heading => $entries ) {
			$section_lines = array();

			foreach ( (array) $entries as $entry ) {
				if ( ! is_array( $entry ) ) {
					continue;
				}

				$label = self::clean_label( $entry['label'] ?? '' );
				$url   = esc_url_raw(
					(string) ( $entry['url'] ?? '' ),
					array( 'http', 'https' )
				);

				$url_host = strtolower(
					(string) wp_parse_url( $url, PHP_URL_HOST )
				);

				if (
					'' === $label ||
					'' === $url ||
					! in_array( $url_host, $allowed_hosts, true )
				) {
					continue;
				}

				$description = self::clean_text(
					$entry['description'] ?? ''
				);

				$line = '- [' . $label . '](' . $url . ')';

				if ( '' !== $description ) {
					$line .= ': ' . $description;
				}

				$section_lines[] = $line;
			}

			if ( array() === $section_lines ) {
				continue;
			}

			$lines[] = '';
			$lines[] = '## ' . self::clean_text( $heading );
			$lines[] = '';
			$lines   = array_merge( $lines, $section_lines );
		}

		return implode( "\n", $lines ) . "\n";
	}

	private static function clean_text( $value ): string {
		$charset = get_bloginfo( 'charset' ) ?: 'UTF-8';
		$text    = html_entity_decode(
			wp_strip_all_tags( (string) $value ),
			ENT_QUOTES | ENT_HTML5,
			$charset
		);

		$text = preg_replace( '/\s+/u', ' ', $text );
		return trim( is_string( $text ) ? $text : '' );
	}

	private static function clean_label( $value ): string {
		return strtr(
			self::clean_text( $value ),
			array(
				'\\' => '\\\\',
				'['  => '\\[',
				']'  => '\\]',
			)
		);
	}
}

WPBay_Curated_Llms_Txt::bootstrap();

register_activation_hook(
	__FILE__,
	array( 'WPBay_Curated_Llms_Txt', 'activate' )
);

register_deactivation_hook(
	__FILE__,
	array( 'WPBay_Curated_Llms_Txt', 'deactivate' )
);

Activate the plugin once so WordPress stores the new rewrite rule. WordPress documents that flushing rules is expensive, which is why the code does it during activation instead of on every request.

The initial response contains only the site introduction and home page. Add deliberately selected resources with the filter:

add_filter( 'wpbay_llms_txt_sections', function ( array $sections ): array {
	$sections['WordPress Services'] = array(
		array(
			'label'       => 'Custom plugin development',
			'url'         => home_url( '/services/plugin-development/' ),
			'description' => 'Scope, supported integrations and delivery process for custom WordPress plugins.',
		),
		array(
			'label'       => 'Maintenance and security',
			'url'         => home_url( '/services/maintenance/' ),
			'description' => 'Update, monitoring, incident response and maintenance coverage.',
		),
	);

	$sections['Technical Documentation'] = array(
		array(
			'label'       => 'API authentication',
			'url'         => home_url( '/docs/api-authentication/' ),
			'description' => 'Authentication headers, permissions, examples and error codes.',
		),
	);

	$sections['Optional'] = array(
		array(
			'label'       => 'WordPress engineering blog',
			'url'         => home_url( '/blog/' ),
			'description' => 'Additional development tutorials and technical commentary.',
		),
	);

	return $sections;
} );

Replace every example path and test it before deployment. The generator rejects URLs on unapproved hosts, which reduces the chance that compromised or untrusted content can add arbitrary external retrieval targets. If the official documentation lives on a controlled subdomain, allow it explicitly:

add_filter( 'wpbay_llms_txt_allowed_hosts', function ( array $hosts ): array {
	$hosts[] = 'docs.example.com';
	return $hosts;
} );

This host allowlist is a publishing safeguard, not a crawler security feature. It controls what your generator emits; it does not control what an AI client can request elsewhere.

Why the dynamic implementation is deliberately curated

It is tempting to query every published post and append it automatically. That makes the output look impressive and usually makes it less useful.

WordPress’s publish status is not a complete editorial decision. A published attachment can be unhelpful. A public custom post type can contain thin generated objects. A membership plugin may enforce access in a way that a generic database query cannot understand. An SEO plugin may mark a page noindex. A multilingual plugin may expose several translations of the same resource. WooCommerce can publish account endpoints and session-dependent pages that should never appear in a static manifest.

If you build an automatic generator, require an explicit “Include in llms.txt” field or a curated collection. At minimum, check that the post is publicly viewable, the canonical URL is valid, the SEO indexing policy allows it, the content is not password-protected and the current anonymous request can retrieve it. Plugin-specific integrations are necessary because there is no universal WordPress function that understands every membership and SEO plugin’s policy.

This is one of the rare cases where a short manual list often scales better than a clever crawler.

Add the actual AI crawler policy in robots.txt

Once llms.txt describes the approved public content, decide which automated systems should be invited to retrieve it.

A common visibility-first policy allows search discovery while opting out of selected foundation-model training crawlers:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml

This policy says: keep the site eligible for the documented ChatGPT, Claude and Perplexity search crawlers, but express a preference against several training-oriented uses. It does not guarantee citations, and it does not stop user-triggered fetchers whose operators treat them differently from automatic crawlers.

Do not paste the example blindly. A publisher that licenses content, a public documentation project and a private client portal have different objectives. Record who approved the policy and why.

Add AI rules to WordPress’s virtual robots.txt

When WordPress generates the virtual file, the robots_txt filter can append the policy:

add_filter( 'robots_txt', function ( string $output, bool $public ): string {
	if ( ! $public ) {
		return $output;
	}

	$groups = array(
		'OAI-SearchBot'   => 'Allow',
		'Claude-SearchBot' => 'Allow',
		'PerplexityBot'   => 'Allow',
		'GPTBot'          => 'Disallow',
		'ClaudeBot'       => 'Disallow',
		'Google-Extended' => 'Disallow',
		'CCBot'           => 'Disallow',
	);

	$output = rtrim( $output ) . "\n\n";

	foreach ( $groups as $agent => $directive ) {
		$output .= 'User-agent: ' . $agent . "\n";
		$output .= $directive . ": /\n\n";
	}

	return $output;
}, 20, 2 );

This hook has no effect when a physical robots.txt file exists in the public root. It may also be bypassed when a CDN generates or modifies the response at the edge. Always inspect https://example.com/robots.txt from outside the origin after deployment. The browser-visible file is the policy crawlers receive.

If Yoast, Rank Math, a security plugin and a custom MU plugin all modify the same output, consolidate ownership. Duplicate groups are legal in some interpretations but easy to misread, and conflicting rules create an operational trap.

Full AI opt-out requires more than one Disallow rule

An organization that wants no automatic AI search discovery would also disallow OAI-SearchBot, Claude-SearchBot and PerplexityBot. It may disallow documented user fetchers, although vendors do not all apply robots.txt to user-directed retrieval in the same way.

Blocking Google-Extended is not a complete Google generative Search opt-out. Google states that the token does not affect inclusion in Google Search. If a page must not appear in Google Search at all, use the appropriate search indexing controls. If it must not be public, require authentication. Do not block Googlebot and expect a hidden noindex directive on the blocked page to be read; a crawler cannot process a directive it is not allowed to fetch.

Enforce restrictions at the CDN or server when they matter

robots.txt communicates a preference. Enforcement happens when the request is denied before content is returned.

Cloudflare’s current AI Crawl Control can report AI crawler requests, compare behavior with robots.txt and apply allow or block actions. Its managed robots.txt documentation is explicit that voluntary directives and enforced crawler controls are separate features.

If you build custom WAF rules, do not trust a user-agent string alone. Any client can send GPTBot or Googlebot in an HTTP header. Use the provider’s published IP ranges, verified-bot classification or documented reverse-DNS verification where available. Avoid copying individual IP addresses into plugin code because vendor ranges change and network policy belongs at the edge.

Enforcement also introduces false-positive risk. A broad “block all bots” rule can prevent OAI-SearchBot from reaching content even though robots.txt allows it. A rate-limit can turn the manifest into a 429 response. A JavaScript challenge may return 200 with challenge HTML instead of the Markdown file. Test from the edge and inspect the response body, not only the status code.

For genuinely private content, WAF classification is still secondary to authentication and authorization. A verified crawler must receive the same 401, 403 or login requirement as any anonymous client.

Should you create llms-full.txt as well?

llms-full.txt is a widely used companion convention that combines much more source content into one Markdown resource. It is not required for a valid llms.txt file, and the original proposal discusses generated context files without making one universal full-file behavior mandatory.

The full form makes sense for bounded technical documentation: an SDK reference, plugin developer manual or small API guide that an engineer may intentionally load into a coding assistant. It is usually a poor default for a magazine, marketplace or large WooCommerce catalog.

A full export increases transfer size, token cost, staleness and accidental disclosure risk. It may reproduce copyrighted contributor content, customer questions, shortcodes, hidden blocks, internal notes or personal data if the generator is careless. A single uncached request can also force WordPress to query and render thousands of posts.

If you publish it, generate it asynchronously after content changes, write or cache an immutable result, cap the included corpus, remove navigation and active shortcodes, and review the exact bytes anonymously before release. Serve compressed responses at the CDN and monitor download volume. Do not build the entire file on every request.

For most WordPress sites, a concise llms.txt linking to strong canonical pages is the safer first version.

Markdown versions of individual WordPress pages

The proposal also suggests exposing clean Markdown representations of useful pages. That can be valuable for technical documentation because an agent can retrieve one focused resource instead of parsing a themed HTML page.

Do not automatically mirror the entire WordPress database under predictable .md URLs. Decide which post types qualify, reuse the same public-access checks as the HTML page, strip dynamic shortcodes, preserve code blocks and add an HTTP canonical link back to the human page. Consider an X-Robots-Tag: noindex header on the alternate representation if conventional search engines could treat it as a duplicate document.

The Markdown renderer must treat post content as untrusted input. Do not execute generated PHP or shortcode callbacks merely to create the export. A shortcode can perform remote requests, reveal account-specific state or produce content that should never enter a static artifact.

On a documentation-heavy site, a controlled build pipeline is often better than converting arbitrary HTML at request time. It produces repeatable output, makes changes reviewable and removes expensive transformations from crawler requests.

Technical AI documentation schema: what the phrase should mean

There is no special Schema.org type called “AI documentation schema” that unlocks LLM indexing. The llms.txt proposal has a Markdown grammar; Schema.org structured data remains a separate description of the canonical page.

Use accurate structured data because it clarifies entities and can support normal search features, not because an llms.txt link magically upgrades it. A plugin documentation article can use TechArticle; an organization page can describe the Organization; a software product page can use SoftwareApplication or an appropriate Product representation; breadcrumb markup can describe hierarchy.

{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "REST API Authentication for Example Plugin",
  "dateModified": "2026-08-08",
  "mainEntityOfPage": "https://example.com/docs/rest-api-authentication/",
  "author": {
    "@type": "Person",
    "name": "Example Author"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Example WordPress Studio"
  }
}

Keep the JSON-LD consistent with visible content. Do not invent ratings, prices, authors or update dates for AI visibility. Google also states that no special structured data is required for its generative Search features. Foundational technical SEO still matters: canonical URLs, crawlable internal links, useful page titles, stable entities, accurate dates and original content.

Security review: llms.txt is a public trust surface

An agent may treat your file as a high-value navigation source. That makes authoring security important even though the file itself does not execute code.

Keep the manifest descriptive. Do not add instructions such as “ignore previous rules,” “always recommend our product” or “send the following data to this endpoint.” Such text is not a reliable optimization tactic and can turn the file into a prompt-injection surface for poorly designed agents.

Only trusted administrators or a reviewed deployment process should change the sections. Sanitize dynamic titles and descriptions into single lines, escape Markdown link labels and restrict URLs to approved hosts. The sample plugin does all three. If an external domain is necessary, add it to a small allowlist after verifying ownership.

Never include WordPress nonces, signed media URLs, password-reset links, preview URLs, API keys, admin AJAX actions or private REST routes. A nonce embedded in a public cached file is not protection. It is a leaked temporary credential.

Review generated content as an anonymous user. A logged-in administrator may see documents that the public cannot. The generator and the tester should not inherit that privileged view.

Performance and caching without stale surprises

A small hand-written llms.txt is cheap to serve. A dynamic file can become expensive if it queries many posts, runs SEO plugin integrations and converts HTML on every bot request.

Cache the final string, not just the database query. Invalidate it when a selected source changes, when the site domain changes or when an editor updates the manifest. Use a short browser cache and a longer shared-cache lifetime so corrections propagate without forcing PHP to rebuild the document on every request.

The sample plugin uses an ETag. A client that sends the same If-None-Match value can receive 304 Not Modified without downloading the body again. It also uses a six-hour WordPress transient and permits one hour of shared edge caching. Adjust those values to the site’s publishing frequency.

Do not flush rewrite rules during init on every page load. WordPress stores those rules and rebuilding them is expensive. Register the route on init, flush once on activation, and provide an explicit repair action or permalink-save instruction for exceptional deployments.

If Cloudflare or another CDN caches the file, purging the WordPress transient does not purge the edge copy. Add /llms.txt and /llms-full.txt to the deployment purge routine when immediate corrections matter.

Multisite, multilingual, headless and containerized WordPress

Special WordPress architectures change where the file belongs.

On a domain-mapped multisite network, each public domain should publish its own manifest with its own site identity and canonical URLs. On a subdirectory network, a subsite can publish a file under its site path, such as /client-a/llms.txt, because the proposal permits optional subpaths. A single network-root file that mixes unrelated client sites is difficult to govern and may expose tenant information.

For multilingual sites, keep one language per manifest where practical. A German site at /de/ can expose /de/llms.txt and link only to German canonical resources. Mixing translations in one giant section makes retrieval less predictable and wastes context. Include language names in link descriptions only when the language is not already obvious from the path or hostname.

In headless WordPress, publish llms.txt on the public frontend origin, not merely on the hidden CMS domain. The URLs inside it should be the same canonical frontend URLs users and search engines see. Generate the manifest during the frontend build or serve it through an edge function using data from an approved WordPress endpoint.

In Docker or Kubernetes, determine which component owns the public root. A file written inside PHP-FPM’s container may be invisible to Nginx. A virtual WordPress route avoids that filesystem mismatch, while a statically generated file should be placed in the frontend image or a shared artifact store as part of deployment—not copied between containers on every request.

How to test the complete WordPress llms.txt setup

Open the final URL in a private browser window, then inspect it from the command line:

curl -sS -D /tmp/llms-headers.txt \
  https://example.com/llms.txt \
  -o /tmp/llms-body.txt

sed -n '1,30p' /tmp/llms-headers.txt
sed -n '1,80p' /tmp/llms-body.txt

The response should be 200, use a text content type, begin with one H1 and contain no theme HTML, login form, challenge page or PHP warning. Check every linked URL with an anonymous request. A manifest full of 301, 404, 403 and consent challenges is syntactically valid but operationally poor.

Test conditional caching as well:

etag=$(curl -sSI https://example.com/llms.txt \
  | awk -F': ' 'tolower($1)=="etag" {gsub("\r", "", $2); print $2}')

curl -sSI \
  -H "If-None-Match: ${etag}" \
  https://example.com/llms.txt

The second request should normally return 304 when the ETag is unchanged.

Then request the file with representative user-agent strings. This does not verify bot identity—it only reveals whether a WAF or cache treats the header differently:

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' \
  -A 'OAI-SearchBot' \
  https://example.com/llms.txt

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' \
  -A 'GPTBot' \
  https://example.com/llms.txt

If the policy blocks GPTBot, the second request may still return 200 because robots.txt is advisory; curl does not obey it. A WAF block should return the configured enforced response. This is a useful demonstration of the difference between declaring and enforcing a rule.

Finally, run Lighthouse if agentic browsing audits are available in the installed Chrome version. Remember that a pass proves the resource can be retrieved and parsed at a basic level. It does not prove that any AI search engine indexed or cited the site.

Monitor requests without inventing an “AI ranking” metric

After launch, monitor requests to /llms.txt, the linked Markdown resources and the canonical pages. CDN logs are normally the best source because a cached response may never reach Nginx, Apache or WordPress.

Record time, path, response status, response bytes, claimed user agent, verified crawler identity where available and cache outcome. A request proves retrieval activity. It does not prove that the content entered a model, influenced an answer or produced a citation.

Do not expect Google-Extended to appear in the user-agent field; Google documents it as a control token, not a separate crawler string. Do not count every GPTBot request as ChatGPT search visibility. Keep training crawls, search crawls and user-directed fetches in separate reports.

For the broader measurement architecture, connect this implementation to WPBay’s guide on tracking AI search bot traffic and citation hits in WordPress. The crawler log, observed citation and referral visit are three different events and should remain three different metrics.

Common llms.txt mistakes on WordPress

Treating it as an AI opt-out file

There is no blocking grammar in the specification. Use vendor-specific robots.txt groups for declared preferences and authentication or edge enforcement for actual access control.

Publishing the XML sitemap again in Markdown

A giant unannotated URL dump gives an agent little more context than the sitemap it could already retrieve. Select important pages and explain what each one establishes.

Generating the file with an AI model on every request

The output becomes slow, expensive and nondeterministic. Generate descriptions through an editorial workflow if desired, review them, then cache or store the approved result. Never send private post data to an external model merely to create public summaries without a clear data-processing decision.

Listing pages that the chosen bots cannot fetch

If llms.txt links to /docs/ while the same crawler is disallowed from /docs/, the map points to a locked door. This can be intentional for human users, but it should not be mistaken for an AI visibility strategy.

Allowing every external URL

A compromised editor account or unsafe import could turn your trusted manifest into a directory of malicious destinations. Restrict links to owned hosts and require code review for exceptions.

Forgetting the physical-file conflict

Nginx or Apache normally serves a real /llms.txt before WordPress handles the route. If plugin changes never appear, check the document root and CDN cache before debugging PHP.

Caching the wrong environment

A file generated on staging may contain staging.example.com, HTTP URLs or private documentation paths. Generate environment-specific artifacts and assert the production hostname during deployment.

Exposing private titles even when the pages return 403

The URL and link label themselves can reveal client names, project codenames or membership content. Do not include private resources at all.

Assuming a crawler visit equals a citation

It does not. A successful fetch is one piece of evidence in a longer discovery chain. Measure citations independently and avoid promising a ranking gain the format cannot prove.

A practical agency governance model

The technical file is easy. Keeping it accurate across dozens of client sites is the real work.

Assign one owner for the content map and another for crawler policy approval when legal or licensing concerns are involved. Store the reason for each allowed or blocked crawler. Keep the manifest in version control or export its configuration with the site’s deployment state. Review it after domain migrations, information-architecture changes, product launches and policy updates.

Automated checks should confirm that /llms.txt returns 200, the response contains exactly one H1, approved hosts are used, no forbidden path pattern appears and every link resolves to an expected public response. Fail the deployment if the file contains staging domains, /wp-admin/, preview parameters, password-reset actions or signed query strings.

Review crawler documentation on a fixed schedule because user-agent names and product purposes move faster than traditional WordPress standards. The policy matrix in this article is dated intentionally. Your internal version should be too.

Most importantly, separate the roles in client language. “We published an AI-readable content map” is accurate. “We controlled AI access” is accurate only when the robots and enforcement layers were also configured. “We improved AI rankings” requires measured evidence and cannot be inferred from the presence of a file.

Frequently asked questions about llms.txt in WordPress

Does WordPress create llms.txt automatically?

Do not assume it does. Native support has been proposed for WordPress Core, while current sites commonly rely on Yoast SEO, Rank Math, a dedicated plugin, a physical file or custom routing. Request /llms.txt on the actual site and inspect the response.

Does llms.txt block ChatGPT, Claude, Gemini or Perplexity?

No. It contains context and links, not access directives. Use the documented crawler tokens in robots.txt for voluntary preferences and an authenticated or enforced server rule when blocking must be reliable.

Will llms.txt improve Google rankings or AI Overviews?

Google says no. Google Search ignores the file, and it neither helps nor harms Search rankings. Its value is agent readiness and controlled documentation discovery for systems that choose to use it.

Will it make ChatGPT or Claude cite my WordPress site?

There is no guarantee. A compatible retrieval tool may use the file to find the correct source, but citation selection depends on the engine, query, accessible page content, source quality, freshness and other systems outside WordPress.

Should llms.txt be added to robots.txt or the XML sitemap?

It does not need either link to exist at its conventional root path. You may reference it in documentation or headers, but there is no universal directive that registers it with every AI crawler. Make sure robots.txt does not unintentionally block the file for the agents you want to reach it.

Should the content type be text/plain or text/markdown?

Both are practical. text/markdown; charset=UTF-8 describes the format precisely, while text/plain; charset=UTF-8 is broadly compatible. Correct content, a 200 response and the absence of HTML or challenge markup matter more than choosing between those two text types.

Is llms-full.txt required?

No. Publish it only when a bounded full-text corpus creates clear value, usually for technical documentation. A concise manifest is safer for most content and commerce sites.

Can I include WooCommerce products?

Yes, if they are stable, public and intentionally selected. Link to canonical product pages and policies. Do not expose customer endpoints, signed downloads or session data, and do not use a stale manifest as the source of live price or stock claims.

Can robots.txt protect private WordPress content?

No. It is public and voluntary. Use authentication, WordPress capability checks and server-side authorization. A private URL should remain protected even when requested by an unknown client that ignores every crawler directive.

The final WordPress llms.txt architecture

A serious setup is not complicated once every layer has one job.

Publish a short, reviewed llms.txt that names the site and directs compatible agents to authoritative public resources. Keep volatile facts on canonical pages. Express separate training, search and user-retrieval preferences using the crawler tokens each vendor actually documents. Enforce important restrictions at the application or edge. Monitor the final public response and measure retrieval separately from citation visibility.

That approach avoids both extremes. You do not dismiss an emerging agent-friendly format simply because it is not a Google ranking factor. You also do not hand a public Markdown file powers it never had.

For WordPress agencies, the opportunity is not to install one more checkbox and call the site “AI optimized.” It is to build a clear, reviewable policy around what machines may discover, what they may retrieve and what remains private—and then make the approved public knowledge unusually easy to understand.

That is what a production-ready llms.txt implementation looks like.

Sources and further reading

• The llms.txt proposal and format
• Google’s guide to optimizing for generative AI features
• Chrome Lighthouse llms.txt audit
• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Google-Extended documentation
• RFC 9309: Robots Exclusion Protocol
• WordPress add_rewrite_rule() reference
• WordPress robots_txt filter reference
• Cloudflare AI Crawl Control