Cloudflare customers are now sending over 1 billion HTTP 402 Payment Required responses to AI crawlers every day (Cloudflare, 2026), and AI Crawl Control is free on every Cloudflare plan. For a WooCommerce store, the right configuration is not “block all AI” or “allow all AI” — it is a per-crawler policy that distinguishes search and shopping crawlers from training-only crawlers. AI bot traffic grew 187% through 2025; human traffic grew 3.1%; ChatGPT-User alone jumped 201% between February and March 2026 (Cloudflare Radar). The traffic is already on your store. The question is which bots earn their visit, which ones owe you a fee, and which ones should not be there at all.

HTTP 402 Was Reserved for Thirty Years. Cloudflare Just Used It.

HTTP 402 Payment Required has sat in the HTTP specification since the early 1990s, unused, waiting for a payment system that never arrived. Cloudflare brought it back. When an AI crawler requests a page on a domain that has set a pay-per-crawl price, the server responds with 402 instead of 200, and the crawler can either pay through Cloudflare’s settlement layer or back off. It is RFC-aligned, browser-irrelevant, and surgical: human visitors hit 200 as always, only the bots in Cloudflare’s AI Crawl Control list see the 402.

New Cloudflare domains now have known AI crawlers blocked by default — that is the safer-by-default stance the platform shifted to in 2025. Early adopters of pay-per-crawl include Condé Nast, Time, AP, BuzzFeed, Reddit, and Pinterest. The publisher set is not accidental: AI training depends on high-quality content, and Stack Overflow’s analysis pins the licensed-data demand to the projected $4.4 trillion annual AI economic contribution. WooCommerce stores are not publishers, but every product page is a structured data asset that an LLM benefits from indexing.

The WooCommerce Allow / Charge / Block Matrix

This is the configuration most articles do not write. The bots that drive ChatGPT Shopping and Perplexity Shopping referrals are not the bots that scrape pages for training corpora. Treating “AI crawlers” as one category is the mistake.

Crawler	Purpose	Referral value	Recommended action
OAI-SearchBot	Powers ChatGPT Shopping & ChatGPT Search	High — converts at ~15.9%	Allow
ChatGPT-User	On-demand fetches when a user asks ChatGPT to look at a page	High — direct user intent	Allow
GPTBot	OpenAI training corpus collection	None — training only	Charge or block
PerplexityBot	Powers Perplexity Search results	High — B2B-skewed referrals	Allow
Perplexity-User	On-demand fetches for live Perplexity queries	High — direct user intent	Allow
Claude-SearchBot	Powers Claude’s web search results	High — Claude visitors convert at ~16.8%	Allow
Claude-User	On-demand fetches for live Claude queries	High — direct user intent	Allow
ClaudeBot	Anthropic training corpus collection	None — training only	Charge or block
Google-Extended	Gemini training (independent of Googlebot)	None — training only	Charge or block
Meta-ExternalAgent	Meta AI training and assistant fetches	Mixed — limited shopping referral today	Charge (review quarterly)
CCBot (Common Crawl)	General web archive used by many AI trainers	None — training only	Charge or block
Googlebot, Bingbot	Traditional search engines	n/a — not AI crawlers	Always allow

The pattern is clean: anything ending in “SearchBot” or “-User” earns its crawl through referral traffic. Everything else is training corpus collection. The named exceptions (Google-Extended, Meta-ExternalAgent, CCBot) are explicit training crawlers that do not contribute to any user-facing search or shopping experience.

You may be interested in: ChatGPT Just Crawled Your WooCommerce Checkout Page 40 Times This Week

The Referral Economics That Drive the Decision

Blocking all AI crawlers is the lazy answer, and on a WooCommerce store it is the expensive one. ChatGPT referral sessions convert at approximately 15.9% versus 1.8% for Google Organic — between four and nine times higher than traditional visitors (Seer Interactive, HubSpot benchmarks 2025). Claude visitors convert even higher, at 16.8% per First Page Sage’s 2026 platform comparison. Those referrals only exist because OAI-SearchBot, Claude-SearchBot, and PerplexityBot index your catalog.

If you block GPTBot, you keep ChatGPT Shopping. If you block OAI-SearchBot, you lose ChatGPT Shopping. The two crawlers come from the same company and behave like different entities — because they are, operationally. Treat them that way.

The Pricing Question (And Why It Is Not the Point)

The practical price range for pay-per-crawl is $0.01-$0.05 per request. Below $0.01, the licensing signal is too weak to register with the AI vendors. Above $0.05, the training-only crawlers simply walk away — which is fine if you wanted to block them, but suboptimal if your goal was the data-licensing optionality.

For a mid-market WooCommerce store, the revenue from pay-per-crawl is not the play. Bandwidth recovery is the play. Explicit consent is the play. Establishing a price for your catalog data — so that when AI vendors do start signing licensing deals, you are already in the system — that is the play. The publishers in the early-adopter list are not getting rich on per-crawl fees; they are positioning for the bilateral licensing deals that come next.

Premium content categories — proprietary product photography, exclusive editorial, owned reviews — justify higher per-crawl prices on a per-asset basis. The default rate is a baseline; the premium rate is where this gets interesting on a B2B or luxury-goods WooCommerce store.

Cross-Reference Bot Traffic Against Real Revenue Before You Decide

The matrix above is the default playbook. Your store’s actual data may shift the recommendation. The only way to know whether allowing PerplexityBot is worth the bandwidth is to cross-reference Perplexity’s bot crawl volume against Perplexity’s referral revenue in your store. That join does not happen in Cloudflare’s dashboard; Cloudflare only sees the bot side. The merchant side lives in your order data.

Here’s how you actually do this on WooCommerce. Transmute Engine™ is a first-party Node.js server — not a plugin — that runs on your subdomain (e.g., data.yourstore.com). The inPIPE plugin captures every WooCommerce order with the session source attached; the Transmute Engine routes the enriched event to BigQuery alongside the Cloudflare bot logs. The BigQuery join — bot crawls per crawler per week vs orders attributed to each crawler’s parent platform — is the data layer that turns the per-crawler decision from a default into a decision specific to your catalog.

You may be interested in: ChatGPT Is Sending Traffic to Your WooCommerce Store and GA4 Cannot See It

Key Takeaways

• 1 billion 402 responses per day. Pay-per-crawl is not a future feature; it is a default-on capability already at scale.
• The right WooCommerce policy is per-crawler, not binary. Allow SearchBot and -User crawlers; charge or block GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, and CCBot.
• Pricing range is $0.01-$0.05 per crawl. Revenue is not the goal; bandwidth recovery and licensing positioning are.
• Googlebot and Bingbot are not affected. Traditional search rankings are not impacted by pay-per-crawl configuration.
• Cross-reference bot crawls against revenue. The Cloudflare side is half the picture; BigQuery joins it to actual order outcomes.

Frequently Asked Questions

Audit your crawler traffic in Cloudflare AI Crawl Control, set your allow/charge/block policy per bot, and cross-reference bot volume against BigQuery referral data to know which crawlers are actually driving revenue. Book a Transmute Engine trial at seresa.io if you want the BigQuery side wired up alongside the Cloudflare side.