Full Answer

The AI crawler landscape splits into two camps. Search crawlers — ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot, and Applebot-Extended — visit your site to build answers that link back to you. When a user asks ChatGPT or Perplexity a question your content covers, these crawlers are the reason your URL appears as a cited source. Blocking them is functionally the same as de-indexing from AI search.

Training crawlers work differently. GPTBot collects data for OpenAI model training, CCBot feeds Common Crawl datasets, and Bytespider scrapes for ByteDance. None of them generate traffic, citations, or visibility for your site. Bytespider is particularly aggressive — it deploys over 100,000 IPs and routinely disregards robots.txt rules, which means server-level blocking via .htaccess or a firewall is the only reliable method (Seresa, 2026). Even a correctly configured robots.txt will not stop Bytespider entirely, because it does not consistently honour the directive.

For WordPress specifically, the robots.txt file handles most compliant crawlers. Add Disallow rules for GPTBot, CCBot, and Bytespider, and leave the search crawlers unblocked. Sites running Cloudflare should also check that Bot Fight Mode is not accidentally blocking legitimate AI search bots — a common misconfiguration that makes your content invisible to AI search without any warning in your analytics. The practical test: if blocking a crawler cuts your AI search visibility, you blocked the wrong one.