Full Answer

WordPress robots.txt works by telling crawlers which parts of your site they can and can't access. Traditional search engines respect these directives, and the impact of blocking them is obvious — your pages disappear from Google. AI crawlers follow the same protocol, but the consequences are invisible. No Search Console alert, no traffic graph drop, no indexing warning. Your content just never enters the AI engine's knowledge base.

The user agents that matter are GPTBot (OpenAI/ChatGPT), Google-Extended (Gemini), ClaudeBot (Anthropic), and Bytespider (ByteDance). If your robots.txt contains a blanket User-agent: * disallow or lists any of these specifically, those AI engines can't read your content — and they can't cite what they can't read.

Here's the thing: security plugins are the most common culprit. Plugins like Wordfence, Sucuri, and iThemes frequently add [AI crawler blocks](https://seresa.io/blog/ai-data-readiness/your-robots-txt-is-deciding-whether-ai-sends-you-traffic-heres-how-to-set-it) to robots.txt as part of their default bot-protection rules. They're optimised to keep bad bots out, and their block lists don't distinguish between a scraper and an AI search engine. Site owners often don't realise the trade-off because there's no feedback mechanism.

The fix is straightforward: open your robots.txt, search for GPTBot, Google-Extended, ClaudeBot, and Bytespider, and remove any disallow rules targeting them. Then check your security plugin's bot-management settings for AI-specific blocks. For a full walkthrough, [this guide covers the WordPress-specific robots.txt configuration](https://seresa.io/blog/ai-data-readiness/your-robots-txt-might-be-blocking-your-next-customer) for AI visibility.