Full Answer

The fragmentation is worse than it looks. citelayer's 62-signature database and GeoPromptTracker's 28-entry dataset overlap only partially, which means the combined universe of documented AI crawlers already exceeds either count on its own. OpenAI alone operates GPTBot for training data, OAI-SearchBot for real-time search results, ChatGPT-User for in-conversation browsing, and ChatGPT Agent for autonomous purchasing — each with different crawl behaviours and robots.txt compliance.

The situation compounds when you add Chinese AI vendors. ByteDance, Alibaba, Zhipu, and Huawei each deploy crawlers under their own naming conventions, and several have been observed ignoring robots.txt directives entirely. New signatures appear monthly, and they rarely announce themselves. For a WordPress site owner, this means a robots.txt file written six months ago is almost certainly incomplete.

Server-side detection is the practical answer. Rather than chasing individual user-agent strings, tracking at the server level lets you identify and log bot traffic patterns as they appear — including bots that disguise their user-agent. A plugin like citelayer provides the broadest single-plugin signature coverage for WordPress, but pairing it with server-side request logging closes the gap between known signatures and actual crawler behaviour. The related article on whether ChatGPT crawls actually send customers covers the revenue side of this equation.