Full Answer

The robots.txt file is still the primary mechanism for controlling crawler access, but AI has split what used to be a single concern — indexing — into two distinct functions: training and search.

Training crawlers (GPTBot, CCBot, Bytespider, ChatGLM-Spider, PanguBot, Google-Extended, FacebookBot, Omgilibot) scrape content to build or fine-tune language models. They consume your pages, return no traffic, and offer no opt-out beyond robots.txt. Blocking them costs you nothing in visibility. OpenAI alone operates at least four distinct user-agents, each with a different purpose — lumping them together means accidentally blocking the ones that send customers.

Search and citation bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot) retrieve content at query time to generate cited answers. These bots drive referral traffic: when a shopper asks ChatGPT for a product recommendation, OAI-SearchBot fetches your page and links back to it. Block that bot and the recommendation disappears. A practical robots.txt uses one block of Disallow rules for each training crawler and explicit Allow rules for each search bot — the Seresa blog covers the full setup and traffic impact in ChatGPT Crawled Your WordPress Site — Did It Actually Send You a Customer?.

One caveat worth noting: AI shopping agents bypass robots.txt entirely because they use standard Chrome user-agent strings inside real browsers, so robots.txt alone won't control that channel.