GPTBot, ClaudeBot, PerplexityBot: the robots.txt lines that decide your AI traffic

Somewhere in the last two years, a handful of user-agent strings became some of the most consequential lines in your robots.txt. Whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot may crawl you now shapes whether AI assistants can recommend you, cite you, and route buyers to you.

Know what each gate controls

These crawlers feed different things: some feed model training, some feed live retrieval and answers, some feed both through different paths. The nuance that matters commercially: blocking retrieval-oriented crawling makes you invisible to the answer, not just absent from training data. If a shopper asks their assistant for "the best option under $200" and your product pages were never readable, you're not in the consideration set. Nobody logs that loss anywhere.

What we see in the wild

Our monthly scans of top e-commerce and SaaS sites keep finding the same pattern: a meaningful share of commercial sites block AI agents outright — often a blanket disallow added during the 2023 scraping panic and never revisited — while publishing zero agent directives of their own. That's a policy set by default, not by decision.

A sane default for a commercial site

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Then scope real restrictions to what genuinely shouldn't be read — account areas, internal search, endpoints that cost you money per hit. If you sell things, letting buying-adjacent crawlers read your catalog is closer to distribution than to leakage. Publishers weighing content-licensing questions have a genuinely harder trade-off; that's a business decision, not a default.

Revisit it like you'd revisit pricing

Whatever you choose, make it a decision with an owner and a review date. The assistants' share of product discovery grows every quarter; a robots.txt line written in a different era shouldn't be the thing silently deciding your position in it.