AI
AI Crawlers and robots.txt: Should You Let ChatGPT and Claude Read Your Website?
By the Coast Creative team4 min read
Every website has a small text file called robots.txt that tells automated visitors what they're allowed to read. For years, it mostly mattered for Google and Bing. Now a new group of visitors reads it too: the crawlers that feed AI assistants like ChatGPT, Claude, Perplexity and Gemini.
Some business owners block all of them by reflex. Others have never looked at the file. Both can be costly. Here's what these crawlers do and how to decide which ones to allow.
Not every AI crawler does the same job
The most important thing to understand is that AI companies run different crawlers for different purposes.
Some collect pages to train future AI models. OpenAI's GPTBot and Anthropic's ClaudeBot are examples. Others fetch pages so the assistant can answer a question right now, with your page as a source. OpenAI uses OAI-SearchBot for ChatGPT search, and both OpenAI and Anthropic have separate agents that visit a page when a user asks the assistant to look something up. Perplexity uses PerplexityBot to build its search results.
That difference matters. Blocking a training crawler keeps your content out of future model training. Blocking a search crawler can keep your business out of AI answers entirely, which is a different decision with a very different cost.
What Google-Extended actually controls
Google has a token called Google-Extended. Blocking it tells Google not to use your content to train and ground its Gemini models. It does not remove you from Google Search, and it doesn't affect your rankings. Google's regular crawler, Googlebot, still handles search.
Many owners block Google-Extended thinking it protects their search presence, or avoid blocking it fearing they'll drop out of Google. Neither is true. It's a separate switch.
Why most small businesses should allow them
For a local business or a service company, being found is the whole point of the website. When someone asks an assistant to recommend a roofer, a realtor or a web designer in their city, the assistant can only mention businesses it can read about. If your robots.txt blocks every AI crawler, you've opted out of a growing share of how people find services.
Your service pages, location pages, FAQ and reviews are written to be read. Letting AI assistants read them is usually a net gain.
When blocking makes sense
There are good reasons to block some crawlers. If your business sells original content, like research, courses, recipes or journalism, you may not want it used to train models for free. Some publishers allow search crawlers while blocking training crawlers for exactly that reason.
You should also keep every crawler, AI or not, out of private areas: admin pages, customer portals, internal tools and anything behind a login. Those belong in a Disallow rule for everyone.
How robots.txt rules work
The file is a list of groups. Each group names a crawler with a User-agent line, then lists what it may and may not read with Allow and Disallow lines. A group with User-agent: * applies to every crawler that doesn't have its own group.
Here's the catch that trips people up: a crawler only follows its own group if one exists. If you give GPTBot its own group to allow it, the Disallow rules you wrote under * no longer apply to it. Repeat your private paths in every named group, or the crawler you just welcomed can wander into your admin pages.
Remember it's a request, not a lock
Reputable crawlers respect robots.txt, and the major AI companies say theirs do. But the file is a polite sign on the door, not a security system. Anything that truly needs to be private should require a login.
Check yours today
Type your domain followed by /robots.txt into a browser. If you see a line that says Disallow: / under User-agent: *, your whole website is blocked from every crawler, which sometimes happens when a developer forgets to remove a setting from a staging website. If AI crawlers are blocked and you didn't mean it, that's a quick fix.
Every website we build ships with a robots.txt that welcomes search and AI crawlers, keeps private areas private and points to the sitemap. If you'd like us to look at yours, every project gets a fixed quote before work begins.