How to Block AI Crawlers with robots.txt (GPTBot, ClaudeBot and More)

Published 2026-10-25

AI companies send crawlers across the web to collect text for training their models and to fetch pages when people ask AI assistants questions. If you’d rather your content wasn’t used that way, robots.txt is the standard way to say so. Here’s how to do it, and the trade-offs to think about first.

Two kinds of AI crawlers

  • Training crawlers collect pages to build and improve AI models. Blocking them asks the company not to use your content for training. You lose little or no traffic.
  • Search and assistant crawlers fetch pages to answer questions in AI search tools, often with a link back to the source. Blocking them can keep your site out of those answers and cost you visitors.

Many publishers block training crawlers but allow search crawlers.

Common AI crawler names

User-agentCompanyPurpose
GPTBotOpenAIModel training
OAI-SearchBotOpenAIChatGPT search results
ChatGPT-UserOpenAIPages fetched when a user asks ChatGPT
ClaudeBotAnthropicModel training
Google-ExtendedGoogleUse in Gemini models (not Google Search)
Applebot-ExtendedAppleUse in Apple’s AI models
CCBotCommon CrawlOpen web archive widely used for AI training
PerplexityBotPerplexityPerplexity search index
meta-externalagentMetaAI training and products
BytespiderByteDanceAI training
AmazonbotAmazonAlexa and AI services

Companies add and rename crawlers from time to time, so check each company’s documentation for the current list.

The robots.txt rules

To block one crawler from the whole site, give it its own group:

User-agent: GPTBot
Disallow: /

To block several training crawlers while keeping normal search engines welcome, your file might look like this:

User-agent: *
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The robots.txt generator builds this for you with one checkbox, plus a second option for AI search crawlers.

What blocking does and doesn’t do

  • It’s a request, not a lock. Major AI companies say their crawlers respect robots.txt, but nothing technically forces a crawler to obey. Determined scrapers ignore it.
  • It isn’t retroactive. Content already collected isn’t removed from existing models.
  • Blocking Google-Extended doesn’t affect Google Search. According to Google, regular search crawling and ranking are handled by Googlebot, which you shouldn’t block.
  • Don’t block Googlebot or Bingbot unless you want to disappear from search engines.

Should you block AI crawlers?

It depends on your site. Publishers whose income depends on original writing, artists and photographers often block training crawlers. Businesses that want to be recommended by AI assistants usually allow search crawlers, since being cited in an AI answer can send visitors. A tools site or a local business gets little from blocking and may lose visibility.

After you change robots.txt

  1. Upload the file to the root of your site so it opens at yourdomain.com/robots.txt.
  2. Open that address in a browser to confirm it’s live.
  3. Run your homepage through the SEO checker to make sure robots.txt doesn’t accidentally block all crawlers and that your sitemap is listed.

Common questions

  • Can I block AI from some pages only? Yes. Replace Disallow: / with specific folders, such as Disallow: /blog/.
  • Is there a meta tag instead of robots.txt? Some sites use noai meta tags, but they aren’t an official standard and support varies. robots.txt is the widely recognised method.
  • How do I know if AI bots visit my site? Check your server or CDN logs for the user-agent names in the table above.