CiteProof.co
Module 3 · Lesson 3.2 · Blocking training without disappearing from answers · 2 min

Block GPTBot without disappearing from ChatGPT

Lesson 3.2 of the free AEO and GEO course: how to block training bots (GPTBot, ClaudeBot, CCBot) without disappearing from the answers of ChatGPT, Perplexity and Claude. Robots.txt rules by bot name, the Google-Extended case, blocks from CDNs and firewalls, and how to check your logs to see who really gets through.

The video for this lesson is coming soon to YouTube. Meanwhile, the full transcript is right here.

Transcript

0:00Introduction

Lesson 3.2: blocking training, without disappearing from answers.

0:06The choice

The basic rule is simple: searchers get in. For the bots that train models, it's your call. Search and assistant bots should always be allowed: they're the ones that bring you citations. Blocking training-only bots is a legitimate choice, and it doesn't cost you any citations.

0:24The rules to write

In practice, you write two groups, calling the bots by name. In the first, the search and assistant bots, with allow. In the second, GPTBot, ClaudeBot and CCBot, with disallow. Why by name? Because a bot follows the group that carries its name, and ignores the generic one with the asterisk. That way, the permission still holds even if someone adds a general rule. For Google-Extended, it's your call, knowing that it also affects Gemini's answers.

0:56CDNs and firewalls

But robots.txt is only a request. Even before it, a CDN or a firewall can stop bots at the door. Since July 2025, for example, Cloudflare blocks known AI crawlers by default on new domains, and asks new sign-ups whether to let them through. If you use Cloudflare, check what was chosen. And also check your hosting's anti-bot protection, any web application firewalls, and your security plugins.

1:27Checking who got through

To know who really gets through, look at your server logs. Search the logs for the bots' names, or check your hosting's statistics. But be careful: anyone can pretend to be a bot, because the name in the request can be copied. That's why you verify the IP address. OpenAI publishes the addresses of its bots, and Google's bots are verified with a reverse DNS lookup.

1:52Your task for today

Your task for today: fix your robots.txt with the two groups. Check your CDN and hosting settings. And find at least one visit from a search bot in your logs.

2:17Next lesson

In the next lesson: llms.txt, what it is and who really reads it. See you there.

Your task

Fix your robots.txt with the two groups. Check your CDN and hosting settings. And find at least one visit from a search bot in your logs.

Track it in the course worksheet.

Sources

Checked 18 October 2026.

  • RFC 9309: a crawler follows the group that names it; the «*» group applies only if no group names it.
  • Crawler documentation from OpenAI (including its published IP addresses), Anthropic, Perplexity and Google (verification with reverse DNS); rules as in AI Crawlers, robots.txt and llms.txt for WordPress Stores: Who to Let In (2026) (explicit Allow for search and assistant bots so the permission survives a catch-all rule, Disallow for training-only bots).
  • MIT Technology Review, 1 July 2025: Cloudflare blocks known AI crawlers by default on new domains and asks new sign-ups whether to let them through.
  • CiteProof: the «Recommended: search yes, training no» setting, visits verified by IP (AI bots: who may read the site).