0:00Introduction
Lesson 3.1: AI bots and robots.txt.
0:06What robots.txt is
An engine that can't read a page can't cite it. And the first place where you decide who gets to read is a file: robots.txt. It's a text file at the root of your site. It holds rules for each bot, called by name: this one can come in, that one can't. And it says where your sitemap is. It's a standard, RFC 9309, and the bots of the major engines respect it.
0:31Three families of bots
AI bots don't all do the same thing. There are three families. Search bots read your pages so they can cite you in their answers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and the ones from Google and Bing. Assistant bots open a page because a user asked for it, right at that moment: ChatGPT-User, Perplexity-User, Claude-User. Then there are training bots, which collect text to train the models: GPTBot, ClaudeBot, CCBot. These ones don't cite. Google-Extended is a special case: as we saw, it also affects Gemini's answers.
1:11Reading your robots.txt
Open yours: type your domain, slash, robots.txt, into your browser. First, look for a rule that blocks everyone: user-agent asterisk, and disallow slash. If it's there, no engine can read you. Then look for rules for AI bots, called by name. And check that the sitemap line is there.
1:34The most common mistakes
Many sites are blocked and don't know it. Three mistakes are the most common. The block from the staging site, still there after going live. A plugin, for security or for SEO, that adds rules for AI bots without you knowing. And misspelled names: a rule for a bot that doesn't exist does nothing at all.
1:55Your task for today
Your task for today: open your robots.txt in the browser. For each family of bots, note whether it's allowed, blocked, or has no rules. And mark the mistakes: we'll fix them in the next lesson.
2:22Next lesson
In the next lesson, we write the rules: blocking training, without disappearing from answers. See you there.