Module 3 · Lesson 3.4 · Pages bots can't see · 2 min

Pages AI bots can't see: JavaScript, indexing, errors

Lesson 3.4 of the AEO and GEO course: why some pages don't get read by the bots of ChatGPT, Claude and Perplexity, even though they exist. JavaScript, indexing (noindex, canonical, sitemap), errors and slow pages, multilingual sites, and three checks to run with free tools.

The video for this lesson is coming soon to YouTube. Meanwhile, the full transcript is right here.

Transcript

0:00Introduction

Lesson 3.4: pages bots can't see.

0:06Four traps

Sometimes bots can get in, but to them, the page might as well not exist. There are four common traps: JavaScript, indexing, errors and slow pages, and languages.

0:18JavaScript

The first one. Many sites only show their text after running JavaScript in the browser. But many AI bots only read the HTML that comes from the server. According to a December 2024 analysis by Vercel, the bots from OpenAI, Anthropic and Perplexity don't run JavaScript. Gemini does, because it uses Googlebot's infrastructure. The test is simple: open View page source, and look for the main text. The price, the description, the answer. If it's not there, for those bots it doesn't exist.

0:54Indexing

The second: a page that isn't in the index is unlikely to be found, and so to be cited. The typical causes: a forgotten noindex on an important page. A canonical that points to a different page. Or a sitemap that leaves out the pages that matter. In Search Console, URL Inspection and the Pages report tell you what's indexed, and why the rest isn't.

1:19Errors, speed and languages

The third and the fourth. Pages that return an error, redirect chains, slow pages: a bot that waits too long moves on. PageSpeed Insights shows you the problems. And if your site is in several languages, each version must point to the others with hreflang, so the engine uses the right one for whoever asks.

1:40Your task for today

Your task for today: take five key pages, and check in the page source that the main text is there. Then check in Search Console that they're indexed, and run them through PageSpeed Insights.

2:04Next lesson

Module 3 is done. In the next lesson: Module 4, where we work on identity, helping AI engines understand who you are. See you there.

Your task

Take five key pages, and check in the page source that the main text is there. Then check in Search Console that they're indexed, and run them through PageSpeed Insights.

Track it in the course worksheet.

Sources

Checked 18 October 2026.

Vercel, «The rise of the AI crawler», 17 December 2024: «none of the major AI crawlers currently render JavaScript» (OpenAI, Anthropic, Perplexity); Gemini uses Googlebot's infrastructure, with full JavaScript rendering. Original source: vercel.com/blog/the-rise-of-the-ai-crawler.
Search Console (URL Inspection, Pages report) and PageSpeed Insights: free Google tools.
CiteProof: broken link, meta, hreflang and structured data checks (Broken Links Finder, SEO Meta Check, Hreflang Check, Schema Audit); proposed fixes waiting for approval (Applying fixes).

End of module 3: the quiz

Five questions on the “Access” module. Get four right and you pass.

Take the module 3 quiz