0:00Introduction
Lesson 3.3: llms.txt, what it is and who reads it.
0:07What llms.txt is
llms.txt is a map of your site's key pages, written for language models. It's a Markdown file at the root of your site. Jeremy Howard proposed it in September 2024, on llmstxt.org. It's a proposal, not a standard. And it doesn't block anything: who can read your site is still decided by robots.txt.
0:33The format
The format is simple. A heading with the site's name: it's the only required part. Then a one-line summary. Then sections with lists of links, and for each link, a short note on what the page contains. By convention, a final section called Optional lists the pages a model can skip.
0:52Who actually reads it
And now the honest part: who actually reads it? Google Search doesn't. Google's own guide says so, AI Overviews and AI Mode included: having one neither helps nor hurts. Lighthouse, Google's tool, checks that the file is there, as a test for agents that browse the web. Not as a ranking signal. OpenAI, Anthropic and Perplexity haven't said they use it. Today, the clearest use is a person pasting the file into an assistant, to explain a site to it.
1:24llms.txt, robots.txt and the sitemap
At the root of your site there are three files, with three different jobs. llms.txt points to the pages to read first. robots.txt decides who can read: it's the only one of the three that can stop an AI engine. And the sitemap lists all the URLs.
1:45Is it worth it?
So, is it worth it? It's worth an hour, not a strategy. It costs little, and picking your key pages is a useful exercise: it shows you which ones are weak or outdated. But on its own, it won't get you cited. You need a bot that can get in, pages that answer, and other sites that back you up. And an outdated file is worse than no file: generate it automatically, or update it along with your sitemap.
2:12How to create it
There are three ways to create it. A generator, which starts from your home page and your sitemap. You'll find a free one on the tools page at citeproof.co. A plugin: on WordPress, there are several that generate it and keep it updated. Or by hand: a Markdown file, uploaded to the root of your site. Then open your domain, slash llms.txt: you should see the file, not a 404 error.
2:39Your task for today
Your task for today: pick the twenty pages that matter most, with a note for each one. Mark the weak or outdated ones: they're the work of the next modules. And if you like, publish the file and check that it opens.
3:06Next lesson
In the next lesson: the pages bots can't see, even when they're there. See you there.