Does llms.txt work? What the logs say, and why we still ship one

llms.txt is a plain-text Markdown file at the root of a website that summarises the site for language models and links to its key pages. Jeremy Howard of Answer.AI proposed it on 3 September 2024, and the format is described at llmstxt.org. The idea is reasonable: give a model a clean map of your site instead of making it parse navigation, scripts and cookie banners.

The question B2B marketers ask is narrower: will adding one make ChatGPT, Perplexity or Google's AI features mention us? Two years in, there is enough data to answer.

What the data says

Ahrefs is careful to say its sample skews towards technical, SEO-aware sites, so 28% adoption is an upper bound. That makes the 97% figure more striking, not less: even on sites run by people who care about this, almost nobody fetches the file.

What Google says

Google's guidance on generative AI features in Search includes a mythbusting section with a plain line: you don't need special files or markup for generative AI search. John Mueller has described llms.txt as "not done for search" and a "temporary crutch, perhaps to save some tokens" for AI coding tools.

There is one twist. Chrome's Lighthouse team added an experimental llms.txt check to its agentic browsing audits. That is Chrome thinking about AI agents that operate a browser, not Google Search ranking anything.

Who does read it

The requests that do arrive tell you what the file is for today. In the Ahrefs data, the named AI fetchers were led by GPTBot, OpenAI's training crawler, and Claude Code, a coding agent. Agents made up the largest share of AI bot requests, ahead of training crawlers, assistants and search retrieval bots.

So llms.txt is mostly useful for agents and developer tools that land on your site and want a quick summary. If your product has an API or docs that developers will point an agent at, the file earns its keep. For a marketing site trying to get recommended in a buyer's ChatGPT answer, it is close to irrelevant.

What actually moves AI citations

For B2B software questions, AI assistants lean on pages other than yours. In a July 2026 study of 128 B2B software queries across Perplexity and OpenAI's search API, a vendor's own pages made up only 12 to 24 percent of citations, and the single most-cited domain was TechRadar. The work that changes answers looks like this:

1. Let the search crawlers in. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Check that robots.txt doesn't block OAI-SearchBot, PerplexityBot or Bingbot, and that your CDN or firewall doesn't block them either.
2. Make your pages answerable. Server-rendered text, one clear definition of what you are, and sections that open with a direct answer a model can quote on its own.
3. Be described the same way elsewhere. Comparison articles, review sites, directories, community threads and your own LinkedIn should repeat the same facts about you. Independent sources carry most of the weight.
4. Measure the answers, not the files. Run the questions your buyers ask in each assistant every month and record who gets named. That is the only number that tells you whether anything worked.

What we ship, and why

We publish an llms.txt on brandseen.app anyway. It took under an hour, it costs nothing to serve, and some agents do read it. We generate it from the same data as our FAQ section and FAQ schema, so it cannot drift out of date and contradict the page.

Our robots.txt names the AI crawlers explicitly, so an audit shows them as allowed rather than merely not mentioned:

const AI_BOTS = [
  "GPTBot", "OAI-SearchBot", "ChatGPT-User",
  "ClaudeBot", "Claude-SearchBot", "Claude-User",
  "PerplexityBot", "Perplexity-User",
  "Google-Extended", "Applebot-Extended", "Bingbot", "CCBot",
];

If a client already has an llms.txt, we leave it alone. If they don't, it goes at the bottom of the technical list, after crawler access, rendering and structured data.

Check it on your own site

You don't need a study to know whether your file matters. Search your server or CDN logs for requests to /llms.txt over the last 30 days and group them by user agent. If the count is near zero, you have your answer. We are logging ours on brandseen.app and will add the numbers to this article after 30 days.