llms.txt for AI search: what the evidence shows

What people are actually doing

llms.txt was proposed by Jeremy Howard of Answer.AI, published 3 September 2024. His stated problem at the time: building the right context for an LLM from a website is ambiguous, since a crawler has to guess whether to pull the whole sitemap, which external links matter and how much source code or detail to include. His fix was to let a site owner hand the model a curated index instead of leaving it to guess what to crawl or how much of a page to keep. The spec was rewritten as a v2 on 10 August 2026, after what Howard describes as two years of adoption, and it adds a second justification: web pages are built for people, so their navigation, ads and JavaScript make converting them to clean text difficult, and even a clean page is often too large to fit inside a model's context window.

The file is plain Markdown, and the spec sets out one required part and several optional ones. An H1 with the name of the site or project is the only thing a valid file must have. Under that, an optional blockquote gives a one or two line summary. Below that, any number of plain paragraphs can add context. Then come H2 headed sections, each one a list of links in standard Markdown link format with a short note after each, pointing at the pages the file's author considers worth a model's time. A final section, labelled Optional by convention, can hold links to material that is useful but skippable, so a model working under a token budget knows what to drop first. The file sits at the domain root, yourdomain.com/llms.txt, served as plain text or Markdown, not HTML. The spec also allows one under a subpath, covering just the content below it, which is how large documentation sites scope a file to one product area.

llms-full.txt is not named in Howard's original proposal. His own tooling generated something else, a processed file called llms-ctx-full.txt, built by a command line tool named llms_txt2ctx. What actually spread across the industry under the name llms-full.txt is a related idea with different mechanics: instead of a short index that links out to pages, it is the complete text of every page in that index, cleaned of navigation and boilerplate and concatenated into one file. llms.txt is the table of contents. llms-full.txt is an attempt at the whole book in one request. The size difference can be large. Cloudflare's own llms-full.txt, fetched directly on 20 September 2026, runs to just under 60 megabytes of Markdown, next to roughly 16 kilobytes for its llms.txt. It was just over 50 megabytes four days earlier, which is its own comment on how a file like this is maintained.

Where there is a documentation platform behind a site, neither file is written by hand. Those platforms generate both automatically for anything they host: Mintlify, Fern, GitBook, Vercel Docs and Supabase Docs all do this for their customers' docs sites without the customer asking. On the content management side, the WordPress plugins Yoast SEO and Rank Math will generate an llms.txt from a site's existing sitemap data. A smaller site with no documentation platform behind it is usually just hand writing a short Markdown file, which is the entire appeal of the format: no build step, no new infrastructure, a page anyone can edit in a text box.

None of this was built with search rankings in mind. It was built so a coding assistant or a documentation tool could find the right page fast. What has actually been tested and said, specifically about whether it changes how often a brand turns up in an AI answer, is a separate question, and the evidence on it is less mixed than the volume of advice suggests.

What the evidence says

Google's own guidance says plainly that it does not use the file. Google published a new AI optimisation guide for site owners in May 2026, later updated, and the version live on 16 September 2026 (last revised 10 July 2026) states: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." On llms.txt specifically, it goes further: keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google's guide gives the reason in the same breath: its generative features are named inside the sentence about Search, so a file Search does not read has no separate door into them.

That view has been consistent from named people at Google for over a year. John Mueller, in an April 2025 Reddit thread reported by Search Engine Journal, wrote that "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)," and compared it directly to the keywords meta tag: "this is what a site-owner claims their site is about... [a]t that point, why not just check the site directly?" At Google's Search Central Live Deep Dive event in Asia Pacific on 23 July 2025, Gary Illyes said on stage that Google does not support llms.txt and has no plans to, a position Search Engine Land and Search Engine Roundtable both reported that day. Search Engine Journal separately reported Google's Amir Taboul confirming the same position at the same event.

The published measurements, checked against the studies that produced them rather than the blog posts summarising them, tell the same story. SE Ranking's own study, published on its blog on 7 November 2025, found no relationship between having the file and how often a domain gets cited: "having an LLMs.txt file didn't make a domain more likely to be cited by AI models. In fact, the model performed better without it" once that variable was removed from their prediction model. A figure claiming only one of the fifty most AI-cited domains carries the file circulates widely across other sites covering this topic, but it does not appear anywhere in SE Ranking's own published study, and no other primary source for it turned up. It is left out here for that reason.

On the crawler side, Limy, an AI visibility company, published its own analysis of bot traffic on 12 May 2026, drawn from the enterprise brands it monitors: across over 500 million AI bot events logged in a 90 day window, only 408 requests targeted /llms.txt directly. Its own conclusion: "GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly skip the file and crawl HTML directly," and 408 out of more than half a billion events is, in the company's own word, "neglectible."

None of that means the file is pointless everywhere. Coding assistants read it the way it was designed to be read. Cursor, GitHub Copilot, Claude Code, Cline and Aider all check for /llms.txt and /llms-full.txt when pointed at a documentation site, using it to find the right reference page instead of guessing from a rendered HTML menu. And the companies with the most reason to know how these tools behave publish the file themselves: Stripe's docs.stripe.com/llms.txt, Cloudflare's developers.cloudflare.com/llms.txt and llms-full.txt, and Anthropic's own platform.claude.com/llms.txt were all live and reachable when checked directly on 16 September 2026. Perplexity, Cursor and GitHub each publish one for their own documentation too. So the honest summary is narrower than either camp wants it to be. llms.txt is a real, working convention for developer tooling and documentation, published by the companies whose documentation those tools are pointed at, and ignored by the one search engine that has said anything on the record about it. Those are not the same claim, and the evidence above treats them as the two separate questions they are.

How you would know if it worked for you

The only way to find out what llms.txt is doing on a given site is to read the server logs, not to check that the file loads. A 200 response on a browser request confirms the file exists. It says nothing about whether a bot that matters has ever asked for it. Filter raw access logs for the user agent strings that actually drive AI citations, the same ones Limy's study tracked: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended, matched against requests for /llms.txt and /llms-full.txt specifically. Run that filter over a period measured in weeks, not days, because a handful of hits on any single day is not a pattern.

The wider category this technique belongs to is defined in what generative engine optimization is, and the reason a before-and-after reading is hard to trust at all is in why AI answers change between runs. Tools that do this measuring are compared in the best AI visibility tools.

Even a clean signal that a bot requested the file does not settle whether it changed anything about how often a brand gets mentioned. AI answers move on their own, with nothing about the underlying site changing at all. A survey of 45 studies in this field, published on arXiv in July 2026, reports that published audits of commercial AI engines "further reveal low source overlap, substantial run-to-run variability, and persistent fidelity gaps". Those audits are academic work examining the engines themselves, not products sold to track brand visibility. A brand's mention rate can swing between two runs of the identical question, asked the same week, for no reason connected to anything a site owner did. Reading that swing as evidence a new file worked, or failed, is a measurement error before it is anything else.

What separates a real before and after from noise is asking enough separate questions, across enough weeks, that the margin around each week's number is narrow enough to tell the two apart. Keylight, an AI visibility tracker, works this way: it asks a fixed set of questions every week, spread across whichever of its eight engines the customer chooses, prints the margin of error beside each week's score, and calls a week-on-week difference a change only when its whole range sits clear of zero. That is a way of finding out whether something moved. It is not a claim that publishing a file is what would move it.