Does llms.txt do anything?
For search and AI answers, there is no evidence that it does. Google Search ignores the file, 97% of llms.txt files in a June 2026 Ahrefs study received no requests at all in a month, and an SE Ranking study of about 300,000 domains found no link between having one and being cited by AI models. Its working use is narrower: helping coding agents read developer documentation.
llms.txt is a Markdown file at a site's root or under a path such as /docs/, proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai. The llms.txt proposal was published on 3 September 2024 and revised as v2 on 10 August 2026. A file starts with an H1 naming the site, which is the only required part, then usually a short summary in a blockquote and lists of links to pages, ideally clean Markdown versions of them, that an agent can follow when it needs detail.
Two misconceptions are common. The file is not a set of rules: the proposal says robots.txt and llms.txt have different purposes, and llms.txt is read on demand when an agent needs information. And it is not a ranking signal anyone has confirmed. SE Ranking's own explainer still calls it a newly proposed AI robots.txt.
What does Google say about llms.txt?
Google Search doesn't use it, and says so in its documentation. Search's position has got firmer over a year, while the Chrome team added a check:
| Date | Who | What was said |
|---|---|---|
| 17 June 2025 | John Mueller, Google, on Bluesky | "FWIW no AI system currently uses llms.txt." Mueller added that server logs show chatbots fetching pages but not the file. |
| 15 May 2026 | Google Search Central, AI optimisation guide | Lists llms.txt under things you can ignore: you don't need machine-readable files, AI text files or Markdown to appear in Google Search, including its AI features |
| 15 June 2026 | Google Search Central changelog | Added a note that the files won't hurt or help visibility or rankings, and that keeping them for other services is fine |
| Page last updated 5 May 2026 | Chrome's Lighthouse docs | An agentic browsing audit checks for llms.txt; a missing file is marked not applicable, because the file is optional |
Mueller's 2025 remark was reported by Search Engine Roundtable. The guide's mythbusting section now reads: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." The changelog entry says the note was added to address questions from the community. Search Engine Roundtable covered the change on 16 June 2026, noting that Google had published the guide about a month earlier.
The Lighthouse audit made people think Google was saying two things. In an r/SEO thread asking exactly that, about 60 comments when checked on 24 October 2026, the most-upvoted explanation was that two Google teams were talking about two uses: Search visibility, and AI agents navigating a site. According to Ahrefs, when Lily Ray asked Mueller about the contradiction, Mueller said llms.txt is not done for search and called it a temporary stopgap for AI coding tools reading developer docs.
Does anything actually fetch llms.txt?
Almost nothing, and most of what does is not an AI search engine. Ahrefs looked at server logs for all 137,210 domains in its Web Analytics product that had traffic in May 2026. Ahrefs says its customers skew technical and SEO-aware, so treat the adoption figure as an upper bound.
| Finding (Ahrefs, May 2026 data) | Figure |
|---|---|
| Domains publishing llms.txt | 28%, about 38,000 |
| Files that received zero requests in the month | 97% |
| Requests to the 3% that were fetched | About 22,000, across roughly 1,100 domains |
| Share of those requests from bots | 96% |
| AI bots of all kinds combined | 19.5% |
| AI retrieval bots (such as OAI-SearchBot and PerplexityBot) | 1.1% |
| AI agents and agent infrastructure (such as Claude-Code) | 10.5% |
| Requests by AI bots for llms.txt files that don't exist | Zero |
Two details stand out. Slackbot, the link-preview bot in a chat app, fetched llms.txt files more often than PerplexityBot did. And about 12% of requests came from tools studying the file: GEO checkers, llms.txt validators and researchers. Ahrefs also stresses that a fetch is not proof that anything read or used the file.
Smaller reports match. In a November 2025 Ask HN post, one team building security analytics said the file was never requested by any LLM in their logs, only by bots from BuiltWith and WebPageTest. That is one team's logs.
Does llms.txt help you get cited by ChatGPT or AI Overviews?
Not in the data published so far; our guide to getting cited in Google AI Overviews covers what Google says does matter. SE Ranking's study of nearly 300,000 domains found llms.txt on 10.13% of them and tested whether the file was linked to how often each domain was cited in sampled LLM answers. It used Spearman correlation and an XGBoost model. Removing the llms.txt variable made the model more accurate, and SE Ranking concluded the file doesn't seem to directly affect AI citation frequency, at least not yet.
The study was covered by Search Engine Journal on 20 November 2025; SE Ranking's page now carries a date of 15 October 2026. SE Ranking adds a caution: its results reflect one model and dataset, and could change if factors are added or removed. Adoption was flat across traffic levels, from 9.88% on the lowest-traffic sites to 8.27% on the highest.
Ahrefs' log data points the same way. AI retrieval bots, the ones that fetch pages to answer a live question, made 1.1% of llms.txt requests. Training crawlers made about five times as many. Ahrefs reads this as meaning any effect on AI visibility would likely come through training data, not at the moment someone asks a question.
In the r/SEO thread on the SE Ranking study, about 50 comments when checked, the most-upvoted text reply pointed at the conclusion: it says the file does nothing, then suggests adding it to prepare for tomorrow.
Where is llms.txt useful?
On developer documentation that coding agents read. The proposal itself says llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials, and that OpenAI, Anthropic and Google's Gemini publish them for their own developer docs.
The Ahrefs logs agree. Claude-Code, Anthropic's coding agent, fetched llms.txt more often than every AI retrieval bot, assistant and training crawler apart from GPTBot and one agent-infrastructure crawler. And agents seem to fetch the file when something points them to it, not by guessing: Ahrefs found no AI bot requests at all for files that didn't exist.
Ahrefs also flags a risk. The largest research crawler in its data identified itself as a prompt-injection survey, meaning someone is studying llms.txt as a way to slip instructions to agents that trust it. If you publish one, version-control it, limit who can edit it, and keep it to plain links and descriptions.
The 10-minute verdict for a small site
If you sell a product with no API or developer docs, skip it. If you have docs that customers' developers read with coding agents, add one to the docs folder and link to it. Use this table:
| Your situation | What to do |
|---|---|
| Marketing site, no docs | Skip it. If your site builder creates one automatically, leave it; Google says it won't hurt. |
| Public API or developer docs | Publish /docs/llms.txt with links to Markdown versions of key pages, and link to it from the docs. |
| You already have one | Keep it if it's accurate. Check your logs before spending more time on it. |
| Someone is selling you llms.txt as an AI visibility fix | Ask for log evidence of AI search bots reading the file. The studies above found almost none. |
To check your own logs, search your web server's access log for requests to /llms.txt and count them by user agent. On a standard combined log format, this one line does it: grep -i "llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn. If the list is empty or only SEO tools, you have your answer.
The doubt shows up in other communities too. In a Shopify Community thread, one Shopify Partner said app store listings for llms.txt apps were making over-the-top claims. In the r/SEO thread on the Ahrefs study, about 80 comments when checked, the commenters defending the file mostly argued it is cheap to add, not that it works.
What should you do instead of llms.txt?
Spend the time on pages that answer buyers' questions, written in crawlable text. Google's guide says that for Search, a clear technical structure and unique, valuable content are the foundation for visibility in its AI features. Start with your product, pricing and comparison pages.
- Check what assistants say about you. Ask ChatGPT, Gemini and Perplexity the questions your buyers ask, and log whether you appear. Our guide to testing whether ChatGPT recommends you has the method.
- Measure before you invest. In the October 2026 studies, ChatGPT sent a small fraction of Google's traffic. Our breakdown of ChatGPT traffic against Google shows how to run the numbers on your own site.
- Know what is new and what is plain SEO. Our look at whether GEO is just SEO separates the two.
- Watch competitors' AI visibility. Siela's Competitor Watch sends a briefing every Monday morning on what each competitor launched, priced, hired or posted, and also checks AI-assistant visibility, who ranks for their terms and their ads.
Frequently asked questions
Does Google use llms.txt for AI Overviews or AI Mode?
No. Google's AI optimisation guide says you don't need machine-readable files, AI text files or Markdown to appear in Google Search, including its generative AI features, because Google Search doesn't use them. A note added on 15 June 2026 says the files will neither harm nor help visibility or rankings in Google Search.
Can llms.txt hurt my SEO?
Not in Google Search, according to Google, which says it ignores the files and that keeping them for other services is fine. The risk Ahrefs points to is different: agents may trust the file, and researchers are already probing llms.txt for prompt injection, so a stale or edited file could mislead any agent that reads it.
Is llms.txt the same as robots.txt?
No. The llms.txt proposal says the two have different purposes: robots.txt tells automated tools what access is acceptable, while llms.txt gives agents a curated overview with links to read on demand. Ahrefs adds that llms.txt controls nothing and blocks nothing. To manage which bots can crawl your site, use robots.txt.
Which bots request llms.txt files?
In Ahrefs' May 2026 log data, the biggest groups were SEO audit tools, unidentified bots, general web crawlers and tech-profiling tools. Named AI bots made 19.5% of requests combined, led by GPTBot and Claude-Code. AI retrieval bots such as OAI-SearchBot and PerplexityBot made only 1.1%.
Should a SaaS company add llms.txt to its docs?
If customers' developers read your docs with coding agents, it is a reasonable small job. The proposal says llms.txt is used most heavily for software documentation, and Claude-Code was the second-biggest AI fetcher in Ahrefs' data, after GPTBot. Put it under the docs path, link to Markdown versions of key pages, and link to the file from your docs.
Sources
- llms.txt proposal, v2 (Jeremy Howard) (3 Sep 2024, modified 10 Aug 2026)
- Google Search Central: Optimizing for generative AI features, mythbusting (last updated 10 Jul 2026)
- Google Search Central: Latest documentation updates (15 June 2026 entry) (15 Jun 2026)
- Chrome for Developers: Lighthouse llms.txt audit (last updated 5 May 2026)
- Search Engine Roundtable: Google says no AI system currently uses llms.txt (18 Jun 2025)
- Search Engine Roundtable: Google says llms.txt files won't help or hurt your search rankings (16 Jun 2026)
- Ahrefs: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (15 Jun 2026, updated 27 Aug 2026)
- SE Ranking: Does LLMs.txt impact your AI visibility and citations? (dated 15 Oct 2026)
- Search Engine Journal: LLMs.txt Shows No Clear Effect On AI Citations, Based On 300k Domains (20 Nov 2025)
- Hacker News: Ask HN: Do LLMs Read Your /llms.txt? (26 Nov 2025)
- Shopify Community: Does llms.txt affect how AI tools read a site? (27 Dec 2025)
- r/SEO: LLMs.txt: Google saying two different things? (31 May 2026)
- r/SEO: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (15 Jun 2026)
- r/SEO: SE Ranking: LLMS.txt does nothing - 300,000 domains analyzed (30 Mar 2026)