LLMs.txt: Does It Actually Work? (Updated October 2025)

In short:

• ✅ LLMs.txt is a proposed standard, not an official one.

• 🧭 It works as a “content map,” not a crawl-control file.

• 🧠 No major AI model fully supports it yet — OpenAI, Google, and Anthropic included.

• 💡 It’s low effort and harmless to implement, with potential long-term upside for AI visibility.

For months, the SEO world has been buzzing about a new file called LLMs.txt, which was often described as a robots.txt for AI. The idea sounds revolutionary: give ChatGPT, Gemini, and Perplexity a roadmap so they understand your content better and cite you more often. But this it actually work? Let's take a look!

The Rise of LLMs.txt

When OpenAI launched ChatGPT’s browsing capabilities and Perplexity introduced “Quick Answers,” a quiet panic swept through marketing departments. Suddenly, searchers were asking AI directly, and skipping Google entirely.

As businesses watched organic traffic drop, a new question emerged:How can we make sure AI tools read, understand, and quote our content correctly?

That’s where LLMs.txt entered the scene.

Proposed in 2023, it promised to be a simple file placed at /llms.txt, telling large language models which pages matter most and what each one is about. The comparison to robots.txt made it easy to understand — but it also created confusion.

According to Search Engine Land, “LLMs.txt isn’t a crawl-blocker. It’s a treasure map for AI.”

In other words, it’s not there to stop crawlers, but rather to guide them.

What LLMs.txt Actually Is

The term stands for Large Language Model Sitemap.Technically, it’s a plain-text or Markdown file designed to:

• Summarise what your website covers

• List key pages with short descriptions

• Point AI crawlers toward authoritative, high-value information

Placed at the root of a site (e.g., https://indexlab.ai/llms.txt), it provides both humans and machines a quick overview of your structure.

Think of it as a cheat sheet for AI visibility: a lightweight, text-based outline that says, “Here’s who we are, what we do, and which pages explain it best.”

Who Created It

LLMs.txt was first proposed by Jeremy Howard, co-founder of Answer.AI, through the open GitHub project AnswerDotAI/llms-txt.

The goal: create a public, machine-readable “content map” that language models could ingest without parsing messy HTML or scripts.

Soon after, a community-run website, llmstxt.org, began cataloguing early adopters and publishing examples.

As the idea spread, documentation platforms like Mintlify started auto-generating LLMs.txt files for developer docs.

As Howard himself noted “this isn’t an official standard.” There’s no W3C, ISO, or IETF stamp of approval. It’s still a grassroots experiment, much like how robots.txt started back in 1994.

Different Variants and Why It’s So Confusing

A big reason marketers are confused is that there isn’t just one version of LLMs.txt floating around.

According to Ahrefs, only the content-map style aligns with the original proposal. Many auto-generators mistakenly produce “crawl control” versions that have no practical effect on LLMs.

In other words, if your file looks like a robots.txt clone, you’re doing it wrong.

Who Actually Uses or Supports It

Here’s where theory meets reality, and where optimism fades.

Despite all the hype, no major AI company has formally adopted LLMs.txt as of October 2025.

OpenAI

Some webmasters noticed the GPTBot hitting their /llms.txt endpoints. According to Seroundtable, OpenAI may be “testing or exploring” the format. But there’s no proof that ChatGPT uses its contents during inference or citation.

Google

According to Gary Illyes from Google Search Relations, “We currently have no plans to support LLMs.txt.” (source)

Anthropic, Perplexity, Gemini:

No statements. No documentation. No evidence of implementation.

SEO platforms

Tools like Yoast, Mintlify, and SAP’s documentation hub generate LLMs.txt automatically, but mostly as a future-proofing measure, not because any LLMs are reading them today.

As PPC Land reported in 2025:

“LLMs.txt adoption has stalled because major AI platforms ignore the proposed standard.”

Does It Actually Work?

Let’s look at the data side-by-side:

According to Yoast, “LLMs.txt makes it easier for future crawlers to understand your content hierarchy — but right now, it’s more about preparation than payoff.”

So while it’s technically useful to have, the return on investment is theoretical, at least for now.

Why Marketers Are Still Adding It Anyway

From an Answer Engine Optimisation (AEO) perspective, the rise of LLMs.txt mirrors an old SEO truth: whoever prepares earliest, wins later.

Businesses that structured their sites early for schema markup and featured snippets reaped huge visibility gains when Google adopted those formats.

Today, LLMs.txt sits in that same “pre-standard” stage.

At IndexLab, we’ve tested this across multiple clients, including Flash Facial, a Dubai-based beauty brand.Before implementing AI SEO optimisation, Flash Facial was invisible to ChatGPT. After a structured visibility audit and inclusion of clear metadata and content mapping, they became one of ChatGPT’s top recommendations for beauty services in Dubai (case study).

While that success didn’t rely on LLMs.txt alone, it shows how clear content structuring, which is the core idea behind LLMs.txt, already benefits AI visibility.

Counter-Arguments and Limitations

Critics argue that LLMs.txt is more symbolic than functional.According to Ahrefs, “no evidence shows that any LLM uses it to decide which pages to summarise or cite.”

Others warn that publishing a full content dump (llms-full.txt) could backfire — effectively giving competitors or AI crawlers your entire content library in one place.

In short: transparency helps, but oversharing can hurt.

That’s why most experts recommend the navigation-only version: a curated, high-signal summary rather than an open buffet.

The Future of LLMs.txt

Even though adoption is minimal, few experts dismiss the idea entirely.

Jeremy Howard predicts that a formal “AI Crawling Protocol” will emerge, possibly evolving from LLMs.txt once major players agree on standards.

IndexLab analysts compare it to schema markup in 2011: initially niche, later essential.

AI systems will increasingly need structured trust signals: lightweight metadata, verified sources, and clear content maps, to separate brand authority from noise.

When that day comes, having a well-structured LLMs.txt already in place may accelerate your inclusion in model retraining cycles and answer-engine citations.

Should You Implement It?

Yes you should, but ensure you manage you own expectations.

Implement the “content-map” style LLMs.txt, not the crawl-directive kind. Here’s what that means in practice:

• Use simple Markdown headings and bullet points.

• Include short page descriptions and canonical URLs.

• Keep the file under 50 KB where possible.

• Reference your sitemap.xml at the end for completeness.

• Update it quarterly or when major pages change.

• Check server logs for visits from GPTBot, ClaudeBot, or GoogleOther.

Adding an llms-full.txt can help internal RAG (retrieval-augmented generation) systems, but it’s not necessary for public SEO.

Think of it as AI discoverability insurance: cheap, optional, but potentially valuable later.

IndexLab’s Expert Take

At IndexLab, our view is clear:

“LLMs.txt won’t make you rank higher tomorrow, but it will make your site easier for AI to understand next year.”

As an AI SEO consultant, we treat it as one component of a larger Generative Engine Optimisation (GEO) framework. The real drivers of AI visibility optimisation remain:

• Structured data and schema

• Clear topical authority

• Quality, conversational content

• Consistent external mentions

• Internal clarity (sitemaps, metadata, content hierarchy)

When those fundamentals are strong, a simple LLMs.txt file becomes an amplifier, not a gimmick.