LLMs.txt: Does It Actually Work? (Updated October 2025)

In short:

• ✅ LLMs.txt is a proposed standard, not an official one.

• 🧭 It works as a “content map,” not a crawl-control file.

• 🧠 No major AI model fully supports it yet — OpenAI, Google, and Anthropic included.

• 💡 It’s low effort and harmless to implement, with potential long-term upside for AI visibility.

For months, the SEO world has been buzzing about a new file called LLMs.txt, which was often described as a robots.txt for AI. The idea sounds revolutionary: give ChatGPT, Gemini, and Perplexity a roadmap so they understand your content better and cite you more often. But this it actually work? Let's take a look!

The Rise of LLMs.txt

When OpenAI launched ChatGPT’s browsing capabilities and Perplexity introduced “Quick Answers,” a quiet panic swept through marketing departments. Suddenly, searchers were asking AI directly, and skipping Google entirely.

As businesses watched organic traffic drop, a new question emerged:
How can we make sure AI tools read, understand, and quote our content correctly?

That’s where LLMs.txt entered the scene.

Proposed in 2023, it promised to be a simple file placed at /llms.txt, telling large language models which pages matter most and what each one is about. The comparison to robots.txt made it easy to understand — but it also created confusion.

According to Search Engine Land, “LLMs.txt isn’t a crawl-blocker. It’s a treasure map for AI.”

In other words, it’s not there to stop crawlers, but rather to guide them.

What LLMs.txt Actually Is

The term stands for Large Language Model Sitemap.
Technically, it’s a plain-text or Markdown file designed to:

• Summarise what your website covers

• List key pages with short descriptions

• Point AI crawlers toward authoritative, high-value information

Placed at the root of a site (e.g., https://indexlab.ai/llms.txt), it provides both humans and machines a quick overview of your structure.

Think of it as a cheat sheet for AI visibility: a lightweight, text-based outline that says, “Here’s who we are, what we do, and which pages explain it best.”

Who Created It

LLMs.txt was first proposed by Jeremy Howard, co-founder of Answer.AI, through the open GitHub project AnswerDotAI/llms-txt.

The goal: create a public, machine-readable “content map” that language models could ingest without parsing messy HTML or scripts.

Soon after, a community-run website, llmstxt.org, began cataloguing early adopters and publishing examples.

As the idea spread, documentation platforms like Mintlify started auto-generating LLMs.txt files for developer docs.

As Howard himself noted “this isn’t an official standard.” There’s no W3C, ISO, or IETF stamp of approval. It’s still a grassroots experiment, much like how robots.txt started back in 1994.

Different Variants and Why It’s So Confusing

A big reason marketers are confused is that there isn’t just one version of LLMs.txt floating around.

According to Ahrefs, only the content-map style aligns with the original proposal. Many auto-generators mistakenly produce “crawl control” versions that have no practical effect on LLMs.

In other words, if your file looks like a robots.txt clone, you’re doing it wrong.

Who Actually Uses or Supports It

Here’s where theory meets reality, and where optimism fades.

Despite all the hype, no major AI company has formally adopted LLMs.txt as of October 2025.

OpenAI

Some webmasters noticed the GPTBot hitting their /llms.txt endpoints. According to Seroundtable, OpenAI may be “testing or exploring” the format. But there’s no proof that ChatGPT uses its contents during inference or citation.

Google

According to Gary Illyes from Google Search Relations, “We currently have no plans to support LLMs.txt.” (source)

Anthropic, Perplexity, Gemini:

No statements. No documentation. No evidence of implementation.

SEO platforms

Tools like Yoast, Mintlify, and SAP’s documentation hub generate LLMs.txt automatically, but mostly as a future-proofing measure, not because any LLMs are reading them today.

As PPC Land reported in 2025:

“LLMs.txt adoption has stalled because major AI platforms ignore the proposed standard.”

Does It Actually Work?

Let’s look at the data side-by-side:

According to Yoast, “LLMs.txt makes it easier for future crawlers to understand your content hierarchy — but right now, it’s more about preparation than payoff.”

So while it’s technically useful to have, the return on investment is theoretical, at least for now.

Why Marketers Are Still Adding It Anyway

From an Answer Engine Optimisation (AEO) perspective, the rise of LLMs.txt mirrors an old SEO truth: whoever prepares earliest, wins later.

Businesses that structured their sites early for schema markup and featured snippets reaped huge visibility gains when Google adopted those formats.

Today, LLMs.txt sits in that same “pre-standard” stage.

At IndexLab, we’ve tested this across multiple clients, including Flash Facial, a Dubai-based beauty brand.
Before implementing AI SEO optimisation, Flash Facial was invisible to ChatGPT. After a structured visibility audit and inclusion of clear metadata and content mapping, they became one of ChatGPT’s top recommendations for beauty services in Dubai (case study).

While that success didn’t rely on LLMs.txt alone, it shows how clear content structuring, which is the core idea behind LLMs.txt, already benefits AI visibility.

Counter-Arguments and Limitations

Critics argue that LLMs.txt is more symbolic than functional.
According to Ahrefs, “no evidence shows that any LLM uses it to decide which pages to summarise or cite.”

Others warn that publishing a full content dump (llms-full.txt) could backfire — effectively giving competitors or AI crawlers your entire content library in one place.

In short: transparency helps, but oversharing can hurt.

That’s why most experts recommend the navigation-only version: a curated, high-signal summary rather than an open buffet.

The Future of LLMs.txt

Even though adoption is minimal, few experts dismiss the idea entirely.

Jeremy Howard predicts that a formal “AI Crawling Protocol” will emerge, possibly evolving from LLMs.txt once major players agree on standards.

IndexLab analysts compare it to schema markup in 2011: initially niche, later essential.

AI systems will increasingly need structured trust signals: lightweight metadata, verified sources, and clear content maps, to separate brand authority from noise.

When that day comes, having a well-structured LLMs.txt already in place may accelerate your inclusion in model retraining cycles and answer-engine citations.

Should You Implement It?

Yes you should, but ensure you manage you own expectations.

Implement the “content-map” style LLMs.txt, not the crawl-directive kind. Here’s what that means in practice:

• Use simple Markdown headings and bullet points.

• Include short page descriptions and canonical URLs.

• Keep the file under 50 KB where possible.

• Reference your sitemap.xml at the end for completeness.

• Update it quarterly or when major pages change.

• Check server logs for visits from GPTBot, ClaudeBot, or GoogleOther.

Adding an llms-full.txt can help internal RAG (retrieval-augmented generation) systems, but it’s not necessary for public SEO.

Think of it as AI discoverability insurance: cheap, optional, but potentially valuable later.

IndexLab’s Expert Take

At IndexLab, our view is clear:

“LLMs.txt won’t make you rank higher tomorrow, but it will make your site easier for AI to understand next year.”

As an AI SEO consultant, we treat it as one component of a larger Generative Engine Optimisation (GEO) framework. The real drivers of AI visibility optimisation remain:

• Structured data and schema

• Clear topical authority

• Quality, conversational content

• Consistent external mentions

• Internal clarity (sitemaps, metadata, content hierarchy)

When those fundamentals are strong, a simple LLMs.txt file becomes an amplifier, not a gimmick.

Frequently Asked Questions About LLMs.txt

1. Is LLMs.txt really necessary for AI SEO?

Not yet, but it’s a smart early move. While LLMs like ChatGPT and Gemini don’t officially use LLMs.txt today, they are increasingly crawling it. Adding one now ensures your site is already structured when these models start reading it systematically. Think of it as preparing for AI discoverability before your competitors do.

2. What’s the difference between LLMs.txt and LLMs-full.txt?

LLMs.txt is a content map: a lightweight file listing key pages with short summaries.
LLMs-full.txt is a content dump: it includes entire pages or long summaries for direct ingestion.
Most experts, including those at IndexLab, recommend sticking to the content-map format to avoid exposing your full content library unnecessarily.

3. Can LLMs.txt help me recover traffic lost to AI search?

Not directly, but it plays a supporting role. As AI search replaces traditional Google results, brands that make their information easy for LLMs to interpret gain visibility faster. Combined with AI SEO optimisation, structured data, and consistent authority building, LLMs.txt can become part of a long-term recovery strategy for traffic lost to AI.

Ready to Build Your Own LLMs.txt?

At IndexLab, we believe in making AI visibility accessible to everyone.
That’s why we’ve built a free ChatGPT-powered tool that helps you generate a compliant LLMs.txt for your website in seconds, no technical skills required.

👉 Try the free LLMs.txt Generator on ChatGPT (it’s free to use and instantly creates the recommended content-map version)

And if you want to go beyond the basics, from AI Visibility Audits to Answer Engine Optimisation (AEO) and full-service AI SEO consulting, our team can help you ensure your brand is seen, trusted, and chosen across ChatGPT, Gemini, and Perplexity.

📩 Contact IndexLab to discuss your AI visibility strategy today.

Conclusion

So, does LLMs.txt actually work?
Right now, no. At least not in the way marketers hope.

There’s no direct ranking effect, no confirmed citation boost, and no formal adoption by the big LLMs.

But it does signal that the web is entering a new phase: one where Answer Engine Optimisation replaces traditional SEO, and businesses must prepare their content for machines that read, reason, and recommend.

By implementing a clean, descriptive LLMs.txt today, you’re laying the groundwork for future AI visibility. It’s low cost, easy to maintain, and when the standards catch up, it could be your early-mover advantage.

In other words: it may not work yet, but when it does, you’ll be glad you were first.