llms.txt: The Complete 2026 Guide (Generator, Examples, Validators)

Explore with AI

Open in ChatGPTOpen in ClaudeOpen in PerplexityOpen in Google AI Mode

Last updated: August 10, 2026

If you have ever opened your website with an AI assistant in mind, you have probably wondered the same thing every other SEO and content team is asking in 2026: do these models actually read my site, and if so, what part? Pages are noisy. Navigation, ads, cookie banners, related articles, footer junk. Models waste tokens parsing all of it before they get to the answer.

llms.txt is the proposed fix. It is a single Markdown file at the root of your domain that hands AI systems a curated map of what matters on your site. This guide walks through the published proposal, practical examples and generators, how to check and host the file, common mistakes, and the honest answer to the question everyone wants answered: does it move the needle in ChatGPT, Perplexity, Gemini, or Google Search AI features yet?

What is llms.txt?

llms.txt is a plain-text Markdown file served at the root path of a website, specifically at https://yourdomain.com/llms.txt. It is intended for large language models and AI agents that are trying to make sense of your site without parsing every HTML page. Think of it as a Wikipedia-style summary written in machine-friendly Markdown, hand-curated to point AI systems at your most important content.

The proposal came from Jeremy Howard, co-founder of Answer.AI, in late 2024. The published specification lives at llmstxt.org. The idea sits philosophically between two existing files you already know: robots.txt (rules for what crawlers can and cannot access) and sitemap.xml (a machine-readable list of every URL). llms.txt is neither. It is a human-curated brief explaining your site, written in Markdown so that LLMs can parse it cheaply and accurately.

Some tools generate larger context files that concatenate page content, but llms-full.txt is not defined by the llms.txt proposal. The proposal instead mentions optional Markdown page variants and implementation-specific context bundles such as llms-ctx.txt.

You can take a look at our llms.txt file here.

Why llms.txt matters for AI visibility in 2026

The reason llms.txt exists is that LLMs do not browse the web the way Googlebot does. When you ask Perplexity, ChatGPT search, or Gemini a question, the model needs to ingest content fast and into a finite context window. HTML is bloated. CSS, JavaScript, navigation, ads, sidebars, and cookie banners eat tokens that could otherwise carry actual answers. Markdown often removes navigation and presentation markup, although the token savings depend on the original page.

For AI search and answer engines, three things matter:

• Clarity. Can the model figure out what your site is about in one short read?
• Curation. Did you tell the model which pages are canonical and which are noise?
• Speed. Can the model fetch your key content without 20 round-trips and a render budget?

llms.txt addresses all three. You write the summary once, you choose the URLs that matter, and you serve them in pre-parsed Markdown form. For documentation sites and knowledge bases, this can be a low-cost experiment, but it is not a confirmed ranking or citation signal.

It is also worth pointing out: AI search is a different game from classic SEO. Ranking inside ChatGPT or Perplexity has less to do with backlinks and more to do with whether your content is clean, citable, and easy to ingest. That is the gap llms.txt is designed to close. For the broader strategic picture, our guide to AI search optimization covers how llms.txt fits alongside structured data, content design, and citation tracking.

The llms.txt specification (structure and rules)

The official spec, as published at llmstxt.org, is intentionally minimal. The whole file is valid CommonMark Markdown. The proposal requires an H1 and describes an optional structure for the elements that follow.

Here is the canonical structure, rendered line by line:

# Project Name

> A short, one-sentence description of what this site or project does.

An optional paragraph or two of additional context. This block is free-form Markdown and can include any details you want the LLM to know before it explores the listed resources.

## Docs

– [Quickstart](https://example.com/docs/quickstart): One-line description of the page.

– [API reference](https://example.com/docs/api): What this page covers.

## Examples

– [Hello world example](https://example.com/examples/hello): Description.

## Optional

– [Changelog](https://example.com/changelog): Less critical, can be skipped if context is tight.

The rules are:

• H1 (required). The H1 is your site or project name and is the only required element.
• Blockquote summary (optional). A short blockquote may follow the H1. The proposal does not set a 200-character limit.
• Free-form prose (optional). Markdown after the blockquote can provide context.
• H2 sections (optional). Sections can group Markdown links under headings such as Docs, API, Examples, or Tutorials.
• Link notes (optional). A link can be followed by a colon and description to add context; the description is not required.
• Optional section. The H2 titled “Optional” has a special meaning: links listed there can be skipped by the LLM if it is short on context tokens. Use this for secondary content like changelogs and legal pages.

That is the whole spec. No XML, no JSON, no front-matter, no schema validation step. The simplicity is the point. An LLM should be able to parse the file with a basic Markdown parser and walk the structure semantically.

llms.txt vs llms-full.txt vs robots.txt vs sitemap.xml

These four files are constantly confused. Each serves a different purpose. Here is the clean breakdown:

Put another way: robots.txt sets crawl permissions, sitemap.xml lists canonical URLs for search engines, and llms.txt proposes a curated Markdown index for compatible AI tools. Generated context bundles are optional tool-specific artifacts. Google says no special AI-readable file is required for AI Overviews or AI Mode.

If you have not yet checked which AI crawlers your site allows, do that first. Our guide on whether your website is crawlable by AI walks through the user-agents and headers you need to inspect.

How to create your llms.txt step-by-step (manual)

Generators are fine for a first pass, but a manually written llms.txt is consistently better. The auto-generators tend to either (a) include too many URLs and bloat the file or (b) miss the context paragraphs entirely. Here is the practitioner workflow.

Step 1: Pick your H1 and one-line summary. The H1 is the name of your company or product. The blockquote summary is the elevator pitch. Both should match how you would describe the site to a human in a single breath. No marketing fluff, no buzzwords.

Step 2: Write the context paragraph. One or two sentences right under the blockquote. Tell the LLM what kind of audience you serve, what topics your site is authoritative on, and what readers should expect. This block is the most overlooked part of the spec and the highest leverage. The LLM uses this to decide how to weight everything that follows.

Step 3: Enumerate your most important pages. Open a spreadsheet. List every URL you would want an AI to cite if a relevant question came up. For a SaaS site: pricing, key product pages, top documentation pages, the most cited blog posts, the changelog, and the trust pages (security, terms, privacy). Aim for 20 to 80 URLs total for most sites. Less than 10 is too sparse. More than 200 starts to defeat the purpose.

Step 4: Group them under H2 sections. Common sections: Docs, API, Tutorials, Examples, Blog, Company, Optional. Pick the groupings that match how a curious LLM would navigate your site. Sections should each have between 3 and 30 links. If a section has 100 links, split it.

Step 5: Write a one-line description per link. The colon-after-the-link description is the second most overlooked feature of the spec. It is your chance to tell the model what each page actually covers. Be specific. “Pricing” is weak. “Plan comparison with feature breakdown and current monthly prices” is strong.

Step 6: Move secondary content to Optional. Changelogs, legal pages, low-priority blog content, anything you do not need cited urgently. The “Optional” H2 tells the LLM it can skip this if context is tight.

Step 7: Create a context bundle only if a specific tool needs one. A generator may assemble selected pages into Markdown for a context window, but the filename and format are implementation-specific and are not part of the llms.txt proposal.

Step 8: Validate, host, and link. Validate the file (next section). Upload to the root of your domain. Optionally reference it in your robots.txt with a non-standard comment so curious crawlers can find it.

Real-world llms.txt examples

Looking at live files is useful, but implementations change and do not prove that a search engine consumes them. The proposal itself points to FastHTML as an example, while documentation platforms such as Mintlify and Perplexity publish or expose llms.txt files. Verify any example before copying its structure.

Mintlify. As a docs platform, Mintlify auto-generates llms.txt for every customer site hosted on the platform. Their own llms.txt is dense, organised by product area, and includes a generous Optional section. A typical line: – [Quickstart](https://mintlify.com/docs/quickstart): Get your docs site live in five minutes.

FastHTML. The proposal links to FastHTML as its reference example. It uses the suggested Markdown structure to point readers and compatible tools toward documentation resources.

Perplexity. Perplexity publishes an llms.txt index for its developer documentation. This shows that the format is useful as documentation infrastructure, but it is not evidence that Perplexity Search ranks third-party sites through llms.txt.

The practical pattern is curation rather than completeness. Keep descriptions accurate, list canonical resources, and avoid turning the file into a second sitemap.

Best llms.txt generators in 2026

If you do not want to hand-write the file, generators can do most of the heavy lifting. Quality varies. Some auto-generators crawl your sitemap and produce a 5,000-line monstrosity with every blog post, every category page, and every legal disclaimer. That is worse than no llms.txt at all because it defeats the curation premise.

Here is an honest comparison of the most-used free generators as of 2026:

The honest take: if your site already runs on a docs-first platform, the built-in generator is the right pick. If you are running a blog or marketing site, the Firecrawl-powered generator will give you a solid first draft that you should then trim by 30 to 70 percent. Auto-generators that simply dump your sitemap into Markdown produce bloated output that no LLM will benefit from. Curation always wins.

We also have a free llms.txt generator you can use for a first draft.

How to validate your llms.txt

llms.txt has no formal schema, so “validation” is less rigid than for JSON-LD or sitemap.xml. There are still checks worth running.

• Markdown parses cleanly. Run the file through any CommonMark parser. If headings, links, or list bullets break, fix them. The most common issues are stray asterisks and unescaped square brackets.
• Exactly one H1. Multiple H1s confuse the parser. The spec is explicit: one H1, at the top.
• Optional structure is intentional. A blockquote and H2 link sections are useful conventions, but only the H1 is required by the proposal.
• All links resolve. Every URL in the file must return a 200 status code. Run a link checker. Dead links erode trust in the file.
• No JavaScript-only pages. If a linked page renders content via JS only, the LLM will fetch the HTML shell and get nothing. Either link to a server-rendered equivalent or add the page’s content directly to llms-full.txt.
• Reasonable size. Keep the index curated enough to be useful. The proposal does not set a 100 KB limit or define a size limit for generated context bundles.
• UTF-8 encoding. The file must be served as plain text with UTF-8 encoding. Test with curl: curl -I yourdomain.com/llms.txt and confirm the Content-Type header.

Community validators have started to appear in 2026. Most are simple web forms that fetch your file, parse it, and flag missing or malformed sections. The Firecrawl validator and the llmstxt.org reference checker are the two most commonly used. For most teams, running a CommonMark parser plus a link checker is enough.

Where to host it and how to verify it is accessible

The conventional location is /llms.txt at the root of a domain. The proposal also allows a file within a subpath when the content applies to a specific section, so root is the default rather than the only valid location.

Hosting steps:

1. Upload llms.txt to your web root (the same directory that serves your index.html and robots.txt).
2. Confirm it is reachable via https://yourdomain.com/llms.txt. Test from a clean browser tab and incognito mode.
3. Verify the Content-Type header is text/plain or text/markdown. If your server returns application/octet-stream, fix the MIME-type configuration.
4. Ensure caching is sane. A Cache-Control max-age of one day is reasonable; LLMs do not need second-by-second freshness.
5. If a specific tool needs a generated context bundle, publish it at the location that tool documents.
6. Confirm robots.txt does not disallow the path. Block /admin, not /llms.txt.

Once live, a quick curl test confirms accessibility: curl -i https://yourdomain.com/llms.txt. You should see a 200 response, plain-text content type, and your Markdown in the body. If a CDN sits in front of your origin, do a hard cache purge on the file path after each update.

Common mistakes to avoid

Most broken llms.txt files share the same handful of issues. Avoid these and you are ahead of 90 percent of implementations.

• Hosting at the wrong path. /static/llms.txt or /assets/llms.txt does not work. Root only.
• Serving as HTML instead of plain text. If your CMS wraps the file in template HTML, AI parsers choke. Serve raw.
• Dumping the entire sitemap. Five thousand links in llms.txt is not curation, it is noise.
• Unclear summary or links. The blockquote and link descriptions are optional, but when you include them they should be concise and accurate.
• Using H1 inside sections. Only one H1 in the entire file. Sections are H2.
• Including JavaScript-only pages. If the page is empty without JS, the LLM gets nothing.
• Forgetting the Optional section. Without it, you give every page equal weight; with it, you guide the LLM under context pressure.
• Letting it go stale. Treat llms.txt like a sitemap: review quarterly, refresh after major site changes.
• Blocking it in robots.txt. If your robots.txt disallows /, you will accidentally block AI agents from reading the very file you wrote for them.

Does llms.txt actually affect AI search visibility today?

Here is the honest answer most other guides will not give you: the proposal is new, adoption by the major AI engines is uneven, and there is no public confirmation in 2026 that ChatGPT search or Google AI Mode treat llms.txt as a primary input signal. What we do know:

• The format is implemented by some documentation platforms and developer tools.
• OpenAI and Google have not confirmed llms.txt as an indexing or ranking input for their search products.
• Google explicitly says no special AI-readable file is needed for AI Overviews or AI Mode.
• A specific agent or documentation tool may use llms.txt when its own documentation says so; do not assume every agent fetches it.

The case for shipping one anyway is straightforward. Writing a thoughtful llms.txt takes a few hours. The file can be small and inexpensive to host, though it still needs maintenance and should not expose private or outdated URLs. Even if no major AI engine treats it as a ranking signal in 2026, the act of writing one forces you to curate your most important pages and articulate your site’s purpose in plain language. Those are both useful regardless.

What you should not do is treat llms.txt as a magic switch. Shipping the file will not catapult you to the top of ChatGPT answers next week. AI visibility comes from accessible, useful content and authoritative mentions across the web. Treat llms.txt as an optional experiment, not a confirmed ranking signal. The folks who win in AI search treat the whole stack as a system. For a deeper look at what actually moves rankings inside AI, our piece on how ChatGPT searches work covers the retrieval and ranking flow inside the most-used AI assistant.

How to track whether AI uses your content

You can ship the perfect llms.txt and still have no idea whether it changed anything. The only way to know is to measure: are AI models mentioning your brand more often, citing your URLs more often, and showing up in answers to the prompts your customers actually ask?

This is exactly the problem LLM Pulse solves. LLM Pulse runs your strategic prompts across ChatGPT, Perplexity, Gemini, Google AI Mode, and Google AI Overviews, tracks brand visibility and citation frequency for you and your competitors, and surfaces share of voice across all five major AI models. If your llms.txt is working, you will see citation counts and mention rates climb over the weeks following deployment. If it is not, you will see noise, and you will know to iterate.

For a practical walk-through of citation monitoring, see our guide on how to monitor citations and sources in AI answers, and the companion piece on how to track brand mentions in LLMs. The measurement layer is what turns llms.txt from a “we shipped it” checkbox into an actual GEO programme.

The same measurement principle applies whether you are running an in-house team or a GEO agency. Without instrumentation, every optimisation (llms.txt included) is theoretical. With it, you can attribute lift, defend budget, and prioritise the work that actually moves AI visibility.

Summary

llms.txt is a simple, low-cost, high-leverage file that gives AI systems a curated map of your site in Markdown. The spec, proposed by Jeremy Howard in 2024, is intentionally minimal: one H1, a blockquote summary, H2 sections with linked resources, an Optional section for less critical content. The proposal includes a FastHTML example, and documentation platforms such as Mintlify and Perplexity expose implementations. Generators can produce a first draft in minutes, but the best implementations are still hand-curated.

The question of whether llms.txt actually moves AI search rankings in 2026 is still open. The major AI engines have not formally committed to it as a ranking signal. The format appears on more documentation sites, but it still needs maintenance and should not expose private or stale URLs. Publish it only if the upkeep fits your workflow, and pair it with proven crawlability and content work.

What you should not skip is the measurement layer. If you cannot see whether AI is citing your content, you cannot tell if llms.txt is working. LLM Pulse tracks brand visibility and citations across ChatGPT, Perplexity, Gemini, Google AI Mode, and Google AI Overviews, gives you visibility scores over time and share of voice against your competitors, and starts at €49 per month. For a wider look at the tool landscape, see our breakdown of the best AI visibility tools in 2026.