What is llms.txt — and why your website needs one

When ChatGPT, Perplexity or Google's AI Mode answer a question, they do not read your website the way Google's classic crawler does. A crawler has all the time in the world; an assistant composing an answer has a context window and a deadline. It looks for a compact, structured summary of what your site offers — and llms.txt is the emerging standard for exactly that: a Markdown file at the root of your domain that lists your most important pages with one-line descriptions.

The problem llms.txt solves

An assistant answering in real time cannot afford to crawl your whole site. It samples: a page from the search index here, a navigation menu there, whatever happens to surface. If the sample misses your pricing page or your best comparison article, the assistant answers without them — or cites a competitor whose site was easier to digest. The result is a quiet injustice: sites with excellent content lose answers to sites with merely legible content.

llms.txt fixes the sampling problem by removing the sampling. Instead of hoping an assistant finds the right pages, you hand it a curated map: this is what the site is, these are the pages that matter, and here is one line on each. Assistants that support the standard read the map first. Assistants that do not lose nothing, because the file is invisible to human visitors and costs a few kilobytes.

What the file looks like

llms.txt is deliberately boring: plain Markdown served at yourdomain.com/llms.txt. A heading with the site name, a short summary in a blockquote, then sections of links with one-line descriptions. No scripts, no markup tricks, no tracking — the entire point is that a language model can read it in one pass without rendering anything.

llms.txt vs. llms-full.txt

The short file is a table of contents. The companion llms-full.txt inlines the actual content of your key pages, so an assistant can quote you without crawling at all. Both files must stay in sync with your site — which is exactly why generating them by hand goes stale within a week. Every published post, every price change, every renamed service silently invalidates a handwritten file, and nobody notices until an assistant quotes last year's offer.

Google now audits it

For a while llms.txt was a community convention. That changed when Google's PageSpeed Insights added an "Agentic Browsing" audit in 2026: it checks whether your llms.txt exists at the root of the domain, whether it parses as valid Markdown with the expected sections, and whether the links in it resolve to real content. Three rules, pass or fail.

The audit matters less for the score and more for what it signals: the largest search company on earth now treats machine readability for AI agents as a measurable property of a website — as normal as mobile friendliness or HTTPS. Sites that pass give assistants a clean map; sites that fail leave the assistant guessing, and assistants that guess tend to cite someone else.

What belongs in the file — and what does not

Not everything. The strongest llms.txt files list 10–30 pages that answer buyer questions: what you do, what it costs, how you compare to alternatives, and proof that you deliver. An assistant reads the file top to bottom, so order is strategy — the pages you pin first are the pages that frame every answer about you.

Blog archives, tag pages and legal boilerplate stay out. They dilute the token budget an assistant is willing to spend on you, and they answer no buyer question. If a page would not help a stranger decide to become a customer, it does not belong in the map.

Setting it up on WordPress

You can write the file by hand — the format is simple enough — but keeping it current is the real work. For WordPress there is a shorter path: the llms.txt generator in AI Rank Suite builds llms.txt and llms-full.txt from your real pages, lets you pin the ones that must come first, adds your preferred citation line and a contact line, shows a live token counter, and re-validates against Google's three audit rules every time your content changes. The WordPress SEO plugin treats the file as a living part of the site, not a one-time export — set it once and it stays green. The step-by-step setup guide walks through it; publishing takes one click and no API key.

FAQ

Does llms.txt replace robots.txt or the XML sitemap?

No — the three do different jobs. robots.txt says what crawlers may read, the sitemap says what exists, and llms.txt says what matters and why. A well-set-up site has all three, and they must not contradict each other.

Will my Google rankings improve because of llms.txt?

Not directly — it is not a ranking factor in classic search. Its territory is Generative Engine Optimization: being found, understood and cited by AI assistants. That said, the discipline it forces — knowing your most important pages and describing each in one line — tends to improve a site's clarity everywhere.

Do all AI assistants read llms.txt today?

Adoption is uneven and moving fast. Some assistants fetch the file on every relevant query, others only when browsing live, and Google audits it in PageSpeed. The cost of having the file is near zero and the cost of missing it grows every quarter — that is the whole calculation.

How large should the file be?

Small enough that an assistant reads all of it. That is what the token counter is for: the short file should stay in the low thousands of tokens, and llms-full.txt should carry only the pages that answer buyer questions in full.