llms.txt: The New Standard for AI Crawlability

llms.txt is an emerging web standard that provides AI models and their associated crawlers with structured information about a website's content, organization, and preferences for AI interaction. Similar to how robots.txt communicates with search engine crawlers, llms.txt communicates specifically with AI systems.

What Is llms.txt?

The llms.txt file is a plain text or markdown file placed in your website's root directory. It provides AI systems with a human- and machine-readable overview of your organization, key content, and instructions for AI interaction.

Why llms.txt Matters

AI crawlers benefit from explicit guidance about what content matters most. Without llms.txt, AI systems must infer your site's structure from HTML alone. Providing a curated overview gives AI models a structured entry point to your most important content, improving citation accuracy.

How to Create an llms.txt File

Start with a brief organization description. List your most important 20-50 pages with concise descriptions. Group by topic. Update whenever you publish significant new content.

llms.txt vs robots.txt

robots.txt controls crawler access permissions. llms.txt provides content guidance about what your site is about and which content is most important. They serve complementary purposes and should coexist.

Implementation Best Practices

Keep descriptions factual and concise. Avoid marketing language. Structure logically. Include authoritative content first. Citerna's site audit checks for llms.txt presence and quality.