llms.txt hit version 2 — but AI crawlers still mostly ignore it
A file finally reaching maturity
llms.txt — a proposed plain-text file that summarizes a site for AI tools, sitting at the domain root the way robots.txt does — reached version 2 of its spec on August 10, 2026, folding in almost two years of real-world feedback since the original proposal.
Adoption is genuinely significant in one specific corner of the web: over 50% of developer-tool sites now ship one, including Stripe, Vercel, Cloudflare, and Anthropic.
Why crawlers ignore it anyway
Analysis of crawler behavior shows GPTBot, ClaudeBot, PerplexityBot, and Google-Extended overwhelmingly skip the llms.txt file and crawl a site's regular HTML instead — the same way they always have.
Google's own May 2026 AI-search guidance goes further and says explicitly that llms.txt isn't needed for AI Overviews or AI Mode. That's a fairly direct answer to a question a lot of site owners have been asking: no, adding this file alone won't move your AI-search visibility.
Where it actually helps
The audience it's proving useful for is different entirely: AI coding assistants reading a product's API documentation while a developer builds against it.
If you run an API or developer product, llms.txt is worth having for that reason. If your goal is AI search visibility specifically, the leverage is still in the structural page-level signals — retrieval, extractability, schema, direct answers.
Based on reporting from Rankability
Is your own content set up for stories like these?
Run a free AI Readiness check — retrieval, extractability, schema.org signals, and a prioritized rewrite brief, scored the way an AI assistant actually reads your page.