AI coding assistants like Cursor and GitHub Copilot need to read your API documentation to generate working integration code. The problem? These tools struggle to extract useful information from HTML documentation pages efficiently. They hit token limits parsing navigation menus, JavaScript, and visual styling before reaching your endpoint descriptions. When your API documentation isn't accessible to AI tools, developers get incorrect code suggestions and slower integration. The llms.txt format solves this by providing a markdown-based version of your docs that AI systems can parse within their token budgets. It strips away everything except the information needed for code generation.
TLDR:
llms.txtprovides a compact overview with links to full documentation, whilellms-full.txtembeds complete content directly in the file without requiring external fetches- AI agents need explicit parameter descriptions and error schemas to generate working code
- Include only technical reference material — strip marketing copy, decorative elements, and anything that doesn't help with code generation
- Tools like Fern auto-generate both
llms.txtandllms-full.txtfrom your documentation as part of the build process
What is llms.txt and why does it matter for API documentation?
llms.txt is a plain-text, markdown-based format designed for AI consumption. It serves a similar role to a sitemap: instead of helping search engines crawl your site, it helps coding assistants locate and understand your API's endpoints, authentication methods, and response schemas.
The format strips away CSS, navigation elements, and JavaScript, leaving clean structured text that AI systems can parse efficiently within their token budgets. In the context of API documentation, coding assistants can spend their context window on your endpoint descriptions, parameter types, and error schemas instead of page chrome.
With 85% of developers using AI tools in 2026, writing integration code happens inside assistants like Cursor and GitHub Copilot. If your docs aren't accessible to these agents, you add friction to developer onboarding. llms.txt removes that friction.
Understanding the llms.txt specification and format
The llms.txt specification uses a structured markdown format optimized for machine readability. Each file starts with an H1 header containing your project name, followed by a blockquote summarizing your API in one or two sentences.
Documentation is organized into sections using H2 headers for major topics like authentication or API endpoints. Within each section, provide core details in plain markdown with links to the corresponding full documentation pages. These links allow AI assistants to reference the condensed llms.txt content while following URLs for deeper context when needed.
Keep descriptions focused on what developers need to integrate your API: parameter types, authentication requirements, and error handling. Avoid marketing language or context that fails to serve code generation. Use markdown code blocks for examples that show usage patterns and clarify how your API behaves in practice.
llms.txt vs. llms-full.txt: choosing the right approach
The specification defines two file variants that serve different AI consumption patterns. llms.txt provides a compact overview with links to full documentation. llms-full.txt embeds complete content directly in the file without requiring external fetches.
llms.txt works best for larger documentation sites where including everything would exceed token limits. AI assistants read the summary, identify relevant sections, and follow links for detailed information.
llms-full.txt suits APIs with concise documentation that fits entirely within AI context windows. The file contains complete endpoint descriptions, authentication details, and code examples without requiring link traversal.
Most API providers should implement both files. Serve llms.txt for AI assistants optimizing for token usage and llms-full.txt for agents that prefer self-contained context. Tools like Fern generate both variants automatically from the same source specification, eliminating manual maintenance.
Best practices for generating and maintaining llms.txt files
Keep your llms.txt accurate and current — stale files generate broken code. Outdated authentication flows or removed endpoints cause AI tools to suggest non-functional code, so breaking changes need immediate regeneration. Minor updates like typo fixes, clarified descriptions, and small parameter changes can wait for scheduled regeneration cycles.
Test AI tool parsing by loading your llms.txt file into Cursor or Claude and asking specific questions about your API. Request code for authenticating with your API, calling a specific endpoint, or handling a particular error response. If the AI tool references outdated information or can't locate documented features, adjust your file structure. Token count matters less than whether AI assistants can extract correct information quickly.
Monitor which sections AI tools reference most frequently by tracking link clicks from llms.txt to your full documentation pages. High-traffic sections deserve more detail in the llms.txt summary, while rarely-accessed content can rely on brief descriptions with links. This usage data reveals whether your information hierarchy matches how developers actually integrate your API. Tools like Fern provide analytics for llms.txt usage, including traffic breakdowns by LLM provider and page-level metrics showing bot versus human visitors.
API versioning adds complexity to llms.txt maintenance. When releasing a new API version, you can either maintain separate llms.txt files per version or include all versions in a single file. Separate files prevent token bloat but require AI tools to know which version a developer is targeting. A unified file with version-specific sections works better when most endpoints remain stable across versions. If your documentation tooling supports versioned docs, generating separate llms.txt files per version gives you the most control over what each AI assistant sees.
Optimizing OpenAPI specifications for AI agent discovery
OpenAPI specifications serve as machine-readable contracts that let AI agents directly identify endpoints, methods, parameters, and response schemas. While llms.txt helps with general documentation discovery, a well-structured OpenAPI file gives agents structured access to your API's capabilities without relying on natural language interpretation.
Be thorough with every parameter — what seems obvious to a human reader is ambiguous to an AI agent. Include realistic example responses in your OpenAPI schema. AI coding assistants use these examples to generate response parsing code. Generic placeholder values like "string" or 123 produce less accurate code suggestions than responses that reflect your actual API output. Some documentation tools, including Fern, can generate realistic examples automatically, but even manually written examples substantially improve code generation accuracy.
Document all error responses with status codes and schemas. When AI tools know your API returns a 429 with retry-after headers, they can generate proper error handling code that respects rate limits. Error schemas are easy to overlook, but they're some of the highest-value content for AI-assisted code generation.
How Fern automatically generates and optimizes llms.txt for your API docs
Fern generates automated llms.txt files (both llms.txt and llms-full.txt variants) when building documentation sites. Fern also serves clean Markdown instead of HTML on any page URL — agents can fetch the source by appending .md to the URL or sending an Accept: text/markdown header. Together, these features reduce token consumption by over 90% compared to AI tools parsing full HTML pages.
The generation process pulls from the same API specification that creates your SDK code and API reference. When an endpoint or authentication method is updated in your OpenAPI file or Fern Definition, the llms.txt files update automatically during the next documentation build. This eliminates drift between your human-readable docs and AI-optimized files.
Both files are available at any level of your documentation hierarchy, not limited to the site root. You can access /docs/llms.txt, /docs/ai-features/llms-full.txt, or any other section path, letting AI tools request only the portion of your documentation relevant to a specific task.
Query parameters provide additional control over AI-accessible content. Append ?lang=python to filter code examples to a specific SDK language, or use ?excludeSpec=true to remove OpenAPI and AsyncAPI specifications and focus AI tools on conceptual guides instead of raw schemas. Fern automatically serves your OpenAPI and AsyncAPI specs at /openapi.json and /asyncapi.json (with YAML variants also available), linking them directly in llms.txt. Fern docs sites also expose a standards-based API catalog at /.well-known/api-catalog (per RFC 9727), letting AI agents and MCP clients discover your APIs without scraping HTML. Fern also automatically prepends a default agent directive to every page served to AI agents, pointing them to your .md URLs, llms.txt, and llms-full.txt — you can override or disable this directive via the agents key in docs.yml. For custom llms.txt files, point to your files under the same agents configuration to tailor exactly what Cursor and GitHub Copilot see.
Custom markdown tags control content visibility between human readers and AI agents. Wrap content in <llms-only> tags to include information exclusively in llms.txt files. This works well for adding technical context like implementation details or architecture notes that would clutter the visual documentation. Use <llms-ignore> tags to exclude marketing CTAs, promotional content, or navigation hints that add no value for AI parsing. To exclude entire pages from llms.txt output, add noindex: true to the page's frontmatter. This removes the page from both AI-optimized files and site navigation while keeping it accessible by direct URL.
Final thoughts on llms.txt adoption for APIs
Developers are already writing integration code inside AI coding assistants. llms.txt provides those tools with clean, structured access to your API documentation instead of struggling to parse HTML pages. When llms.txt is generated automatically from your API specification, your AI-optimized documentation stays current with every API change, and developers get accurate code suggestions on their first try instead of dealing with hallucinated endpoints.
You can also use Fern's Agent Score tool to measure how well your documentation serves autonomous agents and see where you stand on the public leaderboard.
Want to see automated llms.txt generation in action? Book a demo with the Fern team.
FAQ
How do you generate an llms.txt file for API documentation?
The most reliable approach is to integrate llms.txt generation into your documentation build pipeline instead of writing the file manually. If you use Fern, llms.txt and llms-full.txt files are generated automatically from your documentation when building your site, typically reducing token consumption by over 90% compared to HTML pages.
What's the difference between llms.txt and llms-full.txt?
llms.txt provides a compact overview with links to full documentation, making it ideal for larger documentation sites where including everything would exceed token limits. llms-full.txt embeds complete content directly in the file without requiring external fetches, which works best for APIs with concise documentation that fits entirely within AI context windows.
When should you use MCP instead of llms.txt for an API?
Use llms.txt when helping developers write better integration code faster through AI-assisted tools like Cursor and GitHub Copilot. Choose MCP when AI agents should interact with your API autonomously without a human developer in the loop. MCP traditionally requires running your own protocol server, though some platforms — Fern, for example — auto-host an MCP server for your docs site so you skip the infrastructure entirely.
How often should you update your llms.txt file?
Update your llms.txt file whenever you publish documentation changes, and integrate generation into your build pipeline so updates happen automatically. AI coding assistants cache these files, so stale versions can persist for days or weeks, leading to incorrect code suggestions if your documentation has changed.
What information should be included in an OpenAPI specification for AI agents?
Include explicit parameter descriptions stating format requirements and validation rules, realistic example responses instead of generic placeholders, and all error responses with status codes and schemas. AI agents need this level of detail to generate accurate code. Unlike human developers, they cannot infer missing information from naming conventions or context.
