![[Translation] Jeremy Howard's Proposal for Standardizing the Use of llms.txt](/_next/image?url=%2Fblog%2Ftranslation_chinese_jeremy_howard_proposal_llms_txt%2Fai-generated-8237709_640.jpg&w=3840&q=75&dpl=dpl_3FdjBWZrvEfa3HCTwryyEb3Qb5XJ)
[Translation] Jeremy Howard's Proposal for Standardizing the Use of llms.txt
A proposal for standardizing the use of /llms.txt files to provide information that helps large language models use websites at inference time
Author: Jeremy Howard
Published: September 3, 2024
Original: https://llmstxt.org/
Background
Large language models increasingly rely on website information, but they face a critical limitation:
The context window is too small to fully process the entire contents of most websites.
And complex pages are difficult and imprecise to extract into plain text when they contain navigation, advertising, JavaScript, and other noise.
Websites serve both human readers and large language models. But models would be better served by concise, professional-grade information delivered in a single, easily accessible location.
One example: this matters especially for use cases such as development environments, where a large language model needs fast access to and understanding of programming documentation and APIs.
Proposal

We propose adding a static Markdown file named /llms.txt to a website, providing content that is friendly to large language models. The file supplies brief background information, guidance, and links to detailed Markdown files.
The Markdown format used by llms.txt is readable by both humans and large language models. It is also a precise format that still allows widely accepted processing approaches (rendering and other forms of formatting, regular expressions, and traditional programming techniques).
In addition, we recommend that websites include Markdown descriptions of pages that may be useful to large language models, by providing a clean Markdown version of those pages at the same URL as the original page, with an added .md suffix. (URLs without a filename can append index.html.md.)
The FastHTML project follows both recommendations in its documentation. For example, here is the llms.txt for the FastHTML documentation. Here is an example of a regular HTML documentation page, and here is exactly the same URL with the .md extension.
This proposal does not include any specific recommendations for how to process llms.txt files, because that will depend on the use case.
For example, the FastHTML project chooses to automatically expand llms.txt into two Markdown files containing the content of the linked URLs, using an XML-based structure suitable for large language models such as Claude. The two files are:
-
llms-ctx.txt, which excludes the optional URLs,
-
and llms-ctx-full.txt, which includes the optional URLs.
They are created with the llms_txt2ctx command-line application, and the FastHTML documentation includes user information about how to use them.
The versatility of llms.txt files means they can serve many purposes:
- from helping developers understand software documentation,
- to giving businesses a way to outline their structure,
- and even breaking down complex regulations for stakeholders to understand.
- They are equally useful for personal websites, helping answer questions about someone's resume,
- for e-commerce sites, explaining products and policies,
- or for schools and universities, providing quick access to course information and resources.
Note that all nbdev projects now create .md versions of all pages by default. All Answer.AI and fast.ai software projects that use nbdev have already regenerated their documentation with this feature. For example, see the markdown version of fastcore's docments module.
Format
Currently, the most widely used and easily understood format for language models is Markdown. Simply showing where the key Markdown files are located is a great first step. Providing some basic structure helps the language model find the sources of information it needs.
What is unusual about the llms.txt file is that it uses Markdown to organize information, rather than a traditional structured format such as XML.
The reason for this is that we expect many of these files to be read by AI models and AI agents. Even so, the information in llms.txt follows a specific format that can be read with standard programmatic tools.
The llms.txt specification applies to files located at the site root at /llms.txt (or optionally in a subpath). Files that follow the specification contain the following Markdown sections in a specific order:
- An H1 heading with the name of the project or website. This is the only required part.
- A blockquote with a short summary of the project, containing the key information needed to understand the rest of the file.
- Zero or more Markdown sections of any type (paragraphs, lists, and so on), other than headings, containing more detailed information about the project and instructions on how to interpret the provided files.
- Zero or more H2-delimited Markdown sections containing "file lists" of URLs where more detailed information can be found.
- Each "file list" is a Markdown list containing required Markdown hyperlinks
[name](url), optionally followed by:and a note about that file.
- Each "file list" is a Markdown list containing required Markdown hyperlinks
Here is an example:
MARKDOWN# Title > Optional description goes here Optional details go here ## Section name - [Link title](https://link_url): Optional link details ## Optional - [Link title](https://link_url)
Note that the "Optional" section has a special meaning: if it is included, the URLs it provides can be skipped when a shorter context is needed. Use it for secondary information that can usually be omitted.
Existing standards
llms.txt is designed to coexist with current SEO web standards (including sitemap.xml and robots.txt).
While a sitemap lists all pages for search engines, llms.txt provides a curated overview for large language models. It can complement robots.txt by providing context for the content that is allowed. The file can also reference structured data markup used on the site, helping large language models understand how to interpret that information in context.
The approach of standardizing the file path follows the practice of /robots.txt and /sitemap.xml.
robots.txt and llms.txt serve different purposes:
-
robots.txt is generally used to let automated tools know which types of access to a website are considered acceptable, such as search indexing bots.
-
On the other hand,
llms.txtinformation is typically used on demand when a user explicitly requests information about a topic, such as when including coding library documentation in a project, or when asking a chatbot with search capabilities for information. We expectllms.txtto be primarily useful for inference, that is, at the moment a user is seeking help, rather than for training. However, if the use ofllms.txtbecomes widespread, future training runs may also make use of information inllms.txtfiles.
sitemap.xml is a list of all indexable human-readable information on a website. It cannot replace llms.txt, because it:
- generally does not list LLM-readable versions of pages,
- does not include URLs of external websites, even though they may be helpful for understanding the information,
- generally covers documentation that is too large to fit in a large language model's context window, and includes a lot of information that is not necessary to understand the website.
Example
Here is an example of llms.txt, a trimmed-down version of the file used by the FastHTML project (see also the full version):
MARKDOWN# FastHTML > FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications. Important notes: - Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services - FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte. ## Docs - [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features - [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options ## Examples - [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns. ## Optional - [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development.
To create an effective llms.txt file, consider the following guidelines:
- Use concise, clear language.
- Include short, informative descriptions when linking to resources.
- Avoid vague terminology or unexplained industry jargon.
- Run a tool that expands your
llms.txtfile into LLM context files, and test with several language models to see whether they can answer questions about your content.
Directories
The following are some directories that list llms.txt files available on the web:
Available integrations
There are various tools and plugins that help integrate the llms.txt specification into your workflow:
llms_txt2ctx- A CLI and Python module for parsing llms.txt files and generating LLM context- JavaScript implementation - An example JavaScript implementation
vitepress-plugin-llms- A VitePress plugin that automatically generates LLM-friendly documentation for a site according to the llms.txt specdocusaurus-plugin-llms- A Docusaurus plugin for generating LLM-friendly documentation following the llmstxt.org standard- Drupal LLM support - A Drupal recipe providing comprehensive support for the llms.txt proposal for any Drupal 10.3+ site
llms-txt-php- A library for writing and reading llms.txt Markdown files
Next steps
The llms.txt specification is open to community feedback. The GitHub repository hosts this informal overview, allowing version control and public discussion. A community Discord channel is available for sharing implementation experience and discussing best practices.
Cover Image by Franz Bachinger from Pixabay.