Feature Request: Add support for `llms.txt`
Closed this issue · 4 comments
This allows AI agents to use the available docs more effectively to write code
I don't fully follow how that would look like exactly. From the description it sounds like a markdown version of all doc pages would have to be generated? Including bigger crates with millions of pages?
Plus some table of contents?
Do you have a source that would tell us how widely used this standard already is? Which providers respect / use it?
Right nw I'm not convinced this is worth the effort and that it needs to be integrated directly into docs.rs.
Btw, there is always the option that you build a small proxy service based on the rustdoc json output that generates the format you need.
There's also an overlap with #2808 .
There would be two versions, one is just a table of contents like this example, the other would pretty much be the entire content concatenated together, like this example.
A lot of large service providers include it in their docs sites, explaining how exactly to use various features of their SDKs. Here's an example list https://github.com/thedaviddias/llms-txt-hub#llm-tools-and-resources. There has been rapid adoption in the past 6 months.
AI agents would be given this llms-full.txt or llms.txt, and they would use the docs to write better code, else they usually just hallucinate. In code editors like Cursor, the user can just paste the link of a library's llms-full.txt file and have their agent use the full feature set of the library.
Thanks for mentioning #2808, the MCP suggested there would suffice my needs for now. Also thanks for the tip on the rustdoc json output, I'll try building a proxy around that.
I think adding support in first party docs would help a lot of new rust beginners use LLMs more effectively. But, please feel free to close this as not planned if that's not something worth the effort yet.
There would be two versions, one is just a table of contents like this example,
This would be a TOC for one release of one crate? Or a TOC for the whole of docs.rs?
the other would pretty much be the entire content concatenated together, like this example.
the biggest crates have >5 GB of docs with several million pages, for every version. I imagine it would be tricky to have all of that in one txt file.
also thanks for the tip on the rustdoc json output, I'll try building a proxy around that.
That would be awesome!
Even though only AI crawlers give me quite a hard time , I'm not 100% against supporting them a little more. But due to very limited capacity having a actively used POC would be a good idea. Then you can go through all necessary iterations before we even think about integrating it.
The same is true for the MCP.
This would be a TOC for one release of one crate? Or a TOC for the whole of docs.rs?
It'd be one per crate, not for the the entire docs.rs
the biggest crates have >5 GB of docs
That's a lot... I'll check how others solve it
having a actively used POC would be a good idea
Yeah that makes sense, I'll do what I can to contribute
Thanks, closing this as not planned for now. We can reopen it later when needed