Tracking issue: rfc 1990 - add external doc attribute to rustc

RFC PR: rust-lang/rfcs#1990
RFC text: https://github.com/rust-lang/rfcs/blob/master/text/1990-external-doc-attribute.md

Current documentation

Summary of how to use this:

Add #![feature(external_doc)] to your crate.
Add a file "src/some-docs.md" with some docs to your crate.
Add an attribute #[doc(include = "some-docs.md")] to something that needs some docs. The file path is relative to lib.rs, so if you want a doc folder to live alongside src, then all your paths inside the doc(include) attributes need to begin with ../doc.

Summary of current status:

Initial implementation landed in #44781, on 2017-11-22.
Updates based on conversation in this thread landed in #46858, on 2017-12-21.
Please use this in your own projects and comment in this thread if something goes wrong! (Or even if it goes well! Knowing it's working as intended is great!)

Current tasks:

Land #44781
~~(promote the "failed to load file" warnings to proper lints?)~~ (Make them hard errors instead, per the RFC #46858)
Ensure line number info is properly preserved in doctest errors

cc @joshtriplett

At today's Cargo meeting, and interesting question was raised how Cargo should check if it should rebuild the docs when an external file with docs changes.

Here's some info on how dependency-tracking works today:

When doing cargo build, Cargo passes an --emit=deb-info flag to rustc, which causes it to emit dep-info files with .d extension, which lists what files were used for compilation (remember, only crate roots are explicitly mentioned in Cargo.toml). These files then used in fingerprint.rs to compute the fingerprint.

Curiously, cargo doc does not emit dep-info at all, and fingerpint.rs does not use depinfo when doing doc. Instead it asks Source for fingerprint which for the local repository boils down to listing all files in a directory with Cargo.toml.

TL;DR: touch Readme.md today causes cargo doc to rebuild docs for the current crate, which might not be the most efficient thing to do, but which should work with external documentation, unless it is fetched from a place outside of Cargo package.

@matklad So, it sounds like the current behavior is correct but could be made more efficient. That's better than being incorrect.

So, it sounds like the current behavior is correct but could be made more efficient.

There's an edge case where it might not do rebuild when it should, if in src/lib.rs you have something like

#[path="../../bar.rs"]
pub mod bar;

Or

include!("../../bar.rs")

That is, if you touch files outside of the Cargo package directory.

Other than that yes, current behavior is correct!.

This probably won't cause any issues in the real world, but related to the discussion above about dependency tracking, I noticed that #[doc(include="file")] doesn't wind up in the dependency info output by rustc --emit=dep-info, which doesn't seem right. (In practice, since doc comments presumably don't affect the compiler output, it probably wouldn't cause any actual correctness problems).

Compare vs. include_str! for example:

luser@eye7:/tmp$ rustc +nightly --version
rustc 1.24.0-nightly (560a5da9f 2017-11-27)
luser@eye7:/tmp$ cat > test1.rs <<EOF
> #[allow(unused)]
> const s: &str = include_str!("other-file");
> EOF
luser@eye7:/tmp$ cat > other-file <<EOF
> hello!
> EOF
luser@eye7:/tmp$ rustc +nightly test1.rs --emit=dep-info -o test1.d
luser@eye7:/tmp$ cat test1.d
test1.d: test1.rs other-file

test1.rs:
other-file:
luser@eye7:/tmp$ cat > test2.rs <<EOF
> #![feature(external_doc)]
> #![doc(include="other-file")]
> EOF
luser@eye7:/tmp$ rustc +nightly test2.rs --emit=dep-info -o test2.d
cluser@eye7:/tmp$ cat test2.d
test2.d: test2.rs

test2.rs:

It looks like include_str! et. al. simply call cx.codemap().new_filemap_and_lines(&filename, &contents) to make this happen:

rust/src/libsyntax/ext/source_util.rs

Line 152 in a0db04b

// Add this input file to the code map to make it available as

so presumably doing the same thing inside the implementation of #[doc(include="..")] would fix this (probably here, I'm guessing):

rust/src/libsyntax/ext/expand.rs

Line 1117 in 632ad19

let include_info = vec![

I neglected to mention, but it looks like there's even a test to ensure that files used in include_str! and include_bytes! wind up in the dep-info: https://github.com/rust-lang/rust/tree/b1363a73ede57ae595f3a1be2bb75d308ba4f7f6/src/test/run-make/include_bytes_deps

Humorously enough, i asked about that when i was writing it:

<misdreavus> hmm, should i add files loaded by doc(include) into the codemap? include_str does
<jseyfried> I would lean against

Judging from the above discussion in this thread, it may be prudent to add it, but it also may not be necessary. I'd defer to @rust-lang/compiler for that, i guess.

I personally would find it quite unfortunate if changing the included file doesn't lead to rebuilding the code. (One day, incremental compilation should make that a very fast rebuild.) I realize that usually it won't affect the generated code, but I could imagine a syntax extension of some kind processing doc comments (after all documentation has been normalized to the equivalent of a doc attribute containing a string), and it shouldn't be the job of that syntax extension to handle dependencies on included files.

The current implementation seems to interact poorly with #![deny_missing]:

$ cat test.md
foo
$ cat test.rs
#![doc(include = "test.md")]
#![feature(external_doc)]
#![deny(missing_docs)]

fn main() {}
$ rustc +nightly test.rs
error: missing documentation for crate
 --> test.rs:1:1
  |
1 | / #![doc(include = "test.md")]
2 | | #![feature(external_doc)]
3 | | #![deny(missing_docs)]
4 | |
5 | | fn main() {}
  | |____________^
  |
note: lint level defined here
 --> test.rs:3:9
  |
3 | #![deny(missing_docs)]
  |         ^^^^^^^^^^^^

error: aborting due to previous error

I just opened #46858 as a grab-bag to address the discussion so far.

Update: The last change to this feature landed on 2017-12-21. Since this thread has been pretty silent, i'm going to assume that it's fairly acceptable?

I'd like to establish some kind of stabilization plan. @rust-lang/dev-tools, what's an acceptable amount of time for this to bake? I'm not sure of any major users right now, but i know a few people have already approached me who would like to see this hit stable.

I think for a feature this small/contained, one stable cycle is enough. I think we've had more than that.

Also: any stabilization lands in the Nightly channel first, and only in Stable channel two release cycles later. If something comes up in that extra time, as far as I understand we can still make changes before it lands in the Stable release channel.

We don't usually make a timetable up front for stabilisation since how long it takes depends a lot on how much the feature is used. I think that if there is some evidence that the feature is widely used and liked, and there are no significant bugs reported then we could stabilise in six weeks or so, but if there is not so much take up, then we would wait longer.

I'd really like to use this in my DNS crate. I've got a lot of generated documentation from IANA parameters and the IETF's RFC index that would be much easier to integrate with this feature.

Can confirm this is working well for me, I did notice the final implementation bullet point ("Ensure line number info is properly preserved in doctest errors") though. I would extend that to say that the line number info should be correctly preserved with passing doctests as well, currently it says test src/lib.rs - (line 18) ... ok when I have the documentation in an external file.

Happy customer reporting in! - Using this for all our repos on https://github.com/datrs and it's working nicely.

One nit is that images format a bit weirdly on rustdoc (example), but feel that might be more of an issue with rustdoc than this feature specifically.

Oh, and probably the first time I saw error in src/lib.rs on line xx I was surprised that README.md gets inlined and never referenced directly. But that was just a small "oh" moment.

Ohh, and thinking about it - another small oddity was that when linking to files in subdirectories it assumes paths are relative from the root of the crate, rather than relative to the file (example). I figure there's probably good reasons for this, but still took me a bit to figure out.

But yeah, Tl;Dr: real happy with this feature; it's the only unstable feature I currently rely on and super stoked for it to stabilize 😁

Refs

Misattributed source spans and paths not being relative to the file doing the inclusion both sound like bug IMO.

The paths being relative to src is explicitly specified in the RFC. I'm also of the opinion that it's wrong, but I guess changing it would need a new RFC now?

I see. Relative paths being relative to anything other than where that path is itself seems very wrong to me. That’s not how hyperlinks work in anything that has hyperlinks. include_str! is not an accident, it is absolutely doing the right thing here. There is some discussion of it in the RFC thread starting it rust-lang/rfcs#1990 (comment), but I don’t understand the arguments:

@mgattozzi, you wrote little more than “I would think relative to the crate root would be the best way”. Could you explain why?
@nrc, what is the concern with macro hygiene? Don’t the #[doc(include = "foo.md")] tokens carry spans that are associated to a source file?

Also, if some find useful to have crate-relative paths there is no need to have a different attribute or dedicated Rust syntax since paths/URLs already have syntax for this: for example #[doc(include = "/README.md")] where the initial / indicates the "root".

My expectation of being relative to the crate root was thinking "Why would I put documentation with src?" It seemed to me like it would be extra work to have to keep putting something like "../docs/my_thing.md" as the path if it was relative to src rather than "docs/my_thing.md" which would be relative to the crate root. I get the argument of having things relative to the file, but from a usability standpoint it just seemed backward because if I made a cargo project I would think about the layout from the root, not where the file is.

@mgattozzi So if I understand correctly you’re not saying that relative paths make more sense based at the crate root, only that you wish in some cases for paths based at the crate root because that can be more convenient. So #[doc(include = "/docs/my_thing.md")] with an initial slash would work for you?

Basing them at the crate root makes more sense to me because that's how I think of projects as a mental model and also it makes things more convenient. I'd drop the first slash since to me that implies it's like the root of the file system. I'd either leave it out or put ./ as is standard posix convention, but also I can see that being a pain because Windows is also a thing and paths are different for different OS

But I do realize this might not work in cases where say we have a file not in a cargo project. Where does that include take data from? In most cases relative to itself would be the correct choice.

I believe a lot of the RFC conversation at the time had been centered around those semantic differences. That's why the RFC mentions having a starting point, then following it up with another one to flesh that out more.

Oh, and probably the first time I saw error in src/lib.rs on line xx I was surprised that README.md gets inlined and never referenced directly. But that was just a small "oh" moment.

Yes, the error reporting mechanisms haven't been made aware of the file inclusions. The data is there, but it's not in the right places.

(chatter about the root of inclusion)

The reason that lib.rs was chosen was because rustdoc has no concept of Cargo projects, and also because the path behavior of include! and friends is considered a mistake for hygiene reasons. Moving a source file which uses one of these macros changes its behavior. At least, that's the explanation i remember seeing.

I would not be opposed to adding a parameter to rustdoc to set this root arbitrarily, with the expectation that Cargo could use this for cargo doc invocations. The only issue would be that passing this CLI argument would be a pain until Cargo supported it. And i'm not sure how Cargo would be convinced to handle passing the -Z unstable-features flag for such a feature while the flag itself was unstable.

@mgattozzi is it perhaps worth considering how modules will work in the future too? Not sure what the status is of RFCs, but I was under the impression all imports are moving to be relative before year's end.

If that's the case, it might be worth following that change in this feature too.

@yoshuawuyts the RFC was written last year shortly after Rustfest Kiev so a lot of ergonomics decisions were in flux at the time! I'm sure there's some conflicts between this RFC and others in that sense. If it is then we should probably follow convention for consistency's sake!

While experimenting with this feature, I found a rather "funny" bug: sometimes, when runing cargo test --doc, the src/lib.rs file (the one having the #![doc(link = "README.md")]) is deleted from the filesystem, but also the rustdoc.exe binary, resulting in the following error:

$ cargo test --doc
...
error: toolchain 'nightly-x86_64-pc-windows-msvc' does not have the binary `rustdoc.exe`

And then I need to reinstall the nightly toolchain to get it back.

The project is a tiny library (repo link), where the external doc is hidden behind a feature flag (doc), so to run the tests including those from the included file, run the following command with an up-to-date nightly:

cargo +nightly test --doc --features doc

From what I've seen, it seems to happen only when running the doc tests, and never when running cargo +nightly doc --features doc.
The repo contains a workspace with three projects, but only the src folder is documented this way, and use only one external file (src/README.md).
FYI:

nigthly: 2018-05-16 (the last up-to-date as of this comment)
target: x86_64-pc-windows-msvc

Still, I wanted to say, even though this bug can be quite painful (you need to commit your src/lib.rs before running doc tests, and reinstalling a toolchain on Windows with an antivirus software can be quite long), I wanted to say thank you for this feature, editing a plain markdown file for writing documentation while keeping it in the built result is super cool. Combining this feature with what BurntSushi did for csv, namely using a module consisting solely of documentation, allowing him to integrate a cookbook in the doc, can be quite interesting, and help with the short comings of rustdoc.
And having a markdown file for doc looks great when browsing the repo too! Even though this feature might not look vital, I think it can help a lot library devs to write great documentation easily.

That's very alarming! I wouldn't expect it to be able to delete files like that. I wonder what awkward condition is being hit.

I tried to look at your library link, but it looks like the code is private. >_>

I tried to look at your library link, but it looks like the code is private. >_>

Oops, you're right, it's fixed now

Just a quick followup on the "bug" I found: it was McAfee being overly zealot. Apparently, rustdoc.exe has a bad reputation, and so it should be put in quarantine, with all files used (including src/lib.rs).

Sorry for the false report!

I've had a quick play. This is pretty great. I was mainly enthusiastic about using this to include generated documentation, but after using it, writing documentation without /// (particularly with code blocks) is nearly as compelling. 👍

Can I propose we move to stabilize this feature? Stabilization was last proposed in February, and it's been a while since.

Since then there's been discussion, but at this point in time there seem to be no unresolved issues.

If stabilization sounds like a good idea, I'd be happy to help out in getting it there! I don't know which steps to take though? Is it just a PR to rust-lang/rust to change the feature to be available in beta? A bit of guidance would be really helpful! ✨

Thanks heaps!

cc/ @QuietMisdreavus @Manishearth @nrc

One specific thing that i'm not sure we investigated is how line numbers are treated when a doctest is pulled from a #[doc(include="")] file. Such doctests are being picked up and tested, but i fear that if one of those fails, rustdoc/libtest will report the wrong file/line information. I'm not sure whether we fully solidified that aspect of this feature, and i want to make sure we report the right information (preferably through a test) before we move to stabilize.

Relative paths being relative to anything other than where that path is itself seems very wrong to me.

I was reminded of this by code in this repository. In fact every non-test use would be better off with relative paths being relative to the use site:

$ rg '^\w*#\[doc\(include' src/lib**
src/libstd/os/raw/mod.rs
23:#[doc(include = "os/raw/char.md")]
38:#[doc(include = "os/raw/char.md")]
53:#[doc(include = "os/raw/schar.md")]
55:#[doc(include = "os/raw/uchar.md")]
57:#[doc(include = "os/raw/short.md")]
59:#[doc(include = "os/raw/ushort.md")]
61:#[doc(include = "os/raw/int.md")]
63:#[doc(include = "os/raw/uint.md")]
65:#[doc(include = "os/raw/long.md")]
68:#[doc(include = "os/raw/ulong.md")]
71:#[doc(include = "os/raw/long.md")]
74:#[doc(include = "os/raw/ulong.md")]
77:#[doc(include = "os/raw/longlong.md")]
79:#[doc(include = "os/raw/ulonglong.md")]
81:#[doc(include = "os/raw/float.md")]
83:#[doc(include = "os/raw/double.md")]

I would love to use this in my project, but crates.io currently rejects packages using this. Should crates.io support unstable features for packages that require nightly anyway?

Crates.io should be able to accept crates that use unstable features. For example, Rocket uses unstable features but is published on crates.io. What error are you getting when you try to publish? You may need to tell Cargo to include your markdown file(s) in your package by adding a include = ["..."] line to your Cargo.toml. The Cargo Book has a section on this parameter.

@QuietMisdreavus ah, you're totally right! My crate is in a workspace, and the README is at the workspace root. It worked when I created a symlink in the crate root. Thank you :D

I've been running into an issue using this feature.

The particular crate which I would like to use this feature for is https://github.com/thedodd/wither. The crate uses the stable channel. I've been attempting to switch this feature on using the following pattern:

#![cfg_attr(feature="docinclude", feature(external_doc))]
#![cfg_attr(feature="docinclude", doc(include="target-docs.md"))]

NOTE: edited to correct a typo. Accidentally pluralized external_doc, it was singular in the code though. Was only a transcription typo here.

And then I simply invoke cargo +nightly doc --features="docinclude" --open ... seems as though it should work, the compiler doesn't give me any errors, but the docs do not show up. I have the docinclude feature setup properly in my Cargo.toml, so I don't think the issue is there (normally the compiler would complain if there is an error on that front).

It does work as expected if I remove the cfg_attr() wrapping and just enable the feature, but then the crate would require the nightly channel.

Thoughts?

I was just able to get it working with the following, running cargo +nightly doc --features docinclude:

#![cfg_attr(feature="docinclude", feature(external_doc))]

#![cfg_attr(feature="docinclude", doc(include="asdf.md"))]

It's worth noting that the feature gate is external_doc, not external_docs which you have in your sample. One other note is the location of asdf.md here was right next to lib.rs. (However, since you say it works without the cfg_attr, these are probably not that useful...)

@QuietMisdreavus sorry, I noticed my typo of external_docs earlier when I transcribed it here, but forgot to update. I am using external_doc in the code though.

Well, that's quite strange. I will keep hacking on it and see if I can get it to work. Thanks for the feedback.

@QuietMisdreavus so, after hacking on this a bit more, it appears as though I can compile the following code on stable (this time, exact copy-paste):

lib.rs

#![cfg_attr(feature="docinclude", feature(external_doc))]
#![cfg_attr(feature="docinclude", doc(include("../../README.md")))]

Cargo.toml:

# ... snip

[features]
docinclude = [] # Used only for activating `doc(include="...")` on stable.

[package.metadata.docs.rs]
features = ["docinclude"] # Activate `docinclude` during docs.rs build.

The command I run: cargo clean && cargo build --features docinclude.

Cargo version:

$ cargo --version
cargo 1.30.0 (a1a4ad372 2018-11-02)

Rustc version:

$ rustc --version
rustc 1.30.1 (1433507eb 2018-11-07)

The end result is that the build goes through, but still no included docs.

Running the same exact thing, but with a nightly compiler, yields the same result. Successful build, but no docs.

@thedodd it's doc(include = "../../README.md") not doc(include("../../README.md")). It should be a compile error if you don't specify a filename to #[doc(include)] but currently it's just ignored.

So, another pertinent bit of information is that this repo is a workspace with two crates, wither & wither_derive. When I cd down into the wither directory and then cargo build --features docinclude, the compiler errors out as expected. Invoking with +nightly causes compilation to succeed.

However, once again, the docs do not appear when running from the crate specific directory (as opposed to workspace dir) in any of the cases mentioned above. I've cargo cleaned and such ... doesn't seem to be making a difference.

For reference sake, I have pushed a branch called docinclude in this repo. Let me know if there is any other diagnostic info I can pass along.

@ollie27 you may have just saved my life ... testing that out now.

@ollie27 thanks for being the greatest human alive right now. It's always the small things. Thanks for clearing that up for me. I must have edited the attribute at one point while I was setting up the cfg_attr bits and thought that it was working the same way. It appears to be working as expected now.

Please use this in your own projects and comment in this thread if something goes wrong! (Or even if it goes well! Knowing it's working as intended is great!)

I wanted to quickly note that I too love this feature.

I like to document my libraries as much as is required to make them easy to use by anyone interested. But obviously, I dislike having outdated docs no longer reflect the reality of the code. This works really well when using documentation examples that get automatically added to the test suite.

One downside to this approach is that if you want to make your doctests also remain readable/understandable, you end up duplicating your tests many times, each time commenting out a different part of the test, to only show the part that is relevant to the reader of your documentation.

(side note: it would be nice if you could designate a block as "include this in all other code blocks in this documentation section", but I can see how that can get pretty complicated at some point, and perhaps even reduce readability of the unrendered documentation)

I am glad we have this capability in Rust, but the downside is that inline code documentation can become extremely long (hundreds of lines for a single trait method happens), which breaks my flow when reading the code.

Ideally, I'd want to only see a short description of the method in the code, but have a more robust example and description in the documentation.

This feature lets me do exactly that 👍 ❤️

You can see a small example here: rustic-games/things@ae5280d

I plan on using this a lot from now on. The doc feature flag proposed by others here works to keep the crate on stable, while building the docs with this nightly flag, but it would be nice if this feature gets stabilised sooner rather than later, it's a real boon to improved documentation standards in Rust.

about the bad line number info: yeah, I too noticed this while debugging an example that was failing. It was not a big deal, but it does give you a short pause, before you realise what's going on

I have a branch pending that improves the diagnostics for this feature, but it's based on #56258, so I'm going to let that PR get merged before opening another PR.

It seems like a really underpromoted possibility of this feature (though it obviously hasn't gone unnoticed) is the ability to have your README automatically doctested. I think on that basis alone it should be possible to generate a lot of interest and engineering effort toward getting this working well and stabilized.

I've got a proof-of-concept project for using this feature to test code examples in README.md: https://github.com/abonander/readme-doctest-poc

I chose to apply the attribute to a private typedef since no code gets generated for those. Fixing the file/line numbers for test failures in included files is definitely important though; I can see it being a real pain trying to fix failures if you have a lot of examples (it's not exactly common to add a lot of context to asserts in examples since they're usually self-evident).

Another thought: this would be significantly cooler if the Rustdoc CSS had a class that hides its contents so you could just reuse your README as your crate root docs and just hide stuff that doesn't necessarily need to be in the docs like license information or contributing instructions or CI build status.

That use-case is unfortunately a non-starter for most while this feature is unstable anyway. I don't want to require nightly to build documentation (or just sometimes not have crate root docs).

@abonander the way I've usually gone about this is to use cargo-readme to generate the README from the crate-level docs, but that has the issue that relative links won't work.

The paths being relative to the root module's file still seems like the wrong choice to me (at least for the use-cases I have). @yoshuawuyts and @SimonSapin appear to agree and @mgattozzi's last comment seems to be ok with changing it (and I haven't seen any other opinions noted since implementation). Is there any chance of re-discussing this now that there's some usage data to base it off?

I'm sure @QuietMisdreavus has a more informed opinion of this now then I would regarding this. I'm of the opinion of the least amount of surprise meaning consistency with what Rust has done before, whether it's relative to the file or the crate root (I'm not sure which it is tbh).

What Rust has done before is that the include!, include_byte!, and include_str! macros take a path relative to that of the source file that contains the macro invocation. This is consistent with how links work in HTML, or URLs on the web in general.

I have the same opinion about source file directories as in my comment from last year:

The reason that lib.rs was chosen was because rustdoc has no concept of Cargo projects, and also because the path behavior of include! and friends is considered a mistake for hygiene reasons. Moving a source file which uses one of these macros changes its behavior. At least, that's the explanation i remember seeing.

I would not be opposed to adding a parameter to rustdoc to set this root arbitrarily, with the expectation that Cargo could use this for cargo doc invocations. The only issue would be that passing this CLI argument would be a pain until Cargo supported it. And i'm not sure how Cargo would be convinced to handle passing the -Z unstable-features flag for such a feature while the flag itself was unstable.

Thanks! I didn't know if this feature.
I plan on having a README for the library, and also for each module which has its own directory.
This use case benefits from source-file relative path, and I think this structuring would be the cleanest regarding GitHub projects.

I don't really get the reasoning behind the discrepancy in relative paths for #[doc(include)]. It seems to me like this is just needlessly confusing for people trying this feature for the first time. All other parts of Rust that currently act on files at compile time (#[path] on a module declaration, and the include_x! macros) base their paths off of the containing file.

The point that "rustdoc has no concept of Cargo projects" doesn't make much sense to me considering that rustc also doesn't know anything about Cargo project structure (other than module structure, obviously, but rustdoc also knows that).

The existing behavior of #[path] and include_x! might have been a mistake, but I think it would also be a mistake to introduce a similar-looking feature with different behavior without a concrete plan to move everything to that behavior. I guess #[path] and include_x! would be fixable in another edition, since all the include_x! macros are built-in, so changing their behavior based on the edition of the crate that names the macro would be possible.

Is there some way of using this feature with some kind of fallback for stable Rust? I'd love to use this, but completely losing docs for any stable users would be a real deal killer.

@shepmaster Here's my solution that works with docs.rs since they build with nightly.

(@jonhoo It also fixes the problem that you pointed out)

Add a feature external_doc to your Cargo project, and configure docs.rs to use it (Cargo.toml):
```
[features]
external_doc = []

[package.metadata.docs.rs]
features = [ "external_doc" ]
```

Enable the unstable feature when your crate feature is on, and include documentation. Leave an empty placeholder documentation line to resolve any missing docs lints (lib.rs):

#![cfg_attr(feature = "external_doc", feature(external_doc))]

// repeat this for all includes everywhere:
#![cfg_attr(feature = "external_doc", doc(include = "../README.md"))]
//!

#![deny(missing_docs)] // passes now

// ... the rest of your crate

EDIT: To run locally, the command is cargo +nightly doc --features external_doc.

To show docs on stable, it's a bit messy, but you could use #![cfg_attr(not(feature = "external_doc"), doc("..."))] manually. Maybe it can be shortened to a macro invocation like stable_doc!("..."). Personally I just use docs.rs 99% of the time, so I've never really taken the time or effort to do that.

Are there any plans to stabilize this?
From the compiler point of view, I'd like to get rid of this if only possible, in favor of regular #[doc = include_str!("doc.md")].
(Which already partially works, but requires some dance with macros and #[doc = $expr].)

Wouldn't that make it harder to provide correct line numbers in errors for doctests run from included files? include_str can't provide spans for individual tokens inside the string.

Bumping, this is a really needed feature.

@abonander

Wouldn't that make it harder to provide correct line numbers in errors for doctests run from included files? include_str can't provide spans for individual tokens inside the string.

No, both methods end up as strings in the #[doc] attribute after expansion.
The string tokenization is done later by rustdoc itself.

unpopular opinion but I think this should be strictly for translated docs.

by default rustdoc should take the doc from the source. if a specific language/translation is desired it'll use the docfile. this makes it easier to browse the source as you have the docs straight on the source so you just read things as normal. but you also get the benefits of translation.

That it's beneficial for translation discounts other use cases... why?

the point of having in-source documentation is that you don't need to look elsewhere. you can integrate it into your workflow more easily, and thus the documentation more accurately reflects the code and vice-versa. this makes your life easier when reading through the code.

I would never work on a project where documentation is separate from source code, unless such documentation is a translation of the embedded documentation. and I'd be strongly against using such projects, as well. (altho we really need some sort of versioned documentation if we want to make sure those translations are in sync. but that can be deferred to an external tool or w/e, maybe.)

For things like guide-level documentation, these don't really belong in the source, but are very useful on docs.rs (there is an issue to run cargo-external-doc on docs.rs, but I think it'd be nicer to just have it be part of cargo doc anyways). Having to place /// before every line instead of just saying "read this markdown file over here" (where editors could actually give syntax highlighting and other assistance) doesn't seem like an improvement to me.

Translations of API docs would need language tags as well and that isn't mentioned here at all.

I am in favor of having the capabilities of external docs. To respond to a few points...

I would never work on a project where documentation is separate from source code

That's fine; this feature will not force you to use it. I would assume documentation in the source will always be available due to backwards compatibility guarantees if nothing else.

I'd be strongly against using such projects

I don't see why but it frankly doesn't matter — you can choose to use or ignore any project for any reason, and adding this capability doesn't change that. You can choose to not use any project with the letter "a" in the name too, but that doesn't mean that others should be limited.

For things like guide-level documentation

This is exactly my usecase and I am happy with that structure.

where editors could actually give syntax highlighting and other assistance

Some editors do support this already with the inline comments. Mine actually supports editing the Rust code blocks as Rust code inside of a doc comment inside of a Rust source file!

That capability didn't stop me from preferring to split it out into separate files, however.

by "language tags" do you mean --trbasedir=./foo/pt-BR/ and a "doclangs" array in Cargo.toml?

I honestly strongly prefer guide-level documentation, DSL specifications (e.g. regex string description), etc to be in the source file itself.

if you need them separate do it like Rust: write a book.

Even for things like deployment guides and such? These can be a handful of pages, but should not be anywhere near book-size. In other words: there's definitely a middle ground between "API docs" and "book" in the world.

if you're documenting an app... consider using the right tool for the job.

keep in mind rustdoc is itself documented with a rustbook. it is tiny. and that's okay.

#67121 implements #[doc = include_str!("my_doc.md")] which I hope to make a replacement for this feature.

If it doesn't need the //////! in the included file, how is it not this feature, save syntax semantics? Seems perfect.

Adding the language team since the actual implementation (#44781) ended up making changes to the then src/libsyntax/ext/expand.rs which is part of the compiler (and therefore this has language spec implications of the #[doc ...] attribute, which rustdoc uses, but is not part of rustdoc).

Is there a plan to move this into stable?
If so where can i find what is left to be completed?

#78837 implements #[doc = include_str!("my_doc.md")] once more, this time for real, I hope.

🎉

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835), so unassigning myself.

~~For reference, #[doc = include_str!("my_doc.md")] is now stable in 1.50 via-- #78837~~

Edit: Ooh, how embarrassing. Still behind #![feature(extended_key_value_attributes)] feature gate. (tracking #78835) Thanks @jyn514

@trevyn include_str is not stable, it's still behind a feature gate.

To be clear, include_str! is stable, #[doc = include_str!] is not.

EDIT: clarified

doc(include_str) does not exist. extended_key_value_attributes is what's unstable (#[doc = include_str!])

Yeah, that's what I meant. I guess I mixed things up even more 😆

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835)

I opened #82539 deprecating doc(include). Assuming that makes it past FCP, I'll give it a few months then remove the feature altogether.

That relative paths niggle: Has anyone written a macro that fixes the relative links?
#[doc = deepen_relatives!(include_str("../README.md"), 1)]