Making docs optional again
Closed this issue · 69 comments
See as docs are required again due to #903, can we track what's necessary to make them optional again here? As one of my laptops doesn't have space for the docs (16G disk :( ), I'd be interested in helping fix the relevant issues. The two mentioned issues I've seen are:
- #901, which would require a useful message/action when docs are requested and not present
- #891, which would install the docs by default to help new people
- #1006 Add support for default installed optional features to manifest
Are there any others? And if you prefer to track this information elsewhere, please close this bug and point me in the right direction!
Another problem are network file systems. For me it takes minutes to install the docs, the rest finishes almost instantly.
I'd prefer to leave docs as optional since I use the online version only and don't want to waste disk space or increase rustup update time.
The major missing feature is that the manifest format needs to be extended to understand optional features: #1006.
Yes, please make docs optional again! We work a lot with Raspberry Pis where space is very limited!
This would certainly be nice for CI.
And when updating a nightly or when different projects use different nightlies (with a rust-toolchain file) it always takes a long time to install/update them, it would be much faster without the docs.
The major missing feature is that the manifest format needs to be extended to understand optional features
Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).
The problem is (among others) that $toolchain/share/doc contains nearly 15000 files which is a big pressure on disk during toolchain updates - on Windows it takes a really significant amount of time to install rust-docs.
From #903:
It seems that most people are comforted knowing docs are available by default, so I want to turn them back on, then add some more options to remove them.
I am fine with default (though I really doubt that those who really need docs can't install them as a component), but let's make them as opt-out, so rustup would remember during the next update that rust-docs doesn't need to be downloaded.
I can confirm that it takes a long time to install the docs on windows, and I'm dealing with multiple nightlies for different projects (with rust-toolchain files)..
But I'd prefer if the docs were opt-in, because most people only use the online docs. An internet connection is required to be able to use Rust anyway..
The major missing feature is that the manifest format needs to be extended to understand optional features
Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).
@pravic The issue is that the manifest does not have a way express opt-out components, only opt-in.
The playground docker images are also about 300MB larger than they need to be because of the docs (which I'm working on removing at the moment).
It takes ages for us to install Rust on Travis/Windows. Any progress on sorting this out? Really delays builds by 10 minutes or more which is impossible to work in productive way without wasting 2 hours on CI each day.
I have a plan for profiles which will address this. I hope to implement in the next few weeks
@nrc I'm sure this is nothing new to you, but wanted to provide another data point. Building a "Hello World" executable on Travis CI takes 8x longer on Windows. The vast majority of the time is spent installing rust-docs.
https://travis-ci.org/johnthagen/min-sized-rust/builds/465623377
The profile work will have a great impact on CI. ❤️
@nrc What's the status on that? rust-docs takes minutes to install when I never use them.
What's the status on that? rust-docs takes minutes to install when I never use them.
I've started implementation, but been away the past few weeks. I'm going to try and get it done soon.
Looking forward to being able to consolidate to just Travis CI once this is fixed.
some additional context/experience
yesterday i helped ~20 ppl with windows laptops install rust
several people complained about this step and some thought the installation was broken
As an alternative to making docs optional, rustup could gain the ability to just not extract the docs until the user actually needs them. That way the user would have the docs if they go without internet, but they wouldn't have to suffer through the long installation time until they actually need the docs.
I still think there is value in making it optional. For CI runs where installing anything but the bare toolchain is a waste of time. Ignoring docs and other things on all platforms for CI could save quite a lot of time.
Sounds like extracting them on first use is more effort, and someone who chooses not to install the docs probably knows what they're doing. Both could be done, but I'd rather have them optional.
Is there consensus on whether the docs should be installed by default (as they are now)? I suspect that the majority of rustup installations don't use docs, that it would make sense to make it configurable for the interactive install, w/ a corresponding cli flag, maybe printing a note saying how to get the docs.
This seems like such an easy issue to fix, and it's been dragging on so long.
I remember this thread from a couple years ago (funny enough, it was started by you @brson 😄)
There was some feedback on it. I do remember at the time I was against removing rust-doc by default. but I didn't fully realize the ramifications on Windows for how painfully slow the install was (#1540, #769). This is especially painful for tracking nightly as you need to wait for the rust-doc install more often (up to once a day) and of course most of all Windows CI.
Yeah @johnthagen it's all pretty lolworthy. Maybe there's no right answer. Installed by default but decompresed on first use might be a good compromise - presumably its not the download that hurts windows users but touching so many files. Plus options to not install docs at all perhaps.
Decompressing on first use does make the assumption that everybody access the toolchain docs via rustup (or maybe cargo?). I'm not sure that's true, whether its because somebody wants to use a different browser or because rustup doc can't launch a browser.
rustup could even consider rust-docs installed w/o even downloading the tarball - download and decompress on first use. I think that's possible. It would though make the network required for first view of the docs. Probably the size of rust-docs tarball is not so much we need to care.
Just a strawman:
- don't make rust-docs optional again, on the assumption the tarball is sufficiently small that the download time and space for storing it are reasonable for most CI purposes
- rustup stashes rust-docs tarball somewhere and marks it as installed but not decompressed
- rustup puts placeholder index.html's where the main index, book, reference, and std would be that indicates the docs need to be decompressed, here's how
- add
rustup doc --decompressfor good measure - when the user runs
rustup component add rust-docsbut the docs are already installed but not decompressed, tell them so, and to runrustup doc --decompress
Just FYI, the docs tarballs are about 30 MB in tgz format. It's not nothing.
I think it's way easier to not install rust-docs by default and tell folks to run rustup component add rust-docs if they want them. It must be a very small effort to opt-out the docs from being installed by default, not sure what causes this issue to linger here for years.
I'd say don't install rust-docs by default, but prompt about it in rustup-init.
I'd say don't install
rust-docsby default, but prompt about it inrustup-init.
With a short warning that on Windows installing rust-docs takes a long time, to help new Windows users.
rust-doc should be installed by default and have an opt-in (command line argument) to not install it. Because when you need the offline doc (no internet), there's no way to get it.
- how do you install Rust?
- how many toolchains do you usually have installed?
@lnicola Installing rust-doc by default is the decision of Rust team.
All I want is to have an opt-in way to not install it, especially when used in CI.
alternative strawman design following what I believe the OP was based on:
- the
rust-docscomponent is marked "optional but default" in the manifest (#1006) rustup installwill by default install "optional but default" components--[no-]default-optional-featurescan be specified torustup installto force enable/disable installing "optional but default" components for a specific installationrustup updatewill not install "optional but default" components that are not currently installed (maybe it should print a message about them?)rustup component removewill allow removing "optional but default" components from a toolchaindefault-optional-features = true/falsecan be added to~/.rustup/settings.tomlto change the default behaviour for "optional but default" components withrustup installrustup-initwill prompt the user about settingdefault-optional-featureswhen installingrustup
(EDIT: keeping the component compressed and decompressing on use seems like it could also be worth it, but personally I would first prefer to have the ability to opt-out completely for most of my toolchains, only keeping docs for nightly and not the half-dozen nightly-date installs I have, along with removing the overhead from CI).
If I remember correctly the consensus lately is that we’d prefer the docs to be included by default because the first time someone might need a local copy is when they’re offline, and at that point it might be too late to download them. But we want the ability to disable them for scenarios like CI. Unfortunately the rustup manifest format does not support yet having a component that is optional but enabled by default, so this needs some implementation work.
Separately, the rustdoc team wants to find a way to ship a compressed archive that can be used in place, so that there isn’t a need to create so many files: https://paper.dropbox.com/doc/Rustdoc-2019-roadmap-meeting--AXcjEXi3H6gpiujVOgSaSdmEAg-of8FmD8ZbXNvxD4swa45s
all the options proposed so far are reasonable imo, they respect offline access (either always or after the first access) and achieve what is needed here
a 30MB dl is trivial on CI, where many GB typically gets pulled down in 10-20 seconds (e.g. restoring cache on travis), still should be fast enough to not appear broken on regular connections
what is possible to implement right now without major refactoring?
30MB … still should be fast enough to not appear broken on regular connections
Please try to remember that "regular connections" are not the same speed everywhere. 30MB is very large, and would take a significant amount of time or cost a significant amount of money in bandwidth in many places.
i live in australia, we have plenty of terrible internet connections here ;)
if downloading the docs is truly a dealbreaker in many places there are no options other than making it optional
we should not discuss anything else if we agree the bandwidth cost is unacceptable
that said, if 30mb is a significant cost i'm not sure how it would be possible to work with rust on a large project anyway given the size of all the crates that end up as dependencies...
I think @pravic's solution strikes the right balance:
I am fine with default (though I really doubt that those who really need docs can't install them as a component), but let's make them as opt-out, so rustup would remember during the next update that rust-docs doesn't need to be downloaded.
given the age of this issue, i am a little worried that we might "commit" to an ideal that requires a lot of yak shaving... then 12 months passes because nobody has the time/expertise to land what was "committed to", despite best intent
do we have a passable option that doesn't require heroic effort from anyone?
I was hoping to back able to ask @nrc tomorrow/Tuesday, at the Rustup WG meeting.
that sounds awesome
Installing them by default, but being able to fully opt out of their download completely sounds like the only really clean solution to me. Everyone who don't really care will get them by default and have access to them when they are offline, just like now. But CI and those who care about their bandwidth and time it takes to install on Windows can just append something like --minimal and save a lot of time.
I think optional, opt-out, components make sense even outside the scope of docs. By default rustup is somewhat "batteries included", but there is still a way to only get the bare minimum to compile Rust programs. Really useful for build servers or more constrained environments.
There's basically two separate but related features here:
- Allowing components to be optional yet default, with an opt out to never download them in the first place.
- Allowing components to be downloaded without actually being installed. Switch the
srcanddocscomponents to use this by default, since they take a long time to install on NTFS, but aren't necessary to get code building.
Either feature on their own is helpful, but having both of them will make everyone happy. People who don't want any bandwidth wasted at all can opt out, people who are suddenly without internet and realize they forgot to install the docs won't be screwed, and people installing Rust on Windows won't have to suffer through agonizing installation times.
Does anyone know, now that @nrc is leaving Mozilla, what will become of his implementation of this feature?
An initial, still WIP PR was sent last week intended to bring a solution to this issue: #1673.
maybe this got worse, or maybe travis got worse
now seeing intermittent build timeouts:
https://travis-ci.com/holochain/holochain-rust/jobs/184030443
maybe the docs have grown recently and we hit some threshold?
You might be seeing rust-lang/rust#58849. Workaround at rust-lang/rust#58849 (comment).
Ah sorry, the issue I just linked is about running the rustdoc tool but this thread and your comment is about installing the rust-docs component of a rustup toolchain. Never mind.
it seems to me that no one but 0.1% of the community needs the docs by default, but I just can't think of what to do next? shouldn't we create a poll or something somewhere?
Progress is being made, though slowly, on installation profiles to allow for this without the need for polling a subset of the community. All I can ask is that you remain patient.
@kinnison Any way the community could help? If there are smaller sub tasks, I don't mind trying my hand at it.
I am very looking forward to see an update on this issue.
"rustup docs" can be smart: if there is offline docs use it else go online.
The existing required "docs component" could be for online use.
A new optional "DocsOffline component" could be for offline docs.
Does it make sense?
Until there is no way to opt-out of docs, they should not be installed by default
error: component 'rust-docs' for target 'x86_64-pc-windows-msvc' is required for toolchain 'nightly-x86_64-pc-windows-msvc' and cannot be removed
This is especially making me happy
I'm fine with docs by default as long as there is a way to opt out for use cases where they are not needed.
we moved to azure + appveyor "enterprise" and built images containing rust already installed
it was the only way to avoid timeouts
it's not cheap tho
There is hope!
Is there a place where I can read up on profiles. I've only found this open pull without a description #1673
Currently there's not anything concrete to read about, sorry, no.
@softprops
Maybe this gives you a starting point?
let profiles = &[
("minimal", vec!["rustc", "cargo", "rust-std"]),
("default", vec!["rustc", "cargo", "rust-std", "rust-docs"]),
(
"complete",
vec!["rustc", "cargo", "rust-std", "rust-docs", "rust-src"],
),
];Is there a place where I can read up on profiles. I've only found this open pull without a description
You can see the initial design discussion at https://internals.rust-lang.org/t/pre-rfc-improve-access-to-rustup-components/8318
Thanks @ashthespy. I'll follow this issue's progress.
For context I'm interested in slimming down the docker builder image for aws lambda softprops/lambda-rust#1. I really just need to cargo build tool chain. The bundled docs artifacts aren't useful in this context and yet they aren't currently possible to opt out of. This minimal profile looks great!
@kinnison is this issue addressed by the minimal profile now?
This appears to do what I want and not include the docs!
curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain "${channel}"
info: downloading installer
info: profile set to 'minimal'
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2019-09-26, rust version 1.38.0 (625451e37 2019-09-23)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'
❤️
It is also possible again to remove the rust-docs component (independently of the chosen profile).
@vks yup, I was very happy to do
for tc in 1.0.0 1.2.0 1.4.0 1.5.0 1.12.1 1.14.0 1.15.1 1.16.0 1.17.0 1.18.0 1.19.0 1.20.0 1.21.0 1.22.1 1.23.0 1.24.1 1.25.0 1.26.2 1.27.2 1.28.0 1.29.2 1.30.1 1.31.0 1.31.1 1.32.0 1.34.0 1.35.0; do
rustup component remove --toolchain $tc rust-docs
done
Note that this doesn't work for very old releases (like 1.0.0), which predate the concept of components. However, I'll take the savings for the versions which do!
I believe this is now covered so I'm going to close the issue. Thanks all for your patience.
Leaving a trail for those who are looking to do this with the offline installer:
export RUST_VERSION=1.60.0 && \
export TMP_RUST_DIR=/tmp/rust && \
mkdir -p "${TMP_RUST_DIR}" && \
cd "${TMP_RUST_DIR}" && \
curl -sLf "https://static.rust-lang.org/dist/rust-${RUST_VERSION}-x86_64-unknown-linux-gnu.tar.gz" | \
tar xvzf - -C "${TMP_RUST_DIR}" --strip-components=1 --exclude=rust-docs && \
./install.sh --without=rust-docs && \
cd /tmp && rm -rf "${TMP_RUST_DIR}"