Use OS-native config, cache home directories
Earnestly opened this issue · 64 comments
For Linux-like systems, we should be using XDG. See arch's wiki for a non-exhaustive list of software with varying degrees of compliance.
For Windows, everything should go in ~/appdata/locallow or ~/appdata/local,since ~/.cargo is just a cache, AFAICT. This is FOLDERID_LocalAppData for SHGetKnownFolderPath, CSIDL_LOCAL_APPDATA for SHGetFolderPath, and %LOCALAPPDATA% in the environment.
Have been stumped by this as well. Looked for Cargo's config in ~/.config/cargo only to find nothing. Don't really understand why Cargo places its config in ~/.cargo and the environment variable
CARGO_HOME, while nice is entirely useless. If the user's XDG is not set then it would be fine to place in ~/.cargo.
@0X1A If the respective XDG_* variables are not set, they should fallback
to the appropriate defaults as defined by the specification. The unfortunate
legacy behaviour cargo needlessly introduced will likely just add the code
bloat.
Is someone working on this ? The more we wait, the more difficult it will be to change the default.
You can use my xdg crate to implement this.
cargo install now installs into ~/.cargo/bin.
... and it really should use ~/.local/bin; there's precedent already in the behavior pip install --user, at least.
@Stebalien Waiting for rust-lang/rfcs#1615.
It's the only thing I missed in rust ecosystem!
Maybe we can rebase the patch to keep it synced with current development.
Or to separate the patch into three platform so that it will be available on partial inclusion.
I'll rebase the patch or the partial patch if this gets accepted. This won't be the problem.
I don’t know where this came from, but on Linux I apparently have both
CARGO_HOMEbeing~/.cargo/bin/as well asCARGO_HOME2being~/.cargo/bin/bin/. 😕
@sanmai-NL Where does your CARGO_HOME2 come from? I only have CARGO_HOME, which I set to "$XDG_DATA_HOME"/cargo in my ~/.bashrc.
I have no idea, probably just some leftover from messing around when there was no Arch Linux package for Cargo yet. Note that my CARGO_HOME was also wrong, the bin path component should only be appended to CARGO_HOME within the PATH env var instead.
Another issue with not respecting XDG: backups. My current backup profiles knows that it should avoid ~/.cache/ directory. And cargo is creating a lot of things into ~/.cargo that is only cache data, not useful to backup…
cc #5183
and rust-lang/rfcs#1615
Any progress on this?
... and it really should use ~/.local/bin;
~/.local/bin is not specified by the XDG spec.
... and it really should use ~/.local/bin; there's precedent already in the behavior pip install --user, at least.
~/.local/bin is not specified by the XDG spec.
FWIW, there is XDG_BIN_HOME that sadly never got commited to the spec, lost due to bad maintainership. I'd still recommend to use this, because at least it has defined (and agreed on) semantics.
@piegamesde Not that it matters too much, as we'll probably just put things in ~/.local/bin if there's no spec, but I've opened an issue with XDG re: $XDG_BIN_HOME.
Another advantage of complying with XDG: when trying to get some space off of your disks you can often trash ~/.cache.
So I'm asking for cargo to put a project's target/ under ~/.cache/cargo/sha256(project path)/target.
This would replace tools such as https://github.com/mihai-dinculescu/cargo-wipe
If anyone would like to help, I'm going to try to have XDG_BIN_HOME or its equivalent merged: https://gitlab.freedesktop.org/xdg/xdg-specs/-/issues/14#note_682259
@christianbundy Lent my support as much as possible.
However, of note is this: one way or the other, they will be documenting ~/.local (with ~/.local/bin) as the official 'default' location. So at the very least, this is a reasonable default for rust.
I've made a PR with Lennart's original patchset related to this issue (which stalled and languished after all-around approval).
Ok! the PR was merged (issue, pr). The result is that the official location for user-specific executables is as follows:
There is a single base directory relative to which
user-specific executable files may be written.
User-specific executable files may be stored in
$HOME/.local/bin. Distributions should ensure
this directory shows up in the UNIX$PATH
environment variable, at an appropriate place.
Since
$HOME</literal> might be shared between systems of different achitectures, installing compiled binaries to$HOME/.local/bincould cause problems when used on systems of differing architectures. This is often not a problem, but the fact that$HOME` becomes partially achitecture-specific
if compiled binaries are placed in it should be kept in mind.
This has been merged, but has not yet been released. I have no clue what the XDG release schedule is.
Edit of course, this was anticipated, as it's the reasonable default, but they opted not to have an env-var for the location. I.e., it is architecture-specific, and if the user is using their homedir across architectures, they can just go figure it out themselves.
but they opted not to have an env-var for the location
I am mildy confused about realizing this only now, there were way too many similar patches in discussion – this is the first directory location whose path is hardcoded in the spec :/
Anyway, this really isn't a show stopper. Cargo shouldn't place any binaries directly into the .local/bin folder, but rather some script wrappers that gracefully fail when on a non-native architecture.
This has been merged, but has not yet been released.
Given the circumstances, it is fair to assume no breaking changes will happen to the spec between now and an eventual release. (A release would still be great though.)
@piegamesde I fully agree that it's a bad idea not to have an env var for this. I've added your point to the XDG conversation on closing that issue.
I don't even think that the XDG people were expecting dev tools to handle that -- I think it's more along the lines of "If the user/distro is going to try to support multiple architectures for a homedir, that should be detected and handled by links set in ~/.profile or equivalent". Asking every dev tool / local user software (once installed) to fail gracefully if the architecture changes is.. ..well.. ..too much. They'd never get multiple projects, systems, etc to go along with that, even though it might just be a matter of wrapping executables in a short script.
Given the circumstances, it is fair to assume no breaking changes will happen to the spec between now and an eventual release. (A release would still be great though.)
Agreed. At this point, even if they decide to use an env var, it would default to ~/.local/bin.
Six years.
Please add support for XDG...
I pinged the XDG spec in case they're willing to move on XDG_BIN_DIR (there has been some input from other projects in the mean time). I do think that supporting it if it exists without expecting it to exist might be a reasonable step. The only reason for anyone to create an XDG_BIN_DIR env var is for that specific purpose, anyways.
https://poignardazur.github.io/2023/05/23/platform-compliance-in-cargo/ for a summary of this
For backward compatibility it's about correct order of execution to guess placement and logic behind it.
I personally will be happy even if only config.toml could be placed to XDG_CONFIG folder first and only then slowly without rushing a thing other files and folders as per discussion. This is a relatively small change to make, and I guess that community will be happy to have at least this.
The placement of config.toml I propose is following (at the very end of the list):
- If CARGO_HOME is defined, use it
- If XDG_CONFIG is defined, use it
- If ~/.config/cargo/config.toml exists, use it
- If ~/.cargo/config.toml exists, use it.
If any tool uses hard coded path, then this tool must be fixed to use correct order. I don't see a problem with it.
If cargo would have a way from command line to show whole actual configuration, other tools can use this as well without a requirement to hardcode any path
FYI I've generalized this issue to include #1976 since we'll need to discuss and resolve them hand-in-hand.
I finally got annoyed enough about this that I decided to force the issue via my shell-profile creating a temporary CARGO_HOME linking out to the correct locations:
export CARGO_HOME="$XDG_RUNTIME_DIR/cargo-home"
if ! [ -e "$CARGO_HOME" ]
then
(
_link_cargo_dir() {
base="$1"
shift
while [[ $# -gt 0 ]]; do
mkdir -p "$CARGO_HOME/$(dirname "$1")" "$base/cargo/$(dirname "$1-")"
ln -st "$CARGO_HOME/$(dirname "$1")" "$base/cargo/$1"
shift
done
}
_link_cargo_dir ~/.config config.toml
_link_cargo_dir ~/.local/share credentials.toml
_link_cargo_dir ~/.cache git/db/ registry/{index,cache}/ target/
_link_cargo_dir "$XDG_RUNTIME_DIR" git/checkouts/ registry/src/
)
fi(Note, this doesn't cover every path inside CARGO_HOME as on nixos I'm avoiding using some like bin/, more just a starting point if you want to do something similar)
(EDIT: Updated some of the categorisation based on re-evaluation while writing a response to the below Pre-RFC)
@Nemo157 you'll likely quickly see but I went ahead and posted a "Phase 1" Pre-RFC that just allows splittig up ~/.cargo/home by defining env variables.
I would be interested in your choice of putting some things under .local/state vs .cache.
Asking so I can move my cargo folder in place before this gets implemented: is %LocalAppData%\Rust\cargo going to be the location on Windows? Or will it likely be %LocalAppData%\cargo? Or is that still up in the air?
I am not willing to commit to an answer on that as we still need to finish up the Pre-RFC for configurable paths which comes before we try to finalize the default paths should be.
Sorry, this is going to be a rant. I came across this, not because I'm a rust user (developer) - that's still on my bucket list - but because I just installed a program that uses rust, on a system that builds some of the programs you install. I came here to find out how to convince "rust" or actually "cargo" to please not mess up my home directory.
This ticket was started 9 years ago. And the last comment is "I am not willing to commit to an answer as we still need to finish up the Pre-RFC for configurable paths". IS THIS REAL? (I'm not upset about what @epage said, that's reasonable, I'm upset that rust/cargo obviously believes that they have the sovereignty over everybody's home directories. It's up to cargo to decide what is written to /home/me, even if cargo was only used as a build tool to install software (inside a container use to isolate it from the real world).
This is not the case. My home directory belongs to me. Whenever I use software, I delegate access to my home directory. On Linux systems, there are XDG directory settings and if they are set, I can and do expect them to be respected. If people want software put stuff in ~/.<your_great_software>, then they can set the according XDG variables to that effect.
It is a BUG, that cargo ignores these and then does not ask for permission to write files anywhere but in places where software is authorized by platform standards or user consent.
This is the same problem, that is pervading the whole relation between users and producers of software. There used to be a trust relationship. I can install stuff because "they will behave". When I install a VSCode or Docker package, they (did or still do) establish their signing key as authorative for all repo's packages. I don't want that. So now I can't use apt-key anymore and I'm just short of extracting every .deb file to read the installation scripts and verify that they don't install trojans on my system. I don't want that either.
Writing in ~/.cargo is by far not as bad, but it's the same violation of trust. There is an implicit contract between us, especially for software that has such a central and pervasive role as cargo (f.e. when using paru/yay on arch to install rust AUR's - sure people should install stuff manually, but hey, life is short). I don't use rust. I don't want stuff, especially executables in '~/.cargo' - that's not where user-binaries live. I'm not your user. I don't get any benefits from you storing stuff on my computer. Not necessarily. In this case I just wanted to try some rust software that I ran in a distrobox container, so that I don't have to clean up after the fact. And still, now there is a registry with cached stuff that will be obsolete in a week from now in my home directory. It's also on my NAS in a restic repo. And it's on the backup USB disk.
How much disk space is wasted around the globe for stuff nobody ever wants or needs?
On the other hand, what is your benefit for ignoring XDG on Linux, and the equivalent standards on Windows and Mac?
"You do it the UNIX way" - No you don't. The UNIX way was at a time when there was a defined operating systems with releases that changed once a year. People knew the software they used. If they installed new software, they had to download the sources and manually configure them. If someone then wanted to change the path to '~/.<your_stuff>`, they'd edit the source. That was 1980, 1990 and maybe still 2000. If you think you're doing it the UNIX way, get a PDP-11 emulator and spend a week with V7 and come back again.
Actually, you just didn't want to think about a rather complicated problem, categorizing data into caches, config, output, platform specific, etc. I get it, I have to think about that each and every time I start a new project. By now I have a good grip on it, I'm doing it for long enough.
But hey, you're making a rust compiler. That's about one million times more complex. How much time did you safe by vomiting everything into '~/.cargo'? A day? A week?
I just got a new computer and setting it up I wanted to take that as an opportunity to start from scratch with my setup and finally take back control over my data.
I'm thinking of jailing each piece of software I'm not using inteimately into a container (currently looking at tool-/distrobox for that purpuse), where each one get's their own home directory (their home) where I bind-mount just the things these apps or commands are allowed to see and whatever they write into that home gets thrown away every time the container stops. Not as a security measure (not good enough for that), just to get the kids fingers off the cookie jar.
Getting that right is a huge effort and I have to check each tool and look at what kind of mess its creating.
I hate to have to do that. It's boring and annoying. But it seems to be the only way to protect myself from people writing software and ignoring legitimate complaints for 9 years - in words - NINE YEARS! What else do you do that I might not know? Are you mining crypto currency? Well I guess not. But that's the thing, I really miss the time when there was mutual respect and trust. Today being a user feels like the developer expects to hear "a flower pot" when asked what's on the screen.
What makes the whole thing worse is that behind this problem, deep down, there is a really ugly truth. Developers do not respect their users. This is as true for big tech companies who see their users as live stock to farm big data or money, as it is for OSS developers who for some reason far beyond me copy that attitude. Developers see their users as free loaders. They rarely pay for what they get. So there are no contracts, no obligations for no compensation right?
Developers do (in most cases) not develop software for their users. They do it for a salary if employed. Because they're told to. Because they want to become famous in the geek world they live in. Or they need a reference to get a consulting or speaking gig. Most developers don't give a shit for their users. How else can you explain 9 years? And this is the least impactful and least important issue here. It's just messing up the home directories, what's the big deal?
The deal is, that users actually contribute an extremely important part. Without them, your project would simply fail. We users are the live stock to be milked, one way or another, or at least we are the medium in which fame and fortune is build. And all we ask for is a little respect.
And now talking as a developer, which I also am, I do deeply care for users of my software. I take the time to carefully consider every line of code I write. I don't do that because I'm a better person than you, I do it because I enjoy the process. It's one of the reasons why I am still hacking after decades of working in the field. And still I make mistakes. But hell I don't ignore my users for 9 years. That's just rough.
I can understand this is frustrating but ranting is not helpful to making change happen and accusations are not appropriate.
On mobile but from what I remember of the beginning of the thread, this was more of "is this a settled convention?" and they went with the most standard, familiar convention.
As for why it hasn't been fixed yet is compatibility makes it hard (without even getting Mac users to agree on a convention) and developer availability. The team was small enough for a while that we even had a soft feature freeze. So why wasn't this fixed outside of the feature freeze? Each contributor comes with their own priorities in open source. For those putting in the work, it hasn't been enough of a priority to see it to the end to overcome all the challenges. For example, which is more impactful, improving index download times for every user or paths? I hardly notice the paths but always notice download times.
To add, I'm paid full time to contribute to open source and my priority list is mostly my own. If you look over my history, it is user focused. I would say the rest of the team is the user focused as well. This might not always be obvious due to available time and the focus is on long term. You are welcome to come to office hours (link in wiki) where we frequently talk over designs when a contributor doesn't need our assistance.
Calling the fact that cargo writes to ~/.cargo a "violation of trust" seems to be a HUGE overreaction to me and later parts of the rant accuse Cargo's devs of not caring about users.
Just to voice an alternative opinion, I think Cargo is one of the best package managers in the history of computing (so far), and not only has raised the bar for other language package managers, but has also attracted hordes of fans and migrants from other languages. Cargo has done this MOSTLY by being EXTREMELY user friendly!!! It's fantastic!!
I don't have strong opinions on writing to ~/.cargo (and I suspect most people don't), but I wanted to take the time to comment how much I appreciate the obvious effort put into Cargo's usability
I do have strong opinions about the creation and writing to ~/.cargo and I do want Cargo to follow the XDG spec and put their binaries in ~/.local/bin (and I'd honestly like them to respect XDG_BIN_HOME even though it's nonstandard).
BUT I also think it's not at all a violation of trust. If anything, it's a need for more Rust developers that care about the issue. Edit: I meant that to come across as "A need for more rust developers, and if there are, there would be time and effort available to do this." It was intended as a call to help out, not as a criticism of existing developers.
If you put your boundaries out too far, no matter how much other people try to do to accommodate you, you'll always get violated. Reign in your identity a bit -- you're not your homedir.
I still agree the practice sucks, though, and should be changed. If anything, you can sum the rant up as "Hrm, looks like some people are really frustrated about this issue." I mean, if put that way, I fall into that category.
I highly agree with eode. It causes frustration to me that .cargo litters my home directory and I really would like to have that fixed sooner rather than later. I also would not want to call it a breach of trust as overinflating its severity undermines your argument.
Granted this rant did exactly what it intended: Brought attention to the issue.
In case someone isn't aware, you can set RUSTUP_HOME & CARGO_HOME.
@mutech Thank you for your heartfelt comment. It's kinda sad that most of the responses you got are just tone-policing you instead of taking your frustration seriously.
As a person who also tried to fix this back in 2018 (feel free to expand the collapsed comments above and you'll see the links) and then decided to spend time in places I'd feel less unwelcome, I feel you.
Granted this rant did exactly what it intended: Brought attention to the issue.
There is a saying "You catch more flies with honey than you do with vinegar". Whenever there is a rant or a "bump" post, it sours me on the issue and I don't want to touch it. Some of it is my emotions speaking and I try to recognize that and set it aside. Part of it is not wanting to reward unhealthy behavior (see https://siderea.dreamwidth.org/1209794.html). Part of it is the concern that its a sympton for how unhealthy future dialogues on that issue will be which won't be healthy for maintainers and can lead to burn out.
So yes, there is "'bad press" and this kind of behavior is not an acceptable way of moving forward. I will get back to the Pre-RFC when I do which is after I wrap up some of the existing work I have committed to.
But then there's the fact that the more you ignore it, the less healthy the future discussions will become because more and more will get frustrated with it. That starts developing ideas that the project never fixes anything. The more something gets attention, good or bad, the more you should put effort to it in proportion to the work required. Try not to act like a mom to the people that complain.
The more something gets attention, good or bad, the more you should put effort to it in proportion to the work required.
My life improved considerably when I stopped listening to what other people say I should spend my time on. I wish the same to every Rust and Cargo maintainer 🥳
In case someone isn't aware, you can set RUSTUP_HOME & CARGO_HOME.
It's good to have that pointed out here, because it allows someone to work around the issue, but the default location should still be in the relevant os-specific locations.
@epage It's good to remember you don't owe it to any of us. Thanks for your time developing Rust -- hopefully I'll get a little more familiar with Rust, and I can contribute myself sometime. In the mean time, I have a lot of respect for your contributions, and I don't really feel that work is owed to any of us.
It would be much appreciated if people can take look at and share their user stories in the pre-RFC. That's one way out to address this long-standing issue.
Good to know!
@weihanglo thank you for a link
But then there's the fact that the more you ignore it, the less healthy the future discussions will become because more and more will get frustrated with it. That starts developing ideas that the project never fixes anything.
Yes. There is a balancing act. Part of it involves engaging more with the people who respond in a healthy way, Part of it is setting aside those feelings and prioritizing it independent of the negative voice.
The more something gets attention, good or bad, the more you should put effort to it in proportion to the work required. Try not to act like a mom to the people that complain.
This isn't about acting like a mom. This is about trying to make sure there remains a cargo team to respond to these posts, implement features, and review the work of contributors. If we reward abuse, only abusers will be left. If we don't set boundaries, then we will get burnt out from the level of abuse we receive.
The loudest voices are also not necessarily representative of users so it'd be a poor choice to make decisions that way. It is unfortunate in open source that we don't have a good way to keep a pulse on users like at a business where you have customer portfolio to tailor a roadmap to. It can make it hard to know what is important. Even besides that, we are working with volunteers who need to do work that excites them enough that they keep contributing (see again about burn out).
In case someone isn't aware, you can set RUSTUP_HOME & CARGO_HOME.
This is not the same. XDG defines configuration in $HOME/.config/ and cache in $HOME/.cache (and other directories). As far as I know, you can't do that with CARGO_HOME.
One long post that questions how seriously we take an issue if it hasn't been solved in 9 years is not abuse, even if it uses an occasional capitalized word.
That was about the general concept, not the initial post and finding the right words can be hard.
In case someone isn't aware, you can set RUSTUP_HOME & CARGO_HOME.
I'd say that mostly does not fulfill the need of people coming to this issue, who usually want to split managing caches, configs and data. It's also stated in the previous postponed RFC rust-lang/rfcs#1615.
- Putting caches in designated cache directories allows backup tools to ignore
them. (Linux, Windows, macOS. Example: Time Machine ignores the cache
directory on macOS.)- It makes it easier for users to manage, share and version-control their
configuraton files, as configuration files from different applications end up
As a NixOS user who manages immutable cargo config with Nix, I have to do dirty symlink tricks to split $CARGO_HOME/{registry,git,config.toml} to different targets, so that cargo will not try to write to ro-mounted /nix/store and fail. That's all no issue if the RFC is actually implemented since we can set XDG_*_PATH separately.
In this time, I strongly agree this comment from previous RFC
The core problem is that those who are interested in fixing this don't make the decisions, and those making decisions aren't interested.
This issue is bothering a lot for us as cargo users, who may or may not have cargo development skills, while many (maybe only little?) cargo developer more lean towards keeping the current non-ergonomic situation with the reason of "compatibility"1 or "always has been". So the only way left for us users to do is ranting out and gaining attention to show that the current situation is not as good as they think.
Footnotes
-
In one of several relevant issues (sorry I cannot find it immediately), I saw someone treat cargo-audit storing their database inside
$CARGO_HOMEas compatibility concerns. That's blowing my mind that a "security tool" is writing random private stuff into cargo dedicated path without consent, which is taken as a positive example. I'd treat that operation itself as security issue, and hope they won't modify$CARGO_HOME/config.tomlnext time. Thankfully I'm using Nix so the config file is ro-mounted. ↩
I'm going to temporarily lock this thread to provide a cool down period.
I'm unlocking this and hiding the most of the recent posts.
I would ask that people be respectful to others and that includes not assigning intent to others I think a general assumption we can make is that we all want what is best for the users of cargo.
A summary of the conversation from some of the hidden posts:
- A use case raised is "non-rust users", whether as a side effect of system or other language package managers building from source. As Rust is adopted in more places, this will grow to become a bigger problem. This affects more than this issue but also issues like #12633.
- Concern was raised over the wider ecosystem drag Cargo is adding to the effort for adopting XDG in the wider ecosystem
CARGO_HOMEcan be used to move~/.cargobut this won't get the full benefit if you have config in that location or depending on how strict you are about the different paths- You could symlink the different locations. This could likely be improved (cargo auto-detecting certain symlink targets don't exist and creating them).
I think, that I know a hacky solution for locating build directory (target) in the XDG-compliant way.
target is actually a default value for $CARGO_TARGET_DIR, so you can add:
alias cargo = 'CARGO_TARGET_DIR=$XDG_CACHE_DIR/rust-builds/$(cargo metadata --no-deps --offline | extract_name_magic) cargo'Where extract_name_magic is a combination of grep and sed which will extract the package name.
By using this alias, you'll get all the cargo commands working as expected with separate XDG-compliant build directories for each project.
Tomorrow I'll implement extract_name_magic for completeness.
FYI this is about everything in the home directory today. There is a separate issue (and an open RFC!) about centralizing target dirs. On mobile or I'd get you the linksd
FYI this is about everything in the home directory today. There is a separate issue (and an open RFC!) about centralizing target dirs. On mobile or I'd get you the linksd
See rust-lang/rfcs#3371 and #11156 (if I got it correctly).
In one of several relevant issues (sorry I cannot find it immediately), I saw someone treat cargo-audit storing their database inside $CARGO_HOME as compatibility concerns. That's blowing my mind that a "security tool" is writing random private stuff into cargo dedicated path without consent, which is taken as a positive example. I'd treat that operation itself as security issue, and hope they won't modify $CARGO_HOME/config.toml next time. Thankfully I'm using Nix so the config file is ro-mounted. ↩
I believe this was from the Pre RFC thread https://internals.rust-lang.org/t/pre-rfc-split-cargo-home/19747/26