Support package signing and verification
ryankurte opened this issue · 33 comments
It should be pretty easy to have a basic flow so CI created images can be signed, and verified on pull. Not sure what algorithm(s) / tools we should use? Accessibility is key, both IRL and in CI, and it needs to all work well cross-platform.
- Add public key (and key type?) to metadata (let's b64 it or something)
- Search for a signature file corresponding to each package (
.sigor appropriate to the format) - Generate checksum / validate signature against public key on pull
signify could be one way to support that.
I have a Rust implementation of it. Though it's not been touched since 2018 it might be not too much work to update it & expose parts of it as a library so that binstall could use it.
The Cargo.toml would then have the public key listed and .sig` file or similar contains the signature to verify.
So there are solutions like The Update Framework or Docker Content Trust or here on GitHub gpg keys. They are all kinda similar, they both have a client building and signing, a server holding the data and one the meta data. They are both not bound to a specific hoster, gpg keys are not only used on GitHub but for many git server distributions and docker has their protocol of image naming and the oci which can be implemented by any source. (Theres is the method for tls certs, a kind of trust chain but that’s not an option as currently there aren’t thousands of chainable entities, it would be the publisher of a crate and this tool/repo)
So how do I see this as an option for cargo-binstall (Not a maintainer of developer of this projects, so just my 2 cents)
Pre-assumtions
- I do not assume crates.io to be the source of metadata
- I do assume that the source of metadata has authentication in some kind, so we know that there is a limited amount of people publishing crates under a certain crate name
- I do assume that the metadata source is accessed via https, a secure transport, so we can at least trust the network
- I do assume that the source of the binary will not be limited to github
- I do assume that the source of the binary will be accessed via https, so we can trust the network
- I do not assume that you are on a laptop with third party certificates or any of the sources serve self signed certificates (that’s another rabbit hole to go down)
How do I imagine this to work?
For this kind of flow you would have 4 parties:
- crate/metadata host (known as “the meta-host”)-> in the case I make comparable to docker notary server or GitHub’s gpg store
- Binary host (known as “the binary host”)-> comparable to docker hub or github repositories
- The client signing (known as “developer”)-> can be a dev or a ci job
- The client downloading the binary (known as “user”)-> the actual user of
cargo-binstall
In the end we would have two flows: publishing and installing.
How would publishing flow look like:
- the developer builds the binary
- the developer signs the binary with his private key (preferably non detached) -> preferably disposable as we know he is authenticated to the meta host and we do not have a central authority guarding and resigning keys anyway
- the developer attaches the public key to the
Cargo.toml - the developer publishes both binary and meta source
Points I’m not sure about:
- Why should the keys be disposable?:
- Is assume that the authentication with the meta host works properly and the developer know how to choose and maintain his password
- I assume we trust the meta host, as we already gather all the meta data from it, it also contains the download url (format) to use, account hijacking would be fatal but that’s also the case for dockerhub or github where we just hope the hosts have MFA
- This should not be a part of the tool but rather a best practise, if one would need to verify the source by its public key, he should be able to do so, but as mentioned, releases aren’t a central management tool of identities
- (Pro) if one would really try to crack a key, the next version would use a different one
- (Pro) it is basically one shot, fire and forget, no key to steal if it isn’t saved
- (Con) One could not verify if a new release was really published by the source which released previous versions
- If one would like to be as tech savage and be able to let users verify the source, he could use sub signing keys or an authority scheme, where release keys would be signed by his keys, which in fact can be checked by the public key in the meta host and his identity/authority public key (also another rabbit hole to go down)
- Disposable keys would follow (somehow, if released iteratively) the best precise of key rotation
- Disposable keys would not compromise the security of the flow as both parties, the meta host and binary host, need to match (assuming we use tls for communication and trust the hosts)
How would a user flow look like:
- initiate the install via
cargo binstall - Fetch meta data from the meta host
- Check if the user trusts the server, only once needed (kinda like ssh fingerprints)
- Download the binary
- decrypt the binary using the public key from the meta host
- If decrypted correctly without errors, install the binary
- got errors during the process, let the user know and abort the installation
Things I’m not sure about the users flow:
- Should it be allowed to ignore faulty keys? No, that’s what this tries to mitigate
- So what if one just didn’t sign it? Let the user decide, preferably abort but that would be a breaking change, the tool trusts the signing process so we know its safe to install (that doesn’t mean if the binary itself running is safe at all, just the installation part)
- Should we really use the tls fingerprinting? Public certificates change all the time, but its a simple an effective way of telling the user “hey, something about the host changed”, if something like doh/dot is enabled we would only need to trust that dns source.
What general limitations should be given:
- the tool should not allow weak key types
- if there is some faulty key or something faulty in the process, let the user know but keep the actions of the tool on the safe side, no matter user given inputs
- If possible, users should be given a warning if their source uses certificates not available to the public but installed on the system
- The tool should follow the same process and try to notify the user before an install if there is newer version available
Now I know I’m not a developer of this tool, but its a lovely one and this is just tickling the tech security part of my brain and one of the pain points of introducing this tool to other techies I got to know. And again, just my 2 cents (or at least some paragraphs of it 😅)
If someone is keen to have a discussion for effective measures or plans, I’m happy to help. In general, also happy to hear feedback and let some shoot bullets through my ideas (a german saying, hope it translates somewhat into english)
🦀
Some resources about encryption/decryption and secure software updates:
Thanks for the detailed exploration! I've reached for similar designs while thinking about this. However, I think:
- binstall is too small to position as authority
- binstall is too little staffed to implement and maintain something of this complexity
Hence, my position would be that we need to have some third organisation to create and maintain such a design. Fortunately, there are things happening here!
- I've been keeping an eye on cosign, which is working on what I believe is a good solution in this space, though focused on the Go and Container ecosystems at the moment
- the Rust WG for CLI may be moving on the topic, with at least one member interacting with the community on the subject (I haven't looked into that more yet)
Furthermore, for true adoption, my personal belief is that:
- developer tooling should be made ready and plug-and-play: at minimum, a gh action that takes care of the signing and sig publishing and temporary key generation and key publishing against an identity and authentication against the identity provider
- consumer tooling should be made available in the form of libraries (crates) that make easy the essential actions of: fetching metadata, keys, signatures; verifying identify; verifying files
At which point, yes, binstall involvement becomes feasible.
So I just went full on rabbit hole style on this.
My conclusion is, the only potection given is that there was no tampering with the binary since the release.
Again assuming:
crates.iowill not be the only place of meta source- github is the only place of binary storage
- there is no central place of providing correltion between crates, their owners public key and its validity
But if thats the option to go with, I will gladly have a shot at this and open a PR after my final exams (so in a month or so)
Just started to have a go at it and something just struck my brain.
A public key can't be modied if the meta host such as crates.io doesn't let you overwrite the crate. But if I were to use the same key very time, binaries and sig files could be changed, what this verification would be here to prevent.
Well how does cargo handle something like this? the crates index contains a checksum which can be used to verify the file received from crates.io. It could work similar to this, the Cargo.toml could have an array of checksums (as every file would have one) and after downloading the file the checksum could be calculated and compared to the ones in the metadata
this would mean cargo-binstall is as secure as cargo install
A public key can't be modied if the meta host such as crates.io doesn't let you overwrite the crate. But if I were to use the same key very time, binaries and sig files could be changed, what this verification would be here to prevent.
the basic concept i was thinking about was less about tampering more how to tie a given binary from actions or S3 or whatever third party source was used for distribution to a given crate release, so you could be reasonably sure that the binary you have pulled was the one that was created in CI not something else uploaded elsewhere. obviously this still depends on the security of the keys in the CI environment, but i think it still an improvement.
while detecting re-publishing or changes is certainly interesting (as is trustworthy / verifiable CI) i think that has to fall outside of the scope of this project (at least until github provides a secure key store and trusted binary signing API 😂).
So then verifying a tls cert and enforce https would be enought to trust the source of the binaries?
Sigstore has reached v1.0.0 and it provides a rust crate sigstore-rs for verifying the signature.
I don't know if this bit would be an issue...
The crate does not handle verification of attestations yet
but hopefully that's implemented when we get to this
For nextest-rs/nextest#369, I had a look at https://docs.rs/sigstore/latest/sigstore/ and seems like it should be possible to:
- create a signature bundle at release time using
cosign sign-blob: https://docs.sigstore.dev/cosign/signing_with_blobs, including in GitHub Actions - upload the signature bundle to GitHub Releases
- verify the bundle as in https://github.com/sigstore/sigstore-rs/blob/main/examples/cosign/verify-bundle/main.rs
Now, this just verifies that the bundle uploaded also matches the artifact uploaded. I think this would be a really useful initial step.
To make this better:
- We also need to verify the identity of the signer (which should probably be stored in
Cargo.toml? This way crates.io becomes the root of trust). It's not totally clear to me how to do that from the current sigstore API. sigstore/sigstore-rs#274 (comment) mentions this, I believe:I think the verify_blob* methods could use a variant that supports Fulcio based certificates + verification constraints.
- We also need to ensure that the artifact and bundle haven't changed since the initial release, in case a malicious entity takes over control of the GitHub repo and starts clobbering old release artifacts. This might need to be done using timestamps, possibly with https://docs.rs/sigstore/0.7/sigstore/cosign/bundle/struct.Bundle.html#structfield.signed_entry_timestamp. It's possible using OCI to store artifacts rather than GitHub Releases might also help.
I'm not a security expert and I'm almost certainly missing something. It would also be good to maybe open an issue with sigstore-rs people discussing this.
More thoughts.
- I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?
- Timestamps aren't great as a way to verify authenticity, ideally we'd use hashes. But assuming 1 is true, there's no reliable place to store hashes on GitHub Releases.
- Does uploading either the artifact or the signature bundle to OCI solve the "malicious user takes over GitHub Actions and changes old artifacts" threat model? I don't know if OCI artifacts on ghcr or other registries can be changed. It's worth verifying this.
- It would also be really cool to have the binstall verification algorithm be its own non-copyleft-licensed crate, so non-binstall users can use the same algorithm.
I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?
Does uploading either the artifact or the signature bundle to OCI solve the "malicious user takes over GitHub Actions and changes old artifacts" threat model?
hmm, you can delete and re-upload artifacts, but so long as the signature is valid is it important that these are immutable? i think the risks one is attempting to mitigate are worth some though here.
- for folks worried about an exact binary they may need to pin a signature or a hash for a given platform and version
- most users will fetch new version / binaries anyways so long as they're correctly signed so is a new version meaningfully different from an updated binary with the same version? (eg. compromised actions / keys could be used to publish a patch release which any non-exact version filter would update to anyway)
slightly aside one of the things i've been thinking about a bit is how annoyingly repetitive setting up gh actions CI is for rust tools... maybe it'd be worth us investigating putting together a workflow template with variables for the usual stuff (rust version, platforms, platform packages, cross) that could include packaging / signing / publishing with whatever mechanisms we do use.
slightly aside one of the things i've been thinking about a bit is how annoyingly repetitive setting up gh actions CI is for rust tools
A couple of existing efforts:
- https://github.com/taiki-e/upload-rust-binary-action which ideally would follow whatever pattern binstall supports (I know @NobodyXu works on both the projects). I use this for nextest, it's great: https://github.com/nextest-rs/nextest/blob/1cd93240e022f28968bffbbec4e6ab7a9a585dee/.github/workflows/release.yml#L120-L313
- https://github.com/axodotdev/cargo-dist
hmm, you can delete and re-upload artifacts, but so long as the signature is valid is it important that these are immutable? i think the risks one is attempting to mitigate are worth some though here.
for folks worried about an exact binary they may need to pin a signature or a hash for a given platform and version
Yes, security-conscious users would like to pin the version and platform and have it be guaranteed to always resolve to the same artifacts. (modulo cargo-binstall itself being compromised, but they'll likely want to pin that to an exact version too)
most users will fetch new version / binaries anyways so long as they're correctly signed so is a new version meaningfully different from an updated binary with the same version? (eg. compromised actions / keys could be used to publish a patch release which any non-exact version filter would update to anyway)
Yes. I think publishing a new version that is bad is materially different from a malicious actor surreptitiously updating an old binary. (crates.io has the same philosophy, right?)
Ah sorry, NobodyXu doesn't work on upload-rust-binary-action. But it's part of the same general family of actions as https://github.com/taiki-e/install-action which they do work on, haha :)
I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?
@sunshowers We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.
If the registry is hacked, then no matter how secure your GitHub release is, it won't matter since the attacker can change Cargo.toml to point to whatever release they like.
Also, we are working on checksum support for registry #1183 , it will provide guarantees on security (presumably because crates.io index and the crates.io storage can be provided by two different sets of servers).
We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.
Interesting idea -- how would you do that? I guess I imagined modifying Cargo.toml as part of the release process was generally off limits.
We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.
Interesting idea -- how would you do that? I guess I imagined modifying
Cargo.tomlas part of the release process was generally off limits.
You can put a public key inside package.binstall under Cargo.toml, then use that private key to sign your packages released.
Once the Cargo.toml is uploaded to crates.io, it stays there and is immutable.
You could still setup automatic upload to crates.io on GHA using the recent crates.io scoped token (which we are already using) and it can be revoked if your GitHub get hacked somehow.
Gotcha! So, hmm, I think just storing the public key doesn't quite solve the threat model that I outlined, because: assume that the private key is stored as an environment secret on GHA. A malicious actor can:
- gain access to the repository
- read the private key from GHA
- compromise old binaries
- re-sign them with the private key
In other words, we don't just need the public key to be stored in immutable storage, we also need some sort of identifier for the binary. This could be just as simple as a hash, or a certificate, or something that just makes sure that once a binary is published it never changes.
Also, users will have to manage their own keys, which is something experts can likely do but new users could have trouble with. (In this case, to perform a release via GHA, you'd have to store the private key as a secret, then carefully destroy the key material on the local machine.)
While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.
read the private key from GHA
If you store that in the secret, then even the admin cannot read it, it can only be accessed inside GitHub Action.
Although they can try to reveal it by changing the GitHub Action.
In other words, we don't just need the public key to be stored in immutable storage, we also need some sort of identifier for the binary. This could be just as simple as a hash, or a certificate, or something that just makes sure that once a binary is published it never changes.
Hmmm, perhaps we can store a root certificate in Cargo.toml, then derive a public key based on hash of the crates tarball uploaded to crates.io?
(Of course, if you can generate a new hash/public-key and modify Cargo.toml in each release, it will definitely guarantee security.)
While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.
Thanks, I will read it later.
I think the hash of the crate tarball can definitely be used when verifying the pre-built binaries, after all, the binary is built from the crate tarball.
While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.
I skim through it, honestly I don't think it will solve the "someone replace the release artifacts" given that it still requires the developer to provide an identity token either in CI if automated, or locally.
Perhaps I misunderstood and it does have mechanism to solve it, but IMHO the most secure way is still to provide a checksum inside Cargo.toml and update it on every release, given that crates.io is immutable and always trusted.
Also cc @taiki-e since I would also love to hear feedback from you.
We can do away with user key management via the same process sigstore works. That is, in CI:
- Create a key pair
- Optionally add a certificate to sigstore that attests identity via OIDC
- Sigstore-sign all artifacts with that keypair
- Add the public key to Cargo.toml
- Throw away the keypair
- Publish the crate
As crate publishes are immutable, new artifacts can't be uploaded for that version, and as the keypair only exists in the context of that one CI job, it can't be stolen in the future.
Verification is either:
- SHA256-hash a given artifact
- Look it up in sigstore
- List out all crate-metadata attestations attached to the artifact
- Verify one of these matches the public key from crate metadata
or:
- Download associated sigstore certificate/signature distributed alongside artifacts
- Verify it was signed by the public key from crate metadata
- Verify it matches the artifact
FYI, @sunshowers @passcod @ryankurte rust actually has a closed RFC, I think it's worth time reading.
If the RFC is revived, then we definitely would like to follow suit here.
I think that's only for crates, though? not for binary artifacts
Oops, they propose adding sigstore to crates.io, so I thought it could be also used for verifying binaries.
Alright, how about this:
[metadata.binstall.signing]
algorithm = "minisign"
pubkey = "RWT+oj++Y0app3N4K+PLSYTKhtXimltIHxhoFgyWjxR/ZElCG0lDBDl5"
file = "{ url }.minisig"We add support for this optional section to the binstall metadata. algorithm and pubkey are mandatory, file is optional and defaults to { url }.sig (where url is the url we're downloading, and we include all the other fields too just in case you want to do something freaky).
algorithm can initially only be "minisign" because that's pretty popular, self-contained, doesn't have a thousand options, has good rust support, and keys are small. Later we can add GPG and Cosign and whatever else.
pubkey is the string representation of the public key in whatever format is native to the algorithm, in this case base64. Later we can add things like:
pubkey = { file = "signing.pub" }
pubkey = { url = "https://someserver.online/signing.pub" }to support loading from elsewhere maybe.
Then binstall would, if the section is there:
- Check the
filetemplate for syntax (to fail early if it's broken) - Do package resolution as normal, but don't download the file yet
- Render that
filetemplate - Download that file as a signature. I think we can just keep it in memory, for minisign it will be less than 400 bytes and for GPG it should be at most single digit kilobytes.
- Download the actual file, with the checksum thing @NobodyXu added recently enabled and configured for the algorithm
- Verify the signature
- Proceed with installation.
We should add:
- a
--only-signedflag to refuse to install non-signed packages - a
--skip-signaturesflag to disable this entire thing (e.g. if a packager messes up their publish and someone really wants to download things anyway)
That leaves the modalities of signing to the packager. As we've discussed, there's two main approaches:
- Persistent keypair, so the section above would be more or less static and the packager is responsible for protecting their key. This approach supports
--gittoo. - Ephemeral keypair, so a new key is generated before publish and its pubkey is added to the Cargo.toml just in time before publish, then the private key deleted when done with. This approach doesn't support installing via
--gitunless the publish process commits the Cargo.toml with the signature.
That will at least introduce package signing to the ecosystem; with this initial approach implemented we'll then be able to get more feedback from both packagers, users, and other community members. Because we'll namespace under metadata.binstall, we won't step on any feet, and then hopefully in the long term a common solution will emerge (with cargo-dist, crates.io, or whomever).
How does that sound?
It sounds great and aligns with how I would like this to be done.
I've started work on a PR to do this!
Alright y'all, we've shipped the first version of binstall with signing support!
Find more info in the release notes and documentation.
cc @sunshowers @ryankurte @somehowchris @taiki-e
We will also open to new signing algorithms and PRs for improving our signing mechanism!