Failed to reproduce 9.10.0 bitcoin-only
Closed this issue · 13 comments
After having reproduced 9.6, 9.7, 9.8 and 9.9 with my setup for WalletScrutiny, I now fail to reproduce 9.10.0 with a huge diff.
The build instructions had not changed. What else might I be missing?
- My script
- I run it with
scripts/test/hardware/bitBox2.sh 9.10.0
I can reproduce the issue, but it is somewhat expected, though with this release we were a bit unlucky that it happened this fast after the release. The problem has been documented here from the beginning:
The firmware binaries are only practically reproducible for a limited amount of time after the release. In the future, the dependencies of our Dockerfile might have changed, or Docker itself might even become incompatible or obsolete. This historic record allows you to be reasonably sure that the released binary was created correctly, even if it becomes infeasible to re-build the binary.
This happened now. The build.sh rebuilds the docker image and something in the upstream deps has changed since we did the release over a month ago breaking reproducibility. @cstenglein managed to reproduce the firmware back then (see https://github.com/digitalbitbox/bitbox02-firmware/pull/899/files).
If in the build.sh script you remove the line that rebuilds the Docker image and instead use the one we built and used over a month ago (docker pull shiftcrypto/firmware_v2:22) and build the firmware with that image, the firmware release becomes reproducible again.
I see three ways to mitigate this issue now:
- figure out exactly which upstream dependency in the Dockerfile changed and for what reason, fix and document that.
- use the
docker pull shiftcrypto/firmware_v2:22image as described above - make a new release v9.10.1 which should be reproducible again for the time being
I think 1. is out of the question as it could take days to nail down exactly what the change was and document it to an extent where anyone can reproduce the firmware again.
As a side note: the above is the reason that releases should be verified as early as possible. Normally it remains reproducible for many months, but that is not guaranteed. This is also why we'd love to have more community signatures of the builds (see here).
So it is as of now not reproducible?
- Figuring out in which state the system was a month back is probably feasible as you still have your image from "2."
- Trusting the much bigger binary
shiftcrypto/firmware_v2:22from the same provider as the binary we want to test defeats the purpose of reproducing builds - We'll go with that: File this one as not reproducible for us but hopefully next time we will be able to reproduce
As WalletScrutiny sadly does not have many very technical contributors, I can't promise to be faster next time but feel free to ping me via an issue in our GitLab. Maybe, with some assistance I could also do the attestation. Ideally WS would just refer to those instead of implicitly claiming to be authoritative but obviously as I don't know cstenglein, I won't take his word for it neither. Not easy to tackle this.
Figuring out in which state the system was a month back is probably feasible as you still have your image from "2."
It is definitely feasible but would still likely take a lot of time, also for you to in the end configure your setup to account for the changes that would be needed. I agree the best course of action is to go with 3.
Maybe, with some assistance I could also do the attestation
Do you mean the signed assertion like cstenglein's? That would be awesome. The instructions are here. The more assertions the merrier.
Maybe, with some assistance I could also do the attestation
Do you mean the signed assertion like cstenglein's? That would be awesome. The instructions are here. The more assertions the merrier.
All serious wallets should have assertions signed by several competitors or otherwise plausibly non-complicits if there was something wrong with the binary. My project is trying to push the issue more in front of the users.
This is what we are trying with the community assertions:
The incentive to spend time and actually submit such signed assertions is very small unfortunately, which is why we don't have a large number of contributors. Do you have any ideas how to incentivize "non-complicits" to contribute?
I hope your project manages to do this successfully!
I think, the reproduction is only difficult the first time around. After that, you have a script to do it, so the real hurdle is to get people to start. Initial costs are x20 of any later iteration.
WalletScrutiny could feature products that provably get scrutinized by competitors but counting assertions of potential Sybils is not worth it. If you want to help with a framework of how to pull this off and popularize it, it would help me a lot to help transparent projects.
Imagine what a strong message this would send if a group of actually open source HWWs vouched for each-other's transparency but not for Ledger and others that are closed source.
I consider Ledger's popularity a stain on Bitcoin's supposed "Don't trust! Verify!" mantra.
In the end, if wallets kept checks on each other, I could skip re-building and just refer to those other 10 competitors that vouch for the binary matching the code.
I think, the reproduction is only difficult the first time around. After that, you have a script to do it, so the real hurdle is to get people to start. Initial costs are x20 of any later iteration.
I meant a special modification only for v9.10.0. From v9.10.1 onwards your current script will work as is.
I am not sure about the idea of competitors checking on each other. The wider community checking the binaries is more appealing to me.
I think, the reproduction is only difficult the first time around. After that, you have a script to do it, so the real hurdle is to get people to start. Initial costs are x20 of any later iteration.
I meant a special modification only for v9.10.0. From v9.10.1 onwards your current script will work as is.
My comment was about wrapping my head around how your assertions work and vetting the process of reproduction to a degree I feel comfortable to not have been fooled into endorsing something that's obviously a scam, to then put it into a script that will mostly work with all future releases. That process is expensive compared to running the script on future releases. I maybe almost have that script. My script reproduces but doesn't sign and push assertions.
I am not sure about the idea of competitors checking on each other. The wider community checking the binaries is more appealing to me.
With "random plebs" the problem is they can easily be Sybils, don't necessarily know what they are doing and don't have as much at stake as a well known developer of a competing product.
Competing product devs would have the incentive to swap their approval of reproducibility with yours. Especially between Trezor clones for example, it's easier to get started, too, as it's mostly the same.
We'll go with that: File this one as not reproducible for us but hopefully next time we will be able to reproduce
v9.11.0 is released at https://github.com/digitalbitbox/bitbox02-firmware/releases/tag/firmware-btc-only%2Fv9.11.0 - please let me know how it goes with reproducing it.
Closing this since it is resolved as discussed. Please reopen as needed.
Re-opening as the same problem happened to the upcoming v9.12.0 release. Seeing that it happened for the second time shortly after the initial release build, it is now worth investigating what causes this issue.
I found the likely cause of the problems and fixed it here: https://github.com/digitalbitbox/bitbox02-firmware/pull/933/files
I could reproduce v9.10.0 again and get the same hash as here.
@Giszmo you could use build.sh from the fix above, or apply git config --global --add safe.directory /path/to/bitbox02-firmware before building inside the Docker container, same as here.
WalletScrutiny is kind of on halt. Donation drive didn't even finance the 3 weeks we ran it. I'm checking for donations daily but absent funding I'm not inclined to attend to the desires of the 2000 providers or even the 20 or so providers of reproducible products. I will probably put some "last reproducibility check" date further up on the page. If this issue is linked from the Analysis, people worried about the verdict will find it.
Closing as the firmware is reproducible again with the fix linked above.