how to get a reproducible build on linux ?

[jim-perkins]

Congrats on the new version of fishnet. Moving from a readable python script with the ability to utilize your own stockfish to what you have now is going to take me a bit to wrap my head around. this is where I started. the fishnet that was compiled was around 26MB and the downloaded one was around 20 MB. I am sure I did something wrong as this was my first time compiling anything for Rust. the host system is a ubuntu 20.04 server. listed are the command I ran. thanks

109 mkdir src
110 cd src/
111 ls
112 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
113 source $HOME/.cargo/env
114 git clone --recursive https://github.com/niklasf/fishnet.git
115 sudo apt install git
116 git clone --recursive https://github.com/niklasf/fishnet.git
117 cd fishnet/
118 cargo run --release --
119 mkdir downloaded
120 cd downloaded/
121 wget https://github.com/niklasf/fishnet/releases/download/v2.0.0-beta.2/fishnet-x86_64-unknown-linux-gnu
122 ls -lh
123 diff fishnet-x86_64-unknown-linux-gnu ../target/release/fishnet

Binary files fishnet-x86_64-unknown-linux-gnu and ../target/release/fishnet differ

So then I realized that the master probably wasnt the same as the beta 2 release. so I ran this
133 git checkout v2.0.0-beta.2
134 ls
135 mv target/ target.orig
136 cargo run --release --
137 cd target/release/
138 diff fishnet /home/jperkins/src/fishnet/target.orig/release/fishnet
139 history
140 diff fishnet ../../downloaded/fishnet-x86_64-unknown-linux-gnu

yea the new compile was different than the orignal compile but still wasnt the same as the downloaded.

Please forgive my lack of understanding. thanks again

PS: I have the new version running on (29) Intel Xeon E312xx (Sandy Bridge, IBRS update)cores and and it appears to be working fine. 'Nov 29 03:22:54 lichess fishnet-x86_64-unknown-linux-gnu[6239]: ><> fishnet/2.0.0-beta.2: 56 batches, 2847 positions, 6395465406 total node' . Congratulations again !

[niklasf]

There's one more optional build step that should get the binary sizes closer: https://github.com/niklasf/fishnet/blob/d79409dd65e7d3f8d05881f7f7f4122b5f2595ca/.github/workflows/release.yml#L38

That said, it's the Stockfish binaries that are meant to be reproducible byte by byte: https://github.com/niklasf/fishnet-assets. That's because even when compiling the client from source, these are included as binaries.

[jim-perkins]

Thanks for the reply. Using strip did get the files sizes very close.

As you alluded though, it is still not reproducible byte by byte. I know nothing about rust but did notice during the compile it downloaded lots of 'crates'. I imagine that makes it difficult to replicate the entire build environment. Probably lots of other reasons too that it isnt reproducible. Yes I can download the src and compile my own binary from the reviewed src.

At this point, I dont have any more questions about this issue and maybe you wish to close it with 'not a bug' or whatever you consider appropriate. thanks again

[niklasf]

Alright, thanks for testing. It must be other differences of the build environment. All the downloaded crates are pinned to exact versions by https://github.com/niklasf/fishnet/blob/master/Cargo.lock.