Reproducibility between Julia < 1.7 and >= 1.7

Question

Reproducibility between Julia < 1.7 and >= 1.7

xukai92 opened this issue 2 years ago · 3 comments

The default RNG has changed since Julia 1.7 (https://docs.julialang.org/en/v1.7-dev/NEWS/#Random) and some of notebooks with an explicit RNG (e.g. the Bayesian ODE one) are no longer reproducible as the tutorials are generated using Julia < 1.7.
Is there a way we can code the notebook such that we set some RNG globally which works for all Julia versions?

Answer 1 · 2023-01-11T10:29:50.000Z

The only way to really guarantee reproducibility of the random numbers is to use StableRNGs. Other RNGs may vary between Julia versions and different architectures (even if the type of the RNG is the same). But probably even that is not sufficient to completely reproduce the tutorials if you switch Julia versions - some packages (such as Turing) dropped support for Julia < 1.7, and hence new releases (which might change the results, contain bug fixes etc.) are not necessarily available on all Julia versions. Another point is that Julia also changes numerical algorithms between different versions (IIRC e.g. exponentiation recently), and hence it is not guaranteed that you get exactly the same results even if no randomness is involved.

Hence given these limitations and other reproducibility problems, I'm not sure if we should try to use a more explicit RNG in the tutorials. IMO it would make the tutorials less readable (since we have to pass it around all the time) and still we might not be able to reproduce the results exactly on different Julia versions.

Answer 2 · 2023-01-11T15:54:56.000Z

Thanks for the note David. In this case, can we make sure the tutorials are generated using Julia >=1.7? We might need to update the TOML files as well to include the Julia versions.

Answer 3 · 2023-01-11T15:57:40.000Z

I can look into that this evening. IIRC it was very annoying before because one had to go through all the Manifest.toml files and adjust them manually. But with the newly introduced weekly action that updates the Manifest.toml files automatically, it should be much easier and less work for us if we want to keep up with the latest stable Julia release.