/dev/urandom test randomly fails
olafhering opened this issue · 5 comments
After switching to dune, the newly added dune runtest randomly fails at ocaml-cryptokit.rpm. So far I got failures on s390x and ppc64le. Perhaps the build VM does not provide the required environment?
[ 699s] 3. /dev/urandom: chi^2 BROKEN? rerun test!
Unfortunately there is no diagnostic printed. Looking through the sources it is not obvious what could have gone wrong. device_rng seems to handle short reads. This leaves unexpected data as source for the failures.
I think it would be helpful to dump the raw bufer in case of error, to make diagnostic of unattended automated builds easier.
I got failures in CI myself. This is a statistical test, so it is expected that it fails sometimes. However, the confidence interval was too narrow, I think. I just pushed a fix with a wider confidence interval (4 sigmas). The question remains: should dune runtest include a statistical test?
I can probably not answer this question.
In my opinion dune runtest should verify functionality of the package, not functionality of the system where it runs on. This is certainly a simplification because the package relies on certain functionality of the system.
Maybe in this specific case an additional configure option could be added, to enable more tests?
I made a 1.16.1 release (https://github.com/xavierleroy/cryptokit/releases/tag/release1161) with a less sensitive test. Let's see what CI says.
In my opinion dune runtest should verify functionality of the package, not functionality of the system where it runs on
That's a good rule of thumb. However, this package is also an interface with the system (via some of the PRNGs), so there is no way to test that interface without testing a bit of the system too. Just like testing a file I/O library will test file I/O in the system and fail if disks are full, for instance...
Thanks. Now it succeeds with the first run.
Looks like the modified test in 1.16.1 works better. Closing this issue.