JuliaGraphs/Graphs.jl

Stop running CI on nightly?

gdalle opened this issue · 5 comments

gdalle commented

It's not obvious at all that packages should run CI on nightly, for instance because it might hide actual breaking changes in the language itself. I suggest removing the nightly CI

See this discourse thread

simonschoelly commented

I disagree with this - it serves as an early warning of things that might change in the future and allows one to work with Graphs.jl also on nightly. The case, that some change is only temporary is committed to nightly that breaks this package but is then removed afterwards never as far as I know never occurred for any of the packages here.

I also am not sure how this will hide breaking changes in the language itself - if that is indeed the case, then the developers of Julia should provide some kind of mechanism to disable the nightly tests on PkgEval.

gdalle commented

Fixed by making nightly CI failures non blocking

gdalle commented

I'd like to revisit this cause the signal sent by CI is very noisy due to nightly failures that are not always our fault. When you look at the PRs, they're all marked as failing because of nightly (the first three actually pass the tests). CI failures are not blocking for the badge on the README, but it's still a hassle for PR submitters and reviewers alike.
And again, Tim Holy himself stated that testing on nightly is a bad idea because it makes packages adapt to breaking changes instead of forcing the language updates to be non breaking (https://discourse.julialang.org/t/why-do-packages-run-continuous-integration-tests-on-julia-nightly/101208/3?u=gdalle)
@etiennedeg @simonschoelly thoughts?

image

etiennedeg commented

I don't have a strong opinion on this because each time the test fails, I check if this is nightly or not, and it's not that annoying. On the other hand, I never dig to see what is failing on nightly, so nightly tests are kinda useless for me...

  • 👍1
gdalle commented

True but it's an additional step, that may confuse infrequent contributors, and that makes it hard for us to quickly identify which PRs are ready and which are still failing just by looking at the list (since there's a red cross everywhere)