Consider enabling the daily build in TravisCI
Closed this issue · 6 comments
Thank you @taylorhakes for taking care of #95
The conversation is locked there, so I will continue here. CI stands for continuous integration. In the unstable ecosystem of front-end dependencies, it's essential to build software on a daily basis.
If the build is green but happened a month ago, that does not guarantee that is will be successful today.
Enabling daily builds is as easy as three mouse-clicks, docs: https://docs.travis-ci.com/user/cron-jobs/
Hi limonte. Thanks for the suggestion. I gave it some thought and I agree that a build a month ago does not mean it is green today, but I think that would be very rare for this library. It would mean that a dependency changed (this library has no dependencies) or that the build environment changed. If the build environment changed, a failed build on a random day is most likely a case of a false positive versus an actual issue. It seems highly unlikely that a new version of node will come out and break compatibility. I feel false positives are a high concern and will force me to keep an eye on the library on a daily basis, which I would like to avoid. As you have seen, Travis is not perfect and the build can fail because Travis has an issue.
Do you know of other cases where the build breaks and it's an actual issue that needs fixing? My current feeling is if there are no code changes, it still safe to assume this library is working.
All your points are valid @taylorhakes
Except maybe one, from my experience, TravisCI is quite reliable and produces false failures quite rarely.
What I'm concerned about is the CX (contributor experience). Here's the example: a build was green a month ago and in the meantime it became red, but you don't know about it as it was not triggered.
And then, external contributor sends a PR, a build for that PR fails, contributor starts investigation of a failure, realizes that it's not related to his/her change, but to the project CI configuration. Will he/she get a positive experience of that? I don't think so.
Anyways, the happiness of contributors might me out of your priorities scope, and I'll totally understand if you will not enable daily builds.
Hi @limonte . You bring up a great point about contributor experience. I care very much about making it a good experience for contributors.
I think my experience is a little different then yours. Over the life of this project I have had many false positives from Travis somewhere in the range of 5-8 of them and no discovered issues. I fixed them by rerunning the build in the past. I have checks before merge to require the tests to pass, so people can't merge with a failing build. If master is failing, it is almost certainly something wrong with Travis (not 100%).
In regards to the contributors, if contributors or users of the library see a failed build they will assume that the project is not safe to use (as happened in the last issue), which is not the case.
If there is an actual issue and someone tries to create a PR and the build fails through no fault of their own, that is a bad experience. Unfortunately, that will not be solved even by a daily build. There is 24 hours between when the build could pass and someone could submit a PR in which they would see no failed build. In addition, in my experience, very few contributors do serious investigation to even notice the build was failing before their changes. For me, or anyone else, to check the status of this library every day, just in case the build fails seems like a bad trade off to me, based on my experience. It's a lot of work for an extreme edge case.
In regards to this issue, it happened because the PR was passing tests, but I didn't check the masters build status once it was merged. In the future I will make sure master is passing once PRs are merged. That should avoid the issue of seeing a failing build when it's not the case. I know it's not exactly what you wanted, but does that seem like a fair tradeoff?
Worksforme 👍 Thank you for clarifying the details, @taylorhakes!
FWIW, TravisCI is fine, your configurations are not :)
Here's the screenshot of my daily builds in CatLight, all nice and green without manual restarts:
@limonte What is the issue with the configuration? I am happy fix any issue you can find.
Your builds being green has no bearing on whether there is or isn't an issue with Travis. You can't declare Travis bug free because you haven't run into a bug. There are millions of bugs in apps you use all the time that you haven't experienced. Here is a wiki page that describes the logic issue https://en.wikipedia.org/wiki/Evidence_of_absence