Auto-watch

[gracinet]

This is a long standing improvement wish, see the original writing on launchpad

In short, as projects evolve, their dependencies change, and nobody manages to keep the watch list up to date (it could very well be under someone else's responsibility). Consequences are the natural ones: builders keep running for addons that have long been fixed in all incarnations of the buildout, people don't think to track the bulk of odoo and get nasty surprises whereas the buildbot should have warned them.

What held this back for so long is that the polling is done from the buildmaster, hence the watch parameters are set in the master-side MANIFEST.cfg. Updating it would mean many things that the buildmaster is not really fit to perform:

1. actually grabbing the buildouts in the most frequent cases where they are on some separate VCS.
2. installing enough of the buildouts to be able parse the addons and gp.vcsdevelop directives (therefore, with bootstrap-hell)
3. according to the VCS in use, maybe also actually retrieve the sub dependencies for inspection.

This means that the buildmaster must reimplement and run a good deal what the slaves perform, and this almost defeats the idea of a distributed system.

So here's a new yet obvious idea: this just means that this work to be done on the buildslaves! And indeed, in the course of an ordinary build, the buildout lies open, ready for introspection of what is from a live revision/commit and what is not. The helper methods of the recipe are even there to serve, if needed.

At Anybox, we already have crons that does bin/buildbot reconfig every hour, so we don't even need hot-reloading of the configuration upon completion of the analysis, although that'd also be fun. The following should be enough:

• allow watch configuration to be externalized to independent enough files (one per section in MANIFEST.cfg
• have the builds dump the watch directive in a small file right after performing bin/buildout and send it back to the master with a FileUpload step.
• having one watch configuration file per build factory will allow us to put locks on the upload steps without blocking unrelated builds
• some exceptions must be made for the builds that can run on old versions of the buildouts (e.g with the hgtag scheme, or release builds that run by definition on a frozen version of the buildout). The simplest would be to introduce a new option for MANIFEST.cfg: auto-watch, defaulting to True.

That's not a very tight design, and in particular, it means that a change in dependencies would not be immediately applied master-side, but who would care about one hour lag in that context whereas our problem today is that outdated watches stay for weeks if not months ?

That being said, the parsing of those independent watch directives should not fail ever, otherwise one bogus conf would stop the whole updating for a long while, and jeopardize the next restart if needed (this is not new). That raises the question of properly reporting errors

Cc @sbidoul @jssuzanne @pverkest @ccomb for feedback.

I'm willing to come up with a prototype real soon now, being tired of having errors that should have already been detected each time I reforce a bunch of builds to test something else than the builds outcomes themselves.

[gracinet]

Progress can be followed in the auto-watch branch. At this point, all we need is to add the build steps to grab the helper, call it and upload the results.

[gracinet]

Anybox's public buildbot has been running this branch since yesterday. Preliminary results are ok since lp issue has been taken care of.

[gracinet]

This has been running successfully on Anybox's buildbots for a few days… Just merged auto-watch in master (a ff merge actually)

auto-watch's default is still 'false', but that may change soon

[gracinet]

It's been working for all our buildbots flawlessly for a good while, now, time to close