Discussion: How often should we build images?

Question

Discussion: How often should we build images?

fredclausen opened this issue 3 years ago · 9 comments

With the stated goal of this project to reduce download time and disk space utilization for users would it be in our interests to go from daily image builds to something less frequent? I know the major advantage to daily builds is that we capture any installed package updates within 24 hours of them being released, and this is highly advantageous if those updates fix a major bug or security issue.

However, we are doing so at the cost of longer download times for users 7 days a week if they use watch tower or similar systems. With the glacial pace that Debian takes for updates we are very likely seeing no package updates between builds, so we are creating the same image, many days in a row, but docker will see that as a "new" image causing increased downloads.

Security updates, while important, are likely a non-issue for most users as everything is being run behind routers will appropriate firewalls and fine to be captured in the next build a few days out, and if they are a big CVE issue we should care about it's probably big news in tech spaces and we can manually trigger a rebuild on GitHub really easy.

My proposal:

Move to weekly builds. Friday afternoon, 1400 UTC time or thereabouts, seems reasonable. We could have our individual projects set up to build 2 or more hours later so that we capture the new images for the build. Friday also seems like the best day because it's the weekend so we as maintainers can deal with any issues in either the base images or our projects, and users will have a better chance to notice and deal with issues as it's the weekend.
Alternatively, we continue with daily builds but we pin package versions of all installed Debian packages in the dockerfile- as well as their dependencies - and then check to see what has been updated, update the docker file(s) as appropriate and rebuild what needs to get updated.

Ideally, in either case, we also emulate the same process in all of our projects so that all of us are using the same set of base images always.

The second proposal kind of covers all of the bases - we'll get timely updates basically as soon as they're out, as well as not rebuilding what does't need to get rebuilt. However, this will be at significant increased complexity in the docker files as we will have to go through and pin all versions of the software we are installing as well as dependencies, and creating scripts that GitHub can run via actions to manage all of this because right now GitHub has no built-in actions that cover this. I'm willing to put the effort in if this is the desired course of action.

Answer 1 · 2022-02-02T18:30:54.000Z

(updated with a more elaborate reasoning)
I think I agree with your position(s):

I think "default" weekly builds are frequent enough. We could even go to biweekly builds if needed
Similarly, when there are emergency patches that come down from Debian, or Dependabot, or similar, we can choose to accellerate the weekly build by doing one manually.
Also, if there are changes that we make to the repo (Dockerfile, etc), we can choose to do a manual build after merging in a PR. I'd assume that we'd have a short discussion to agree that it's "build-worthy", or that one of us simply takes an executive decision to press the Manual Build button.
If you must update your stuff more often, then by all means, add an apt upgrade command to your downstream local Dockerfile and manage your own app's build frequency that way. It's not efficient, but if you can't wait for the next programmed build, then efficiency is obviously not your main concern.

Either way, this will give us most flexibility:

if nothing happens, then we're building max. once per week
if needed (urgent security updates or new features deployed), we can do an immediate build
if a user of the base image doesn't like this frequency, they can add things to their Dockerfile to update at build time, or simply not use the base image

Answer 2 · 2022-02-02T20:33:52.000Z

One additional question that just occurred to me...
Since we are depending on often older debian-slim build for the base image, shouldn't we do a apt upgrade as part of the build process? That way we are sure that the applications inside the container are at least up to date as per our build date.

Not sure if this is desirable / not done / etc?

Answer 3 · 2022-02-02T23:40:41.000Z

That's a really good point. We have the -slim image version pinned, and dependabot watches for changes in that. At worst we are a month from system packages being updated in the current configuration, but the rebuilds daily are only capturing updates to the packages we are explicitly installing and the package dependencies installed.

I'm not sure how to rectify that with my second idea above; it wouldn't be feasible to monitor what package versions the system has installed and kick off rebuilds for those; or at least more accurately, my desire to figure it out is zero if I'm honest.

The consequence of that would be that at best we are in the same situation we are in already with a need to rebuild daily.

Maybe this isn't a big deal because, as I said, the package updates from Debian are glacial. I really lean towards a weekly, or maybe twice a week, build as the most reasonable option, and add in the "apt upgrade" as part of the rebuild for the base tag.

Answer 4 · 2022-02-02T23:45:16.000Z

(updated with a more elaborate reasoning) I think I agree with your position(s):

I think "default" weekly builds are frequent enough. We could even go to biweekly builds if needed

Similarly, when there are emergency patches that come down from Debian, or Dependabot, or similar, we can choose to accellerate the weekly build by doing one manually.

Also, if there are changes that we make to the repo (Dockerfile, etc), we can choose to do a manual build after merging in a PR. I'd assume that we'd have a short discussion to agree that it's "build-worthy", or that one of us simply takes an executive decision to press the Manual Build button.

If you must update your stuff more often, then by all means, add an apt upgrade command to your downstream local Dockerfile and manage your own app's build frequency that way. It's not efficient, but if you can't wait for the next programmed build, then efficiency is obviously not your main concern.

Either way, this will give us most flexibility:

if nothing happens, then we're building max. once per week

if needed (urgent security updates or new features deployed), we can do an immediate build

if a user of the base image doesn't like this frequency, they can add things to their Dockerfile to update at build time, or simply not use the base image

I would just like to point out that submitting a PR and merging it in will always cause a full rebuild, at least loosely speaking if the PR included a docker file. If we just update documentation it shouldn't kick off a build....if I have it set up right. We should be very circumspect in all PRs going forward - perhaps requiring at least one other reviewer, maybe all three of us - before the merge button can be hit. That's an option we can set on GitHub; I think, anyway. That way any change can be done with full knowledge of everyone and the impact can be considered by everyone.

ETA: Circumspect with new PRs going forward once we're all "done" with the set of base images. I know Mike probably has a few more images he might need to add to support all of his projects so having a review process for those seems unnecessary and the rebuilds triggered by that are not a big deal.

Answer 5 · 2022-02-03T04:53:03.000Z

I think that “at least 1 reviewer” works fine for most, maybe all cases, especially if we are 3 people in total. That will ensure that the majority of us agrees.

I’d think that weekly or biweekly (once every 14 days) autobuilds with apt upgrade would be sufficient. We can decide to manually kick another rebuild if needed.

And yes, fully agreed that we are discussing a policy that will take effect only after things stabilize.

Answer 6 · 2022-02-03T05:39:22.000Z

Regarding builds:

I like the idea of just building when dependabot pushes an update to the base image, or if we push changes.
I'm happy to write a workflow that checks for updates in upstream packages that we manually pull (such as readsb etc), and kick off a workflow when it detects a change - something like this may already exist as an action...

Regarding PRs:

Agree that PRs should require at least one reviewer, possibly multiple (a quorum of us?).
I'd like to implement functional testing of the base images, however not sure how to do this without setting up a worker at home that has access to an SDR... This would significantly reduce the risk of a PR causing an issue, as images can be tested on "real" hardware prior to merge. Something to think about for the future perhaps.

Regarding base images, I think as of today I'm "done". I can't see benefit in adding any more at the moment.

Answer 7 · 2022-02-03T07:56:02.000Z

Sorry, I just merged a PR without review...
Once I've got all the base images up, I promise I'll stop doing this. :-)

Answer 8 · 2022-02-03T13:51:23.000Z

Agree that PRs should require at least one reviewer, possibly multiple (a quorum of us?).

I can change the repo settings to require a review before merge, but I'll wait until we're in a stable state.

I think it would also be worth maybe having more than just me with "the keys" to this repo. I'm not sure if that means some kind of "GitHub org" or if I'm missing some obvious setting in the repo, but I think that would be hugely beneficial to not have one person as a failure point. I'll have a look and see what I can figure out.

I'm happy to write a workflow that checks for updates in upstream packages that we manually pull (such as readsb etc), and kick off a workflow when it detects a change - something like this may already exist as an action...

That would be excellent. If we do builds when dependabot finds an updated, and kick off automated builds if the major software packages we install are updated, I think that covers all of the concerns and lets just not do daily builds. Since the big packages are all on GitHub that I'm aware of I'm sure there are actions already in place we could use.

What would also probably be beneficial would be to start tagging our builds with a version number (maybe the -slim version number plus an incremented build number? Something like that) so that dependabot in all of our repos could get notified of changes to the images here and we could easily do builds in our projects downstream. Maybe I'm overthinking it and the same kind of workflow for watching for updates in the major package repos and kicking a build off could be applied to our projects as well.

Answer 9 · 2022-02-03T14:14:06.000Z

More research

Githuub collaborators basically you two already have access to everything but changing repo settings and some destructive actions. For all intents and purposes your roles cover what we need, but....

Github Organizations give us the ability to have higher access roles for everyone. I feel like this is the direction we should take.