Cancel builds when disabling a chroot as EOL

[FrostyX]

We have some builds running for months, e.g.
https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/5942294/
That's because we disabled F36 as EOL and the build couldn't finish fedora-36-s390x before that.

We need to mark all fedora-36-* build chroots (or maybe all not yet running) as canceled as a part of the chroot EOL process.

[FrostyX]

It produced a bunch of tracebacks every few seconds in frontend.log, so I ran:

coprdb=# UPDATE build_chroot SET status=2 WHERE mock_chroot_id in (149, 150, 151, 152, 153, 154) AND status=4;
UPDATE 3

[praiskup]

Is this #2842?

[FrostyX]

Please see PR #2905, I explained how it relates to #2842 there

[praiskup]

Could we do this with all states that are not finished()?

@FrostyX canceling builds is not wise - we want to cancel just particular buildroot.

Could we maybe fail the chroot when getting the task, when it is already EOLed and we can not do anything better? perhaps in #2842?

[nikromen]

re-raising something from frontend won't help here since the job is stuck in /backend/penging-jobs so frontend will process it every N secs but it won't touch it in the end.

Could we maybe fail the chroot when getting the task

Failing already EOLed/disabled chroots sounds reasonable to me