Handle signing-event failures more gracefully

Question

Handle signing-event failures more gracefully

Opened this issue a year ago · 5 comments

EDIT: This issue was originally filed for all tuf-on-ci repository tools but it really only applies to signing event right now: see comments.

I'm not sure what the correct way to implement this is but: failures in automation should be really visible. We can't allow e.g. signers not being notified of expiry because of a bug -- all workflow failures should lead to an issue being filed or something like it

one example of this is when calls like this are made

if tuf-on-ci-status;  then
  ...
else
  ..
fi

if playground-snapshot fails here with a python stack trace, it just leads to the else-branch running: we can't use tool return values like this

at the very least we need to check for specific return values and make sure our "successful" return values are clearly special (values 3-125 should be fair game for app specific ones)
alternatively we could stop using return values to pass information: it is a bit ugly

Answer 1 · 2024-02-02T14:27:49.000Z

Assuming we keep using exit codes to communicate the "result":

exit codes 1 - 2, 126 - 165, and 255 are pretty much reserved https://tldp.org/LDP/abs/html/exitcodes.html
if we pick a value or two from around 200, we should be good

The more professional way to fix this might be to change the printed output to something structured: Instead of e.g. tuf-on-ci-status printing the raw status message, we'd print json with fields "success" and "status message". This way the exit value could always be 0. I don't think I will go that far yet though

Answer 2 · 2024-02-06T12:20:12.000Z

Using script specific exit codes is a good way to report error. If we need more details, we can always print to stderr and have the parent process capturing that.

Answer 3 · 2024-02-07T13:22:35.000Z

I'll have a go at this

Answer 4 · 2024-02-07T15:24:31.000Z

It turns out that this is not nearly as critical as I thought:

online-sign does not use the pattern in the original comment anymore
the only action that does care about exit values is signing-event: failures in signing event are unfortunate and should lead to a comment saying so... but they won't go unnoticed like the same pattern in online-sign could

Answer 5 · 2024-02-08T08:56:02.000Z

I have a branch https://github.com/jku/tuf-on-ci/tree/exit-value-refactor but as mentioned

this only modifies the signing event, where the only advantage is that there is a comment saying "oops something failed" instead of just a failing check
As long as we're going to do #78 in near future, I think that is not very valuable (because in PRs failing checks are visible on their own)

So I think I'm going to drop this for now.