Exit code handling
timsnyk opened this issue · 13 comments
Looks like no matter what the result is, the exit code is always 0.
Can we have an option to change the exit code if one or more issues are found?
Would be great! Perhaps something that let the user choose the severity to fail, if something is present at this severity or above, like so:
$ bomber --fail MODERATE
What do folks feel the default behaviour should be?
Like the suggested flag.
Failing on vulnerability detection would be backwards incompatible, but Bomber is pre-1.x.
The change itself should be straightforward if we have consensus on the UX.
Thoughts @djschleen?
That's a great idea and I like the flag that @Feelemoon suggested.
--fail will be the flag, and options are as follows:
| Option | Return Code |
|---|---|
| NONE (default) | 0 |
| ERROR | 1 |
| UNSPECIFIED (This is a status where the provider gives us something wacky, or no info) | 10 |
| LOW | 11 |
| MODERATE | 12 |
| HIGH | 13 |
| CRITICAL | 14 |
0 is an "I don't care about filters" value and is the default (shows all output)
1 is that an error occurred
...
13 means at least 1 HIGH or "1" critical is found
14 means at least 1 CRITICAL is found.
I think those return codes shouldn't conflict with reserved exit codes like 1, 2, 127, 255 etc...
Example functionality:
This flag will apply to ALL renderers as well Output will be filtered so the proper data is displayed to STDOUT, JSON, or HTML. HTML and STDOUT will all have a message echoing the filter that is used.
A user wants to fail if there are any LOW vulnerabilities or higher. They enter -fail LOW as a flag and option. If there are any LOW vulnerabilities bomber will output info to STDOUT and exit with an error code 11.
In the case a user enters --fail HIGH as a flag and option, bomber will output an error code if there are any HIGH vulnerabilities or above. If only vulnerabilities with lower severity are found, bomber will exit with a 1.
@timsnyk @garethr @Feelemoon What do you think?
Regarding 1.0 of bomber it's definitely going to break the output. There is a branch I'm working on called VDR, where the default output of bomber will be a first class citizen formatted to the CycloneDX VDR format. We could put this feature into 1.0, or could release it from a different branch.
If only vulnerabilities with lower severity are found, bomber will exit with a 1.
I like the general proposal, but have a concern with this part.
I think this should exit 0.
If I'm using in CI, in effect what I'm saying with --fail HIGH is, "fail if you find HIGH or CRITICAL vulnerabilities". But returning 1 is going to fail the build. Which then needs custom handling. I think that's unexpected. It raises the barrier to easily integrating Bomber into existing tools.
Sorry about that! It should be a 0 on that line
I'm fine with that, although it might be better to filter output using an additional switch. Imagine a CI/CD action, where bomber outputs an error code (it found something above LOW). You will be notified and look, what is going on and might want to see the full output. Since bomber is baked in your action, you cannot re-run it easily.
How do you handle UNMAINTAINED packages?
Hey @Feelemoon if I understand your comment correctly, I think the --fail could act as that additional switch. The default would only be 0 or 1 as it is now, but if you use the --fail you'll get the extended flags.
For unmaintained, there's no output with that flag right now. Anything without a high, med, low, critical severity should dump out in UNSPECIFIED.
Error codes are in branch errorcode, now just to filter responses
Beta functionality on this branch: #165
Filtering is in. I'm thinking I may just have to fix some of the output handling for HTML, JSON, and STDOUT to mention that the output has been filtered/flag was used.
Need to update the README.md as well.
@garethr @Feelemoon - After I released the code, I think @garethr was correct about an additional flag. I named it exitcode (until we can think of a better term) and marked that flag as experimental. Here is the README.md changes.
Filtering Output
You may set the severity level with the --severity flag in order to return specific vulnerability severities. For example, if you set --severity=moderate only vulnerabilities with a severity of MODERATE or above will be returned.
For example, the following command will return only high and critical vulnerabilities.
bomber scan bom.json --severity=high Highest Severity Return Codes (Experimental)
Using the flag --exitcode, will return with an exit code representing the highest vulnerability severity found. Without this flag you can expect an exit code of 0 for success, or 1 if an error was encountered.
Assuming there is no error, the following values will be returned by bomber when --exitcode
| Severity | Return Code |
|---|---|
| UNSPECIFIED (This is a status where the provider gives us something wacky, or no info) | 10 |
| LOW | 11 |
| MODERATE | 12 |
| HIGH | 13 |
| CRITICAL | 14 |
@djschleen: I'm running bomber as follows:
bomber scan ${BOMBER_FLAGS} --ignore-file doc/bomber-ignore.txt --exitcode --severity=UNSPECIFIED --output=html container/syft.bom.json
Run bomber scan ${BOMBER_FLAGS} --ignore-file doc/bomber-ignore.txt --exitcode --severity=UNSPECIFIED --output=html container/syft.bom.json
bomber scan ${BOMBER_FLAGS} --ignore-file doc/bomber-ignore.txt --exitcode --severity=UNSPECIFIED --output=html container/syft.bom.json
shell: sh -e {0}
env:
BOMBER_VERSION: 0.4.7
BOMBER_FLAGS: --debug
[...]
2023/11/15 11:26:29 *** Vulnerabilities detected...
2023/11/15 11:26:29 *** - ansi_term is Unmaintained...
[...]
2023/11/15 11:28:37 Filtered out 0 vulnerabilities for package pkg:cargo/ansi_term@0.12.1
[..]
2023/11/15 11:28:39 EPSS response total: 2
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 strconv.ParseFloat: parsing "": invalid syntax
2023/11/15 11:28:39 fail severity: 14
What is the correct format of the ignore-file? How can I filter out the known "Unmaintained" package? Is there a way mark or suppress the ignored vulnerabilities in the HTML report?
@Feelemoon the ignore file should have a CVE on each line so:
CVE-1234-1234
CVE-1222-3333
Ignored vulnerabilities shouldn't show up on any output, but they need to be in the ignore file.
@djschleen Thanks for the explanation, this allows to supress well known CVEs. But how can I supress the "Unmaintained" message for a specific package?