Distinguish warnings and errors in verification
Opened this issue · 0 comments
Currently the verifier runs a suite of unit tests against the set of WebMs and reports success and failures based on assertion exceptions being raised.
However, there is an intended variance of severity level that is missed with this design. For some tests, we want to fail the test. For example, if the video codec is not VP9, this is an outright violation of the encoding standards. However, there are some tests that are intended as warnings, flagging a file property or format meant to be scrutinized further.
I believe we should adopt a new design that does not make use of unit tests and instead collects a list of warnings and errors using the same test criteria. This list is then outputted to the user upon completion.