veristat-compare.py does not react well to new programs in the log
Closed this issue · 3 comments
Consider the following veristat input files:
two-ok:
file_name,prog_name,verdict,total_states
file-a,a,success,12
file-b,b,success,67
add-success:
file_name,prog_name,verdict,total_states
file-a,a,success,12
file-b,b,success,67
file-b,c,success,32
veristat produces the following log (veristat -e file,prog,verdict,states -o csv -C two-ok add-success > $tmp):
file_name,prog_name,verdict_base,verdict_comp,verdict_diff,total_states_base,total_states_comp,total_states_diff
file-a,a,success,success,MATCH,12,12,+0 (+0.00%)
file-b,b,success,success,MATCH,67,67,+0 (+0.00%)
file-b,c,N/A,success,N/A,N/A,32,N/A
And veristat-compare.py fails with the log below:
Traceback (most recent call last):
File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 256, in <module>
sys.exit(main(args.filename, summary_filename))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 233, in main
veristat_results = parse_table(csv_file)
^^^^^^^^^^^^^^^^^^^^^
File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 173, in parse_table
diff = get_state_diff(record[VeristatFields.TOTAL_STATES_DIFF])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 139, in get_state_diff
if percentage_diff := matches.group("percentage_diff"):
^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
This causes CI failures, e.g. here:
https://github.com/kernel-patches/bpf/actions/runs/6567849918/job/17845229398#step:35:111
I had a small "testsuite" when I initially developed the script.
It should probably be integrated as a local CI run.
I can do it over the weekend (unless someone else beats me to it :) )
I had a small "testsuite" when I initially developed the script. It should probably be integrated as a local CI run. I can do it over the weekend (unless someone else beats me to it :) )
I was telling @yurinnick about it, both offline and in #237 (review)
This issue is a good example as to why it will matter long term :) If not foreseeing the issue, at least we can ensure we don't regress.