pilgun/acvtool

Adding .ec files but get less reported coverage

Closed this issue · 11 comments

jwlin commented

Hi,

I have four .ec files by executing four tests on the same app, but the reported coverage from two of the .ec files is HIGHER than the coverage from all four of them. This is strange to me because intuitively the cumulative coverage should be only increased by adding more .ec files. Specifically, I have the following metadata and two ec folders:

q
├── astro.pickle
├── ec1
│   ├── onstop_coverage_1610318914750.ec
│   └── onstop_coverage_1610318965061.ec
└── ec2
    ├── onstop_coverage_1610318902020.ec
    ├── onstop_coverage_1610318914750.ec
    ├── onstop_coverage_1610318947862.ec
    └── onstop_coverage_1610318965061.ec

After executing acv report com.metago.astro -p ~/q/astro.pickle -ec ~/q/ec1 -o ~/q, I got a coverage: 76534 of 627614 | 12.19444%

Then, after executing acv report com.metago.astro -p ~/q/astro.pickle -ec ~/q/ec2 -o ~/q, I got a coverage: 69355 of 627614 | 11.05058%

This is weird because the files in ec1 are also in ec2. How can the cumulative coverage decrease by just adding more .ec files?

The files can be downloaded here:
https://drive.google.com/file/d/1W---ONiffuW9Shje1Hth2lgkghQY-5H0/view?usp=sharing

pilgun commented

Hi @jwlin, in this version acvtool calls the stop command each time the report generated. It needs a small change in the code to achieve cumulative measurements. Could you try for now to use acvtool from another project https://github.com/pilgun/acvcut? It contains a more fresh acvtool version (just use the same way acvtool.py). To generate cumulative ec files call acvtool.py stop command. You will also need to kill your app when you want to start measurements from scratch again. I will later merge this update into this acvtool project.

jwlin commented

Thank you @pilgun for the quick response. Do I need to collect the ec files again with the acvtool.py of acvcut? I tried to start over by instrumenting the apk with the acvtool.py of acvcut, but the instrumented apk crashed after installation.

I also tried to generate the report for the same ec files with the acvtool.py of acvcut, but got errors as follows:

$ python acvtool.py report com.metago.astro -p ~/q/astro.pickle -ec ~/q/ec1 -o ~/q
report generating...
concise report...
Traceback (most recent call last):
  File "acvtool.py", line 247, in <module>
    main()
  File "acvtool.py", line 244, in main
    run_actions(parser, args)
  File "acvtool.py", line 96, in run_actions
    ignore_filter=args.stubs)
  File "/home/junwei/acvcut/smiler/reporter.py", line 41, in generate
    save_concise_report(report_dir, ec_files, smalitree)
  File "/home/junwei/acvcut/smiler/reporter.py", line 56, in save_concise_report
    cover_smalitree_cumulatively(st, ec_coverage)
  File "/home/junwei/acvcut/smiler/reporter.py", line 369, in cover_smalitree_cumulatively
    if cl.is_coverable():
  File "/home/junwei/acvcut/smiler/instrumenting/apkil/classnode.py", line 260, in is_coverable
    if self.ignore:
AttributeError: 'ClassNode' object has no attribute 'ignore'
pilgun commented

Yes, you need to instrument it with the new version and use the new pickle file.

jwlin commented

I tried to instrument the app with the new acvtool.py but the instrumented app crashes...

Edited: The apk file can be found here:
https://drive.google.com/file/d/1Jt1Tp0i3DxtpOgTaY1OfgjnJFAuohuwS/view?usp=sharing

pilgun commented

that's a pitty, would you mind to share the apk? maybe I could take a look this week

jwlin commented

I am a little confused about this tool (the current version). Does it support multiple .ec files and report cumulative coverage? Particularly,

  • I "acv start -> test manually -> Ctrl + C" for three times, and the tool generates three .ec files. When I acv report, does the tool correctly report the cumulative coverage?
  • Alternatively, If I manually pull the three .ec files and acv report with -ec option, does the tool correctly report the cumulative coverage?
pilgun commented

The current version (old) may not generate several ec files in a single run. The terminal should take Ctrl+C only once, the ec file gets generated and the tool finalises the app. Extra files could left from other unsuccessful tries.

pilgun commented

For the second part of the question, the tool can generate a report from all ec files from a directory. The report is cumulative in this case. But not sure that it will be correct, because with this version you can run tests only not cumulatively.

jwlin commented

Thank you for the clarification. If I understand correctly, It basically says that if I have multiple test case scenarios, I need to run them all in a single run to get the coverage as a whole, i.e., with only one .ec file. The tool (current version) does not support multiple "single runs" and then reporting the cumulative coverage.

pilgun commented

Ok, it took time to rethink your question. I think it's correct to apply tests on separate app runs, pull ec files, then put them into the same directory and generate the report. The report in this case represent the cumulative coverage of all runs. E.g. if you put ec1 and ec2 in the same folder and generate the report you may get at least 12.19444% from your example.

jwlin commented

No. ec2 already includes the files in ec1. In other words, ec1 is a subset of ec2, but reports larger coverage than ec2.