unit test results in the wrong place in PRs
Closed this issue · 10 comments
as observed in strands-project/strands_qsr_lib#68 it seems the unit test results in PRs are stored in ~/.ros while for devel builds they are correctly found, e.g.
https://lcas.lincoln.ac.uk/jenkins/job/devel-indigo-strands_executive/ws/ARCH_PARAM/amd64/UBUNTU_PARAM/trusty/label/devel/test_results/repos/
not sure this is the actual problem... inestigate https://lcas.lincoln.ac.uk/jenkins/job/pr-indigo-strands_qsr_lib/63/console further... seems to put it in the right place originally, but then the test themselves end in the wrong place?
Here is the line that should ensure it's correct: https://github.com/strands-project/jenkins_scripts/blob/master/test_repositories.py#L424 and looking at the logs it seems perfectly fine to be called with the right value. So it must be something in this file, I think.
if @cdondrup has any ideas, they are very welcome... I'm pretty lost.
It's quite weird... the only difference I see is that the test results in the devel build are in
ARCH_PARAM / amd64 / UBUNTU_PARAM / trusty / label / devel / test_results / repos / gcal_routine /
whereas the pr results are in
test_results / repos / qsr_lib /
So in the devel case it creates a whole bunch of parent directories in the workspace before putting the test anywhere.
so, not only in PR they are not working. They are not working anywhere anymore. See strands-project/strands_hri#131 (comment)
The main problem is this: ros/catkin#728
and then this: ros/ros_comm#611
Obviously, comparing https://lcas.lincoln.ac.uk/jenkins/job/pr-indigo-strands_executive/95/console and https://lcas.lincoln.ac.uk/jenkins/job/pr-indigo-strands_executive/96/console there was a change in the call to rostest:
(this is a diff of build 95 and 96 showing the different calls. 95 has properly archived test results, 96 doesn't.
The new version, already call rostest using the new catkin (as in ros/catkin#728), but apparently rostest doesn't interprete this correctly... which is strange, as the latest release version (1.11.13) should have the corresponding changes in. So, it either is a bug in rostest or I'm lost on this. maybe @dirk-thomas could shed some light... I'm not prepared to open an issue on ros-comm yet, as I don't quite understand what the problem actually is...
From the console output I don't know why it would fail. If you could come up with a reproducible example where it fails with the latest versions of the packages I could look into it.
Cheers! I'll see what I can do. Much appreciated. You can confirm your rostest on your jenkins work as expected @dirk-thomas? Ours all succeed, because they are not transferred into the jenkins result dir, it seems. results for 95 look alright, as they contain the individual test outcomes, but results for 96 lack the __main__ package in the test results. Any pointer where I could read how these results are "transferred" to jenkins, or check if they are even generated (the workspace shows they are not there, so they are not generated apparently)?
One of the latest CI jobs looks good to me: http://jenkins.ros.org/job/devel-indigo-ros_comm/lastCompletedBuild/testReport/
In the current state every package should generate its test results in its build folder. I guess the best way is to reproduce the build locally and check where the test result files are being generated.
possibly (hot-)fixed by strands-project/jenkins_scripts@c70729a