strands-project/strands_qsr_lib

are rostests correctly executing on jenkins?

Closed this issue · 5 comments

gatsoulis commented

@marc-hanheide

Are rostests correctly executing in jenkins build server? I have noticed the following two issues in a latest build:

  • multiple_test took only about 8s, while in my system it takes over 2mins and in jenkins server if I recall well over 5mins.
  • all tests although have different execution times so hopefully they run correctly they all have the same header (saying running qsr_lib/tests/argd_tester.test and 3 tests), more or less as follows:
[ROSTEST]setup[/var/lib/jenkins/workspace/pr-indigo-strands_qsr_lib/monitored_vcs/qsr_lib/tests/argd_tester.test] run_id[1e01e192-530f-11e5-bdbf-fa163e293a7c] starting
?[1mstarted roslaunch server http://ci-slave-6:39845/?[0m

SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.13

NODES

[ROSTEST]setup[/var/lib/jenkins/workspace/pr-indigo-strands_qsr_lib/monitored_vcs/qsr_lib/tests/argd_tester.test] run_id[1e01e192-530f-11e5-bdbf-fa163e293a7c] done
[ROSTEST]Running test [arg
...[truncated 536 chars]...
sunit-argd_tester.xml]
[ROSTEST]test [argd_tester] results summary: 3 errors, 0 failures, 3 tests
[ROSTEST][ROSTEST] test [argd_tester] done
[ROSTEST]tearDown[/var/lib/jenkins/workspace/pr-indigo-strands_qsr_lib/monitored_vcs/qsr_lib/tests/argd_tester.test]
shutting down processing monitor...
[rosout-1] killing on exit
... shutting down processing monitor complete
[ROSTEST]rostest teardown /var/lib/jenkins/workspace/pr-indigo-strands_qsr_lib/monitored_vcs/qsr_lib/tests/argd_tester.test complete
Standard Error

?[31m[argd_tester-2] process has died [pid 11893, exit code 1, cmd /var/lib/jenkins/workspace/pr-indigo-strands_qsr_lib/monitored_vcs/qsr_lib/tests/argd_tester.py --gtest_output=xml:/var/lib/jenkins/.ros/test_results/qsr_lib/rosunit-argd_tester.xml __name:=argd_tester __log:=/var/lib/jenkins/.ros/log/1e01e192-530f-11e5-bdbf-fa163e293a7c/argd_tester-2.log].
log file: /var/lib/jenkins/.ros/log/1e01e192-530f-11e5-bdbf-fa163e293a7c/argd_tester-2*.log?[0m

Not a clue what is going on here as argd has 3 tests, but others have more or less tests. And also why say 3 failures but pass the test?

All tests run fine in my machine both within unittest as well as calling manually each via rostest. Any way to run all of them from the CMakelists to see if something is wrong there?

gatsoulis commented

Found here how to run them all.
They run fine for me with

catkin_make run_tests

so not clue what is happening when they run in jenkins server.

gatsoulis commented

I guess I know the answers to those now.

multiple_test took only about 8s, while in my system it takes over 2mins and in jenkins server if I recall well over 5mins.

Cause it is failing but jenkins still gives the green light.

all tests although have different execution times so hopefully they run correctly they all have the same header (saying running qsr_lib/tests/argd_tester.test and 3 tests), more or less as follows:

I guess always displays the header of the report?

marc-hanheide commented

no... the results are not collated properly... see strands-project/strands_ci#13 There might now be a few test runs going on and you might see some builds failing while I try to fix this.

gatsoulis commented

OK, thanks. Let me know when fixed as I can run an after test here also with PR
#163 which I made it for this specific purpose.