scylladb/scylla-ccm

nodetool_additional_test.py:TestNodetool.concurrent_repair_test occasionally fails to set up CLASSPATH

bhalevy opened this issue · 1 comments

bhalevy commented

Seen in the following runs for example:
https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-release/523/testReport/nodetool_additional_test/TestNodetool/concurrent_repair_test/
https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-release/535/testReport/nodetool_additional_test/TestNodetool/concurrent_repair_test/

Standard Output
Failed repair  with  Nodetool command '/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/2d91e5f6a06586cccc8d2904198ea44192f4a193/scylla-tools-java/bin/nodetool -h 127.0.9.1 -p 7109 repair' failed; exit status: 1; stderr: You must set the CASSANDRA_CONF and CLASSPATH vars

Failed verify_info  with  Nodetool command '/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/2d91e5f6a06586cccc8d2904198ea44192f4a193/scylla-tools-java/bin/nodetool -h 127.0.9.1 -p 7109 info' failed; exit status: 1; stderr: You must set the CASSANDRA_CONF and CLASSPATH vars

Failed verify_info  with  Nodetool command '/jenkins/workspace/scylla-master/dtest-release/scylla/.ccm/scylla-repository/2d91e5f6a06586cccc8d2904198ea44192f4a193/scylla-tools-java/bin/nodetool -h 127.0.9.1 -p 7109 info' failed; exit status: 1; stderr: You must set the CASSANDRA_CONF and CLASSPATH vars

I managed to reproduce this locally after a few runs and the issue stems from node1/bin/cassandra.in.sh being empty,
causing CLASSPATH to remain unset.

$ ll /local/home/bhalevy/.dtest/dtest-4hdqgq43/test/node1/resources/cassandra/bin/cassandra.in.sh /local/home/bhalevy/.dtest/dtest-4hdqgq43/test/node1/bin/cassandra.in.sh
-rw-rw-r--. 1 bhalevy bhalevy    0 Jun 28 14:56 /local/home/bhalevy/.dtest/dtest-4hdqgq43/test/node1/bin/cassandra.in.sh
-rw-rw-r--. 3 bhalevy bhalevy 5687 Jun 23 17:44 /local/home/bhalevy/.dtest/dtest-4hdqgq43/test/node1/resources/cassandra/bin/cassandra.in.sh
bhalevy commented

It looks like the test is calling node.nodetool multiple times in parallel and they all end up calling self.get_env in parallel copying $install_dir/cassandra.in.sh to $node_path/cassandra.in.sh using shutil.copy and it ends up empty.