conda/scans.py heterogenous behavior on Harvard cluster (Odyssey)

[josephvitti]

I'm running XPEHH calculations for a bunch of different population pairs. Using the same script, a bunch of these have gotten started running, while the latter half fails with an unexpected error:

2017-01-28 09:48:28,476 - cmd:112:main_argparse - INFO - software version: 2.0.0-beta-181-ge43cb00-dirty, python version: 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
2017-01-28 09:48:28,477 - cmd:115:main_argparse - INFO - command: /n/home08/jvitti/cms/cms/scans.py selscan_xpehh tmpDirKeep=False inputRefTped=/n/regal/sabeti_lab/jvitti/clear-synth/1kg_tpeds/chr1_strictMask_PJL_clear-synth-20161227.tped.gz maf=0.05 threads=8 inputTped=/n/regal/sabeti_lab/jvitti/clear-synth/1kg_tpeds/chr1_strictMask_FIN_clear-synth-20161227.tped.gz outFile=/n/regal/sabeti_lab/jvitti/clear-synth/1kg_scores/xpehh/chr1_strictMask_FIN_clear-synth-20161227_vs_PJL tmpDir=/tmp gapScale=20000 loglevel=DEBUG truncOk=False
2017-01-28 09:48:28,478 - cmd:127:main_argparse - DEBUG - using tempDir: /tmp/tmp-scans-selscan_xpehh-semx0iw6
Traceback (most recent call last):
File "/n/home08/jvitti/cms/cms/scans.py", line 580, in
util.cmd.main_argparse(commands, doc)
File "/n/home08/jvitti/cms/cms/util/cmd.py", line 130, in main_argparse
ret = args.func_main(args)
File "/n/home08/jvitti/cms/cms/scans.py", line 365, in main_selscan_xpehh
gap_scale = args.gapScale
File "/n/home08/jvitti/cms/cms/tools/selscan.py", line 530, in execute_xpehh
toolCmd = [self.install_and_get_path()]
File "/n/home08/jvitti/cms/cms/tools/init.py", line 124, in install_and_get_path
self.install()
File "/n/home08/jvitti/cms/cms/tools/init.py", line 101, in install
m.attempt_install()
File "/n/home08/jvitti/cms/cms/tools/init.py", line 145, in attempt_install
self._attempt_install()
File "/n/home08/jvitti/cms/cms/tools/init.py", line 441, in _attempt_install
pkg_version = self.verify_install()
File "/n/home08/jvitti/cms/cms/tools/init.py", line 386, in verify_install
installed_version = self.get_installed_version()
File "/n/home08/jvitti/cms/cms/tools/init.py", line 480, in get_installed_version
matches = package_info_re.match(installed_package_string)
TypeError: expected string or bytes-like object
srun: error: holy2a18202: task 0: Exited with exit code 1

And I now appear to get the same error when I run scans.py from the command line. I don't have this issue on the Broad cluster. I tried rolling back to conda 4.0.10 (which is what's installed on the Broad) but ran into another unexpected error:

(cms-env3) [jvitti@holy2a18308 /]$ conda install -q -y conda==4.0.10
Fetching package metadata ...........

InstallError: Install error: Error: 'conda' can only be installed into the root environment

I can always rsync the data to the Broad server and run the calculations there, but it'd be great to understand why this (apparently) stopped working -- spooky.

[josephvitti]

Yeep, it seems like this is also making problems for Travis...

[tomkinsc]

For the first error, replacing tools/__init__.py with the one available here should hopefully resolve it. The error, 'conda' can only be installed into the root environment is accurate; conda cannot be installed when an environment is active—it has to be installed when miniconda is included in the system PATH (but not when an environment is active). A sticking point is that conda v4.0.10 is (at last check) now unavailable. Fortunately, I submitted a fix for the activation race condition that has been accepted by conda, so we can use v4.2.16 and later (not sure if it is in the 4.3.x tree yet).

[josephvitti]

That does the trick -- thanks @tomkinsc !