shandley/hecatomb

Problem with running

Closed this issue · 11 comments

wertick01 commented

When I'm running the hecatomb, program stops at:
Downloading and installing remote packages...

No processes are running, the program is not executed.

beardymcjohnface commented

Hi,
That would be the conda/mamba stage; there should be a mamba process running. Does it terminate or does it just hang at that point? How are you running the pipeline and what sort of system are you running it on?

wertick01 commented

I'm running commands in ubuntu 20.04 lts server. The process just hang at this point.

hecatomb run --test --fast
Running Hecatomb
Running snakemake command:
snakemake -j 32 --use-conda --conda-frontend mamba --rerun-incomplete --printshellcmds --nolock --show-failed-logs --conda-prefix /home/user/anaconda3/envs/hecatomb/snakemake/workflow/conda -s /home/user/anaconda3/envs/hecatomb/snakemake/workflow/Hecatomb.smk -C Reads=/home/user/anaconda3/envs/hecatomb/test_data Host=human Output=hecatomb_out SkipAssembly=False Fast=True
Config file /home/user/anaconda3/envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Creating conda environment anaconda3/envs/hecatomb/snakemake/workflow/envs/plotly.yaml...
Downloading and installing remote packages.

And nothing else is happening.

beardymcjohnface commented

The conda env creation steps can take a little while. It should only be a matter of minutes each and it will only need to do it once. If it's taking longer than say 30 minutes and you can't see any conda/mamba jobs with top/htop then something is definitely broken.

You could also check your mamba install to see if it's working correctly for the above plotly env. To do this your could run:

mamba create -n testEnv -c defaults -c bioconda -c conda-forge plotly=5.3.1 python-kaleido=0.2.1
wertick01 commented

Everything is fine with mamba. Env is working properly. But problem is the same.

wertick01 commented

And every time it hangs on downloading packages for different envs: krona.yaml, mmseqs2.yaml and etc. I tried updating them manually, but it didn't help. Could it be a conflict between the versions of the layouts in each envs?

beardymcjohnface commented

Hi,
I'm able to recreate the problem with new builds of hecatomb, both for the conda build and my older builds on my anaconda channel. The problem is mamba is dying for some reason and Snakemake is not picking it up for some other reason. I'll see what I can to do fix the issue but for now you can copy and paste the snakemake command for running the test dataset and manually substitute out --conda-frontend mamba with --conda-frontend conda. Once the environments are created you should be find to go back to using the launcher.

wertick01 commented

Thanks very much, thats helped.

beardymcjohnface commented

So this is a new issue with snakemake + mamba; specifically mamba 18.* seems to be having issues snakemake/snakemake#1287

There's a new release of mamba--19.0.0--which seems to be working with snakemake again. You should be able to activate your Hecatomb environment and update mamba:

mamba update mamba

Assuming everything is working again, when I push the next Hecatomb update I'll also update the build with the current mamba and snakemake versions and close this issue then.

wertick01 commented

Thanks, updating mamba helped.
And hecatomb has one more problem with importing pysam. Your algorithm uses pysam 0.10.0 wich works with python 3.4.

I'm running hecatomb with assembly works out at 92% and giving an error
/home/user/anaconda3/envs/hecatomb/bin/python3.10 /home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py
Traceback (most recent call last):
File "/home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py", line 8, in
import pysam
ModuleNotFoundError: No module named 'pysam'
[Tue Nov 30 17:58:23 2021]
Error in rule contig_read_taxonomy:
jobid: 51
output: hecatomb_out/RESULTS/contigSeqTable.tsv
log: hecatomb_out/STDERR/contig_read_taxonomy.log (check log file(s) for error message)
Logfile hecatomb_out/STDERR/contig_read_taxonomy.log not found.

Traceback (most recent call last):
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 593, in _callback
raise ex
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 579, in cached_or_run
run_func(*args)
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 2461, in run_wrapper
raise ex
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 2358, in run_wrapper
run(
File "/home/user/anaconda3/envs/hecatomb/snakemake/workflow/rules/03_mapping.smk", line 97, in __rule_contig_read_taxonomy
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 1369, in script
executor.evaluate()
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 381, in evaluate
self.execute_script(fd.name, edit=edit)
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 580, in execute_script
self._execute_cmd(
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 412, in _execute_cmd
return shell(
File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/shell.py", line 287, in new
raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail; /home/user/anaconda3/envs/hecatomb/bin/python3.10 /home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py' returned non-zero exit status 1.

I tried to install pysam with mamba, but...

Encountered problems while solving:

  • package pysam-0.10.0-py34_1 requires python 3.4*, but none of the providers can be installed
beardymcjohnface commented

ah, there is a conda env for that rule that calls for pysam>=0.16 but I forgot to add that to the rule when I moved it from a run: directive to a script: directive. I'll fix that in the next release but for now you should be able to add pysam to your hecatomb env after changing to python 3.9

mamba install python=3.9
mamba install -c bioconda pysam
beardymcjohnface commented

I've created a new release for hecatomb that fixes this issue, and it should be available soon from bioconda (it will be version 1.0.0.beta.3). You should be able to run mamba update -c bioconda hecatomb once it's available.