ntm-profiler running failed in docker container
Opened this issue · 5 comments
yujun2017 commented
We can't run NTM-profiler 0.3.0 due to update_db failed.
when we tried to run NTM-profiler 0.2.0 inside docker container, we got the error. I have attached error text file in the tickets. Could you please let us know how to fix the issues?
Running command:
set -u pipefail; kmc -t1 -sf1 -sp1 -sr1 -k31 171cd1ce-30eb-4cff-b32f-2a8032c51490.list 171cd1ce-30eb-4cff-b32f-2a8032c51490 171cd1ce-30eb-4cff-b32f-2a8032c51490
Traceback (most recent call last):
File "/opt/conda/envs/ntm-profiler/bin/ntm-profiler", line 322, in
args.func(args)
File "/opt/conda/envs/ntm-profiler/bin/ntm-profiler", line 89, in main_profile
species_prediction = pp.speciate(args)
File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/cli.py", line 64, in speciate
kmer_dump = fastq_class.get_kmer_counts(args.files_prefix,threads=args.threads)
File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/fastq.py", line 127, in get_kmer_counts
run_cmd(f"kmc {bins} -t{threads} -sf{threads} -sp{threads} -sr{threads} -k{klen} @{tmp_file_list} {tmp_prefix} {tmp_prefix}")
File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 391, in run_cmd
raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,stderr.decode()))
ValueError: Command Failed:
set -u pipefail; kmc -t1 -sf1 -sp1 -sr1 -k31 171cd1ce-30eb-4cff-b32f-2a8032c51490.list 171cd1ce-30eb-4cff-b32f-2a8032c51490 171cd1ce-30eb-4cff-b32f-2a8032c51490
jodyphelan commented
Hi @yujun2017 ,
Do you have the error report for v0.3.0?
yujun2017 commented
Jody:
Yes. Here is the error file when I tried to run update_db in 0.3.0 in my
mac pro.
We would like to use the 0.3.0 version in our application. but we run
into errors and try to solve it before we can push to prod.
I have another question regarding the ntm-db location. When we push an
application to our prod, where should we put the ntm-db? I just
noticed ntm-db will be created in the location that we are running the
update_db command. When we are using AWS, how do we solve this problem?
Thanks!
John
…On Thu, 14 Dec 2023 at 18:10, Jody Phelan ***@***.***> wrote:
Hi @yujun2017 <https://github.com/yujun2017> ,
Do you have the error report for v0.3.0?
—
Reply to this email directly, view it on GitHub
<#33 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
John Yu
# ntm-profiler error report
* OS: darwin
* ntm-profiler version: 0.3.0
* pathogen-profiler version: 3.1.0
* Program call:
```
{'no_clean': False, 'logging': 'INFO', 'repo': 'https://github.com/ntm-db/ntm-db.git', 'branch': 'master', 'commit': None, 'dir': '.', 'temp': '.', 'func': <function main_update_db at 0x109b35ea0>, 'software_name': 'ntm-profiler', 'tmp_prefix': 'b8f464ca-37e6-4940-a5cd-62a3651e5459', 'files_prefix': './b8f464ca-37e6-4940-a5cd-62a3651e5459'}
```
## Traceback:
```
File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 390, in <module>
args.func(args)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 216, in main_update_db
pp.run_cmd(f'ntm-profiler create_resistance_db --prefix {species.replace(" ","_")} --csv resistance_variants.csv {barcode_arg} --load')
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 392, in run_cmd
raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,result.stderr.decode()))
```
## Value:
```
Command Failed:
/bin/bash -c set -o pipefail; ntm-profiler create_resistance_db --prefix Mycobacterium_intracellulare_subsp._chimaera --csv resistance_variants.csv --barcode barcode.bed --load
stderr:
Traceback (most recent call last):
File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 390, in <module>
args.func(args)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 177, in create_resistance_db
pp.create_db(args,extra_files=extra_files)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/db.py", line 418, in create_db
mutation_lookup = get_snpeff_formated_mutation_list(hgvs_variants,"genome.fasta","genome.gff",json.load(open("variables.json"))["snpEff_db"])
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/db.py", line 193, in get_snpeff_formated_mutation_list
converted_mutations = verify_mutation_list(hgvs_variants,genes,refseq, snpEffDB)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/hgvs.py", line 322, in verify_mutation_list
mutation_conversion = get_ann(mutations_genome,snpEffDB)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/hgvs.py", line 200, in get_ann
for l in cmd_out(f"snpEff ann {snpEffDB} {uuid}"):
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 398, in cmd_out
run_cmd(cmd)
File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 392, in run_cmd
raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,result.stderr.decode()))
ValueError: Command Failed:
/bin/bash -c set -o pipefail; snpEff ann Mycobacterium_chimaera_ZUERICH-1 5753bf95-2f43-475c-8c25-e3c941e7a2a6 > bc4ae575-5929-46b6-adc2-52087cf8bd56
stderr:
java.lang.RuntimeException: Property: 'Mycobacterium_chimaera_ZUERICH-1.genome' not found
at org.snpeff.interval.Genome.<init>(Genome.java:104)
at org.snpeff.snpEffect.Config.readGenomeConfig(Config.java:784)
at org.snpeff.snpEffect.Config.readConfig(Config.java:751)
at org.snpeff.snpEffect.Config.init(Config.java:529)
at org.snpeff.snpEffect.Config.<init>(Config.java:116)
at org.snpeff.SnpEff.loadConfig(SnpEff.java:429)
at org.snpeff.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:889)
at org.snpeff.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:875)
at org.snpeff.SnpEff.run(SnpEff.java:1141)
at org.snpeff.SnpEff.main(SnpEff.java:160)
Cleaning up after failed run
```
yujun2017 commented
Jody:
I also tried to install ntm-profiler 0.3.0 in the miniconda3 docker
image, ntm-profiler can't be updated db.
here is the part of dockerfile, You can try
# Miniconda Image: Install Conda package
FROM continuumio/miniconda3:latest as conda
RUN conda config --add channels defaults && \
conda config --add channels bioconda && \
conda config --add channels conda-forge && \
conda create -n mykrobe -y -c bioconda mykrobe && \
conda install -c conda-forge zip && \
conda create -n tb-profiler tb-profiler=4.4.2 cryptography=39.0.1 && \
# conda create -n ntm-profiler ntm-profiler=0.2.0 cryptography=39.0.1
conda create -n ntm-profiler ntm-profiler=0.3.0
ENV PATH="$PATH:/opt/conda/envs/ntm-profiler/bin"
RUN /opt/conda/envs/ntm-profiler/bin/ntm-profiler update_db
John
…On Thu, 14 Dec 2023 at 18:10, Jody Phelan ***@***.***> wrote:
Hi @yujun2017 <https://github.com/yujun2017> ,
Do you have the error report for v0.3.0?
—
Reply to this email directly, view it on GitHub
<#33 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
John Yu
yujun2017 commented
Jody:
This is John Yu from BCBB/NIH and I just want to check with you what's
the status for the NTM profiler? We plan to move NTM profiler to Prod in
February and hope you can fix the issue before we move on.
Thanks!
John
…On Fri, 15 Dec 2023 at 06:56, John Yu ***@***.***> wrote:
Jody:
I also tried to install ntm-profiler 0.3.0 in the miniconda3 docker
image, ntm-profiler can't be updated db.
here is the part of dockerfile, You can try
# Miniconda Image: Install Conda package
FROM continuumio/miniconda3:latest as conda
RUN conda config --add channels defaults && \
conda config --add channels bioconda && \
conda config --add channels conda-forge && \
conda create -n mykrobe -y -c bioconda mykrobe && \
conda install -c conda-forge zip && \
conda create -n tb-profiler tb-profiler=4.4.2 cryptography=39.0.1 && \
# conda create -n ntm-profiler ntm-profiler=0.2.0 cryptography=39.0.1
conda create -n ntm-profiler ntm-profiler=0.3.0
ENV PATH="$PATH:/opt/conda/envs/ntm-profiler/bin"
RUN /opt/conda/envs/ntm-profiler/bin/ntm-profiler update_db
John
On Thu, 14 Dec 2023 at 18:10, Jody Phelan ***@***.***>
wrote:
> Hi @yujun2017 <https://github.com/yujun2017> ,
>
> Do you have the error report for v0.3.0?
>
> —
> Reply to this email directly, view it on GitHub
> <#33 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
John Yu
--
John Yu
yujun2017 commented
Jody:
I just want to update the issue we have found for latest ntm-profiler. The issue is only "ntm-profiler update_db" command.
As soon as we created db, we can run ntm-profiler without any issue.