Error in rule concatenate_contigs

Question

Error in rule concatenate_contigs

Closed this issue 8 months ago · 14 comments

Hi Vini,

I am having this error, any clue?

Config file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Using shell: /bin/bash
Provided cores: 48
Rules claiming more threads will be scaled down.
Job stats:
job count

DAS_tool 1
all 1
assembly_report 1
bins_report 1
cog_functional_parser 1
concatenate_benchmarks 1
concatenate_contigs 1
concatenate_taxonomies 16
concoct 1
create_contigs_index 1
decompress_catalogue 1
diamond 1
fastp_pe 5
index_reads 5
jgi_summarize_bam_contig_depths 1
lineage_parser 2
map_reads 5
megahit 5
merge_fastqs 10
metabat2 1
plot_benchmarks 1
plot_cog_functional 1
prodigal 1
rename_contigs 5
sort_reads 5
taxonomy_parser 1
vamb 1
total 76

Select jobs to execute...

[Wed Jan 17 02:47:18 2024]
rule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz
output: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit.html, output/qc/fastp/SF6228_S311_single_unit.json
log: output/logs/qc/fastp/SF6228_S311-single_unit.log
jobid: 9
benchmark: output/benchmarks/qc/fastp/SF6228_S311-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz
wildcards: sample=SF6228_S311, unit=single_unit
threads: 12
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024]
rule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz
output: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit.html, output/qc/fastp/SF6227_S301_single_unit.json
log: output/logs/qc/fastp/SF6227_S301-single_unit.log
jobid: 4
benchmark: output/benchmarks/qc/fastp/SF6227_S301-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz
wildcards: sample=SF6227_S301, unit=single_unit
threads: 12
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024]
rule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz
output: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit.html, output/qc/fastp/SF6229_S321_single_unit.json
log: output/logs/qc/fastp/SF6229_S321-single_unit.log
jobid: 14
benchmark: output/benchmarks/qc/fastp/SF6229_S321-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz
wildcards: sample=SF6229_S321, unit=single_unit
threads: 12
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024]
rule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R2_001.fastq.gz
output: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz, output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz, output/qc/fastp/SF6231_S341_single_unit.html, output/qc/fastp/SF6231_S341_single_unit.json
log: output/logs/qc/fastp/SF6231_S341-single_unit.log
jobid: 24
benchmark: output/benchmarks/qc/fastp/SF6231_S341-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz, output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz
wildcards: sample=SF6231_S341, unit=single_unit
threads: 12
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmprqv2flns.wrapper.py
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmpae7zvi8l.wrapper.py
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmpw1tal6ih.wrapper.py
/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmpwf7uklzc.wrapper.py
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
(fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz --out1 output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz --json output/qc/fastp/SF6227_S301_single_unit.json --html output/qc/fastp/SF6227_S301_single_unit.html ) > output/logs/qc/fastp/SF6227_S301-single_unit.log 2>&1
(fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R2_001.fastq.gz --out1 output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz --json output/qc/fastp/SF6231_S341_single_unit.json --html output/qc/fastp/SF6231_S341_single_unit.html ) > output/logs/qc/fastp/SF6231_S341-single_unit.log 2>&1
(fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz --out1 output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz --json output/qc/fastp/SF6228_S311_single_unit.json --html output/qc/fastp/SF6228_S311_single_unit.html ) > output/logs/qc/fastp/SF6228_S311-single_unit.log 2>&1
(fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz --out1 output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz --json output/qc/fastp/SF6229_S321_single_unit.json --html output/qc/fastp/SF6229_S321_single_unit.html ) > output/logs/qc/fastp/SF6229_S321-single_unit.log 2>&1
[Wed Jan 17 02:47:30 2024]
Finished job 24.
1 of 76 steps (1%) done
Select jobs to execute...

[Wed Jan 17 02:47:30 2024]
rule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz
output: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit.html, output/qc/fastp/SF6230_S331_single_unit.json
log: output/logs/qc/fastp/SF6230_S331-single_unit.log
jobid: 19
benchmark: output/benchmarks/qc/fastp/SF6230_S331-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz
wildcards: sample=SF6230_S331, unit=single_unit
threads: 12
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmpu9wepsa4.wrapper.py
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
(fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz --out1 output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz --json output/qc/fastp/SF6230_S331_single_unit.json --html output/qc/fastp/SF6230_S331_single_unit.html ) > output/logs/qc/fastp/SF6230_S331-single_unit.log 2>&1
[Wed Jan 17 02:49:01 2024]
Finished job 9.
2 of 76 steps (3%) done
Select jobs to execute...

[Wed Jan 17 02:49:01 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz
output: output/qc/merged/SF6228_S311_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6228_S311.R1.log
jobid: 8
benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R1.txt
reason: Missing output files: output/qc/merged/SF6228_S311_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz
wildcards: sample=SF6228_S311, read=R1
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz > output/qc/merged/SF6228_S311_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6228_S311.R1.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Wed Jan 17 02:49:01 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz
output: output/qc/merged/SF6231_S341_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6231_S341.R1.log
jobid: 23
benchmark: output/benchmarks/qc/merge_fastqs/SF6231_S341.R1.txt
reason: Missing output files: output/qc/merged/SF6231_S341_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz
wildcards: sample=SF6231_S341, read=R1
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz > output/qc/merged/SF6231_S341_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6231_S341.R1.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Wed Jan 17 02:49:01 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz
output: output/qc/merged/SF6228_S311_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6228_S311.R2.log
jobid: 10
benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R2.txt
reason: Missing output files: output/qc/merged/SF6228_S311_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz
wildcards: sample=SF6228_S311, read=R2
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz > output/qc/merged/SF6228_S311_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6228_S311.R2.log

[Wed Jan 17 02:49:01 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz
output: output/qc/merged/SF6231_S341_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6231_S341.R2.log
jobid: 25
benchmark: output/benchmarks/qc/merge_fastqs/SF6231_S341.R2.txt
reason: Missing output files: output/qc/merged/SF6231_S341_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz
wildcards: sample=SF6231_S341, read=R2
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz > output/qc/merged/SF6231_S341_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6231_S341.R2.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Wed Jan 17 02:49:02 2024]
Finished job 23.
3 of 76 steps (4%) done
Removing temporary output output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz.
[Wed Jan 17 02:49:02 2024]
Finished job 25.
4 of 76 steps (5%) done
Removing temporary output output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz.
Select jobs to execute...
[Wed Jan 17 02:49:03 2024]
Finished job 10.
5 of 76 steps (7%) done
Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz.
[Wed Jan 17 02:49:04 2024]
Finished job 8.
6 of 76 steps (8%) done
Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz.
[Wed Jan 17 02:49:11 2024]
Finished job 19.
7 of 76 steps (9%) done

[Wed Jan 17 02:49:11 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz
output: output/qc/merged/SF6230_S331_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6230_S331.R2.log
jobid: 20
benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R2.txt
reason: Missing output files: output/qc/merged/SF6230_S331_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz
wildcards: sample=SF6230_S331, read=R2
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz > output/qc/merged/SF6230_S331_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6230_S331.R2.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Wed Jan 17 02:49:11 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz
output: output/qc/merged/SF6230_S331_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6230_S331.R1.log
jobid: 18
benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R1.txt
reason: Missing output files: output/qc/merged/SF6230_S331_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz
wildcards: sample=SF6230_S331, read=R1
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz > output/qc/merged/SF6230_S331_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6230_S331.R1.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Wed Jan 17 02:49:14 2024]
Finished job 20.
8 of 76 steps (11%) done
Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz.
Select jobs to execute...
[Wed Jan 17 02:49:14 2024]
Finished job 18.
9 of 76 steps (12%) done
Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz.
[Wed Jan 17 02:49:16 2024]
Finished job 14.
10 of 76 steps (13%) done

[Wed Jan 17 02:49:16 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz
output: output/qc/merged/SF6229_S321_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6229_S321.R1.log
jobid: 13
benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R1.txt
reason: Missing output files: output/qc/merged/SF6229_S321_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz
wildcards: sample=SF6229_S321, read=R1
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz > output/qc/merged/SF6229_S321_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6229_S321.R1.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Wed Jan 17 02:49:16 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz
output: output/qc/merged/SF6229_S321_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6229_S321.R2.log
jobid: 15
benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R2.txt
reason: Missing output files: output/qc/merged/SF6229_S321_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz
wildcards: sample=SF6229_S321, read=R2
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz > output/qc/merged/SF6229_S321_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6229_S321.R2.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Wed Jan 17 02:49:18 2024]
Finished job 15.
11 of 76 steps (14%) done
Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz.
Select jobs to execute...
[Wed Jan 17 02:49:18 2024]
Finished job 13.
12 of 76 steps (16%) done
Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz.
[Wed Jan 17 02:49:24 2024]
Finished job 4.
13 of 76 steps (17%) done

[Wed Jan 17 02:49:24 2024]
rule megahit:
input: output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz
output: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa
log: output/logs/assembly/megahit/SF6230_S331.log
jobid: 17
benchmark: output/benchmarks/assembly/megahit/SF6230_S331.txt
reason: Missing output files: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa; Input files updated by another job: output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz
wildcards: group=SF6230_S331
threads: 48
resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6230_S331

    megahit -1 output/qc/merged/SF6230_S331_R1.fq.gz -2 output/qc/merged/SF6230_S331_R2.fq.gz                       -o output/assembly/megahit/SF6230_S331                       --presets meta-large                                   --out-prefix SF6230_S331                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6230_S331.log

    rm -rf output/assembly/megahit/SF6230_S331/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Wed Jan 17 04:09:59 2024]
Finished job 17.
14 of 76 steps (18%) done
Select jobs to execute...

[Wed Jan 17 04:10:00 2024]
rule megahit:
input: output/qc/merged/SF6231_S341_R1.fq.gz, output/qc/merged/SF6231_S341_R2.fq.gz
output: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa
log: output/logs/assembly/megahit/SF6231_S341.log
jobid: 22
benchmark: output/benchmarks/assembly/megahit/SF6231_S341.txt
reason: Missing output files: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa; Input files updated by another job: output/qc/merged/SF6231_S341_R2.fq.gz, output/qc/merged/SF6231_S341_R1.fq.gz
wildcards: group=SF6231_S341
threads: 48
resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6231_S341

    megahit -1 output/qc/merged/SF6231_S341_R1.fq.gz -2 output/qc/merged/SF6231_S341_R2.fq.gz                       -o output/assembly/megahit/SF6231_S341                       --presets meta-large                                   --out-prefix SF6231_S341                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6231_S341.log

    rm -rf output/assembly/megahit/SF6231_S341/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Wed Jan 17 04:10:17 2024]
Finished job 22.
15 of 76 steps (20%) done
Select jobs to execute...

[Wed Jan 17 04:10:17 2024]
rule megahit:
input: output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz
output: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa
log: output/logs/assembly/megahit/SF6229_S321.log
jobid: 12
benchmark: output/benchmarks/assembly/megahit/SF6229_S321.txt
reason: Missing output files: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa; Input files updated by another job: output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz
wildcards: group=SF6229_S321
threads: 48
resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6229_S321

    megahit -1 output/qc/merged/SF6229_S321_R1.fq.gz -2 output/qc/merged/SF6229_S321_R2.fq.gz                       -o output/assembly/megahit/SF6229_S321                       --presets meta-large                                   --out-prefix SF6229_S321                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6229_S321.log

    rm -rf output/assembly/megahit/SF6229_S321/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Wed Jan 17 05:17:14 2024]
Finished job 12.
16 of 76 steps (21%) done
Select jobs to execute...

[Wed Jan 17 05:17:14 2024]
rule megahit:
input: output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz
output: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa
log: output/logs/assembly/megahit/SF6228_S311.log
jobid: 7
benchmark: output/benchmarks/assembly/megahit/SF6228_S311.txt
reason: Missing output files: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa; Input files updated by another job: output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz
wildcards: group=SF6228_S311
threads: 48
resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6228_S311

    megahit -1 output/qc/merged/SF6228_S311_R1.fq.gz -2 output/qc/merged/SF6228_S311_R2.fq.gz                       -o output/assembly/megahit/SF6228_S311                       --presets meta-large                                   --out-prefix SF6228_S311                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6228_S311.log

    rm -rf output/assembly/megahit/SF6228_S311/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Wed Jan 17 06:06:58 2024]
Finished job 7.
17 of 76 steps (22%) done
Select jobs to execute...

[Wed Jan 17 06:06:58 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz
output: output/qc/merged/SF6227_S301_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6227_S301.R2.log
jobid: 5
benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R2.txt
reason: Missing output files: output/qc/merged/SF6227_S301_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz
wildcards: sample=SF6227_S301, read=R2
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz > output/qc/merged/SF6227_S301_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6227_S301.R2.log
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Wed Jan 17 06:06:58 2024]
rule rename_contigs:
input: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa
output: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_SF6228_S311.log
jobid: 6
benchmark: output/logs/assembly/rename_contigs_SF6228_S311.txt
reason: Missing output files: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa
wildcards: group=SF6228_S311
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "", $0); print $0; next}{print}' output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa > output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 06:06:58 2024]
rule merge_fastqs:
input: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz
output: output/qc/merged/SF6227_S301_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6227_S301.R1.log
jobid: 3
benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R1.txt
reason: Missing output files: output/qc/merged/SF6227_S301_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz
wildcards: sample=SF6227_S301, read=R1
resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz > output/qc/merged/SF6227_S301_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6227_S301.R1.log

[Wed Jan 17 06:06:58 2024]
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
rule rename_contigs:
input: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa
output: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_SF6230_S331.log
jobid: 16
benchmark: output/logs/assembly/rename_contigs_SF6230_S331.txt
reason: Missing output files: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa
wildcards: group=SF6230_S331
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa > output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa

[Wed Jan 17 06:06:58 2024]
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
rule rename_contigs:
input: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa
output: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_SF6231_S341.log
jobid: 21
benchmark: output/logs/assembly/rename_contigs_SF6231_S341.txt
reason: Missing output files: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa
wildcards: group=SF6231_S341
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "", $0); print $0; next}{print}' output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa > output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 06:06:58 2024]
rule rename_contigs:
input: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa
output: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_SF6229_S321.log
jobid: 11
benchmark: output/logs/assembly/rename_contigs_SF6229_S321.txt
reason: Missing output files: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa
wildcards: group=SF6229_S321
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "", $0); print $0; next}{print}' output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa > output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb
[Wed Jan 17 06:06:58 2024]
Finished job 21.
18 of 76 steps (24%) done
Removing temporary output output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa.
[Wed Jan 17 06:06:59 2024]
Finished job 6.
19 of 76 steps (25%) done
Removing temporary output output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa.
[Wed Jan 17 06:06:59 2024]
Finished job 11.
20 of 76 steps (26%) done
Removing temporary output output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa.
[Wed Jan 17 06:06:59 2024]
Finished job 16.
21 of 76 steps (28%) done
Removing temporary output output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa.
[Wed Jan 17 06:07:00 2024]
Finished job 5.
22 of 76 steps (29%) done
Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz.
[Wed Jan 17 06:07:01 2024]
Finished job 3.
23 of 76 steps (30%) done
Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz.
Select jobs to execute...

[Wed Jan 17 06:07:01 2024]
rule megahit:
input: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz
output: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa
log: output/logs/assembly/megahit/SF6227_S301.log
jobid: 2
benchmark: output/benchmarks/assembly/megahit/SF6227_S301.txt
reason: Missing output files: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa; Input files updated by another job: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz
wildcards: group=SF6227_S301
threads: 48
resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6227_S301

    megahit -1 output/qc/merged/SF6227_S301_R1.fq.gz -2 output/qc/merged/SF6227_S301_R2.fq.gz                       -o output/assembly/megahit/SF6227_S301                       --presets meta-large                                   --out-prefix SF6227_S301                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6227_S301.log

    rm -rf output/assembly/megahit/SF6227_S301/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Wed Jan 17 07:57:46 2024]
Finished job 2.
24 of 76 steps (32%) done
Select jobs to execute...

[Wed Jan 17 07:57:46 2024]
rule rename_contigs:
input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa
output: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_SF6227_S301.log
jobid: 1
benchmark: output/logs/assembly/rename_contigs_SF6227_S301.txt
reason: Missing output files: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa
wildcards: group=SF6227_S301
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "", $0); print $0; next}{print}' output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa > output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb
[Wed Jan 17 07:57:50 2024]
Finished job 1.
25 of 76 steps (33%) done
Removing temporary output output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa.
Select jobs to execute...

[Wed Jan 17 07:57:50 2024]
rule assembly_report:
input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa
output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv
log: output/logs/assembly/assembly_report.log
jobid: 26
benchmark: output/benchmarks/assembly/assembly_report.txt
reason: Missing output files: output/assembly/assembly_report/assembly_report.tsv; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Set of input files has changed since last execution; Params have changed since last execution
resources: tmpdir=/tmp

[Wed Jan 17 07:57:50 2024]
rule concatenate_contigs:
input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa
output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz
log: output/logs/mapping/concatenate_contigs/cobinning.log
jobid: 30
benchmark: output/benchmarks/mapping/concatenate_contigs/cobinning.txt
reason: Missing output files: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Set of input files has changed since last execution
wildcards: group=cobinning
resources: tmpdir=/tmp, mem_mb=520000, mem_mib=495911

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_
python -c "from future import print_function; import sys, json; print(json.dumps([sys.version_info.major, sys.version_info.minor]))"
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_
python /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py
Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_
/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py:89: SyntaxWarning: invalid escape sequence '#'
column.replace(" ", "_").replace(".", "").replace("#", "n").lower() + ".png"
01/17/2024 07:57:56 Starting script 'assembly_report.py'.
01/17/2024 07:57:56 Full script path: '/fs04/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/scripts/assembly_report.py'.

01/17/2024 07:58:00 could not convert string to float: 'Sequence file is empty.'
01/17/2024 07:58:00 Traceback (most recent call last):
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../scripts/utils.py", line 40, in driver
main_fn(args)
File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 96, in main
df = metrics_to_df(args.fastas, args.assembly_report)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 49, in metrics_to_df
metrics = [run_seqstats(file) for file in fastas]
^^^^^^^^^^^^^^^^^^
File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 39, in run_seqstats
metrics = dict(
^^^^^
File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 40, in
(k, float(v)) if "." in v else (k, int(v)) for k, v in metrics.items()
^^^^^^^^
ValueError: could not convert string to float: 'Sequence file is empty.'

Waiting at most 5 seconds for missing files.
MissingOutputException in rule assembly_report in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/assembly.smk, line 103:
Job 26 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
output/assembly/assembly_report/avg_length.png
output/assembly/assembly_report/max_length.png
output/assembly/assembly_report/median_length.png
output/assembly/assembly_report/no_bp.png
output/assembly/assembly_report/no_contigs.png
output/assembly/assembly_report/n50.png
output/assembly/assembly_report/assembly_report.tsv
[Wed Jan 17 07:58:11 2024]
Error in rule concatenate_contigs:
jobid: 30
input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa
output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz
log: output/logs/mapping/concatenate_contigs/cobinning.log (check log file(s) for error details)
conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_
shell:

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log
    
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job concatenate_contigs since they might be corrupted:
output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-17T024534.473506.snakemake.log
Traceback (most recent call last):
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in
sys.exit(main())
^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main
args.func(args)
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main
retcode = run_cmd(cmd)
^^^^^^^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd
retcode = check_call(cmd.split())
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', 'metaphor_settings.yaml', '--cores', '48', '-p', '-r', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'samples=samples.csv', '--config', 'max_mb=650000']' returned non-zero exit status 1.

Answer 1 · 2024-01-17T01:45:01.000Z

Hi @francesco-ricci,

Thank you for reporting this.

What version of Metaphor, Snakemake and Python are you using? There have been a few updates recently with the release of Snakemake 8 and it's better to make sure everything is up-to-date.

Could you also print the contents of this log file: output/logs/mapping/concatenate_contigs/cobinning.log

Thanks
Vini

Answer 2 · 2024-01-17T02:11:58.000Z

Hey Vini,

snakemae 7.32.4, Python 3.9.18, metaphor v1.7.9.

Print:

Traceback (most recent call last):
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$
rename=(not args.keepnames))
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$
for entry in entries:
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$
raise ValueError('Empty or outcommented file')
ValueError: Empty or outcommented file

Answer 3 · 2024-01-17T03:39:17.000Z

I think those versions are stable but I recommend creating a new environment from scratch, with Metaphor and Snakemake. Your versions should be: Metaphor=1.7.10, Snakemake>=8, Python=3.11.*

From that and your other log, it seems some of your assembly files were created but some may be empty. Have a look, if that's the case, delete the empty ones and resume the analysis, it should complete the missing assemblies and continue.

Let me know how it goes,
Vini

Answer 4 · 2024-01-17T04:44:47.000Z

Thanks Vini!

Answer 5 · 2024-01-17T23:27:17.000Z

Hey Vini,

so I installed all the versions you said but now I have a problem with checkm, cause it's not compatible with python3.11. Any advice?

Answer 6 · 2024-01-17T23:32:44.000Z

This is the error:

(metaphor) [fricci@m3-login2 deep_sea_corals]$ mamba install bioconda::checkm-genome python=3.11

Looking for: ['bioconda::checkm-genome', 'python=3.11']

bioconda/linux-64 Using cache
bioconda/noarch Using cache
conda-forge/linux-64 Using cache
conda-forge/noarch Using cache
bioconda/linux-64 Using cache
bioconda/noarch Using cache
pkgs/r/noarch No change
pkgs/main/linux-64 No change
pkgs/main/noarch No change
pkgs/r/linux-64 No change
Could not solve for environment specs
The following packages are incompatible
├─ checkm-genome is installable with the potential options
│ ├─ checkm-genome [1.0.11|1.0.12|...|1.0.7] would require
│ │ └─ python [2.7* |2.7.* |>=2.7,<2.8.0a0 ], which can be installed;
│ ├─ checkm-genome 1.0.11 would require
│ │ └─ python <3 , which can be installed;
│ ├─ checkm-genome 1.0.7 would require
│ │ └─ python 3.5* , which can be installed;
│ ├─ checkm-genome 1.0.7 would require
│ │ └─ python 3.6* , which can be installed;
│ ├─ checkm-genome [1.1.0|1.1.1|1.1.2|1.1.3] would require
│ │ └─ pysam >=0.12.0.1 with the potential options
│ │ ├─ pysam [0.12.0.1|0.13.0|...|0.15.4] would require
│ │ │ └─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
│ │ ├─ pysam [0.12.0.1|0.13.0|...|0.15.2] would require
│ │ │ └─ python [3.5* |>=3.5,<3.6.0a0 ], which can be installed;
│ │ ├─ pysam [0.12.0.1|0.13.0|0.14.0|0.14.1] would require
│ │ │ └─ python 3.6* , which can be installed;
│ │ ├─ pysam [0.14.1|0.15.0|...|0.15.4] would require
│ │ │ └─ python >=3.6,<3.7.0a0 , which can be installed;
│ │ ├─ pysam [0.15.2|0.16.0.1|...|0.20.0] would require
│ │ │ ├─ python >=2.7,<2.8.0a0 , which can be installed;
│ │ │ └─ python_abi 2.7.* _cp27mu, which can be installed;
│ │ ├─ pysam [0.15.2|0.15.4|...|0.21.0] would require
│ │ │ ├─ python >=3.6,<3.7.0a0 , which can be installed;
│ │ │ └─ python_abi 3.6. _cp36m, which can be installed;
│ │ ├─ pysam [0.15.2|0.15.4|...|0.21.0] would require
│ │ │ ├─ python >=3.7,<3.8.0a0 , which can be installed;
│ │ │ └─ python_abi 3.7. _cp37m, which can be installed;
│ │ ├─ pysam [0.15.2|0.15.3|0.15.4] would require
│ │ │ └─ python >=3.7,<3.8.0a0 , which can be installed;
│ │ ├─ pysam [0.15.2|0.16.0.1|...|0.22.0] would require
│ │ │ ├─ python >=3.8,<3.9.0a0 , which can be installed;
│ │ │ └─ python_abi 3.8. _cp38, which can be installed;
│ │ ├─ pysam [0.16.0.1|0.17.0|...|0.22.0] would require
│ │ │ ├─ python >=3.9,<3.10.0a0 , which can be installed;
│ │ │ └─ python_abi 3.9. _cp39, which can be installed;
│ │ └─ pysam [0.19.1|0.20.0|0.21.0|0.22.0] would require
│ │ ├─ python >=3.10,<3.11.0a0 , which can be installed;
│ │ └─ python_abi 3.10. _cp310, which can be installed;
│ └─ checkm-genome [1.1.10|1.1.11|...|1.2.2] would require
│ └─ pysam >=0.15.3 , which can be installed (as previously explained);
└─ python 3.11* is not installable because there are no viable options
├─ python [3.11.0|3.11.2|...|3.11.7] conflicts with any installable versions previously reported;
└─ python [3.11.0|3.11.1|...|3.11.7] would require
└─ python_abi 3.11.* *_cp311, which conflicts with any installable versions previously reported.

Answer 7 · 2024-01-18T02:13:06.000Z

Metaphor does not use CheckM, maybe install it in a separate environment?

Answer 8 · 2024-01-25T12:40:02.000Z

Damn right!

Anyway, I am having a similar issue again, can you help me solve it Vini?

Config file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Retrieving input from storage.
Using shell: /bin/bash
Provided cores: 64
Rules claiming more threads will be scaled down.
Job stats:
job count

DAS_tool 1
all 1
assembly_report 1
bins_report 1
cog_functional_parser 1
concatenate_benchmarks 1
concatenate_contigs 1
concatenate_taxonomies 16
concoct 1
create_contigs_index 1
decompress_catalogue 1
diamond 1
fastp_pe 4
index_reads 4
jgi_summarize_bam_contig_depths 1
lineage_parser 2
map_reads 4
megahit 1
merge_fastqs 8
metabat2 1
plot_benchmarks 1
plot_cog_functional 1
prodigal 1
rename_contigs 1
sort_reads 4
taxonomy_parser 1
vamb 1
total 62

Select jobs to execute...
Execute 4 jobs...

[Thu Jan 25 16:12:43 2024]
localrule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz
output: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit.html, output/qc/fastp/SF6230_S331_single_unit.json
log: output/logs/qc/fastp/SF6230_S331-single_unit.log
jobid: 10
benchmark: output/benchmarks/qc/fastp/SF6230_S331-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz
wildcards: sample=SF6230_S331, unit=single_unit
threads: 16
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:43 2024]
localrule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz
output: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit.html, output/qc/fastp/SF6227_S301_single_unit.json
log: output/logs/qc/fastp/SF6227_S301-single_unit.log
jobid: 4
benchmark: output/benchmarks/qc/fastp/SF6227_S301-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz
wildcards: sample=SF6227_S301, unit=single_unit
threads: 16
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:43 2024]
localrule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz
output: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit.html, output/qc/fastp/SF6228_S311_single_unit.json
log: output/logs/qc/fastp/SF6228_S311-single_unit.log
jobid: 6
benchmark: output/benchmarks/qc/fastp/SF6228_S311-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz
wildcards: sample=SF6228_S311, unit=single_unit
threads: 16
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:44 2024]
localrule fastp_pe:
input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz
output: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit.html, output/qc/fastp/SF6229_S321_single_unit.json
log: output/logs/qc/fastp/SF6229_S321-single_unit.log
jobid: 8
benchmark: output/benchmarks/qc/fastp/SF6229_S321-single_unit.txt
reason: Missing output files: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz
wildcards: sample=SF6229_S321, unit=single_unit
threads: 16
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_
[Thu Jan 25 16:13:52 2024]
Finished job 6.
1 of 62 steps (2%) done
Select jobs to execute...
Execute 2 jobs...

[Thu Jan 25 16:13:52 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz
output: output/qc/merged/SF6228_S311_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6228_S311.R2.log
jobid: 12
benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R2.txt
reason: Missing output files: output/qc/merged/SF6228_S311_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz
wildcards: sample=SF6228_S311, read=R2
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz > output/qc/merged/SF6228_S311_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6228_S311.R2.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Thu Jan 25 16:13:54 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz
output: output/qc/merged/SF6228_S311_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6228_S311.R1.log
jobid: 5
benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R1.txt
reason: Missing output files: output/qc/merged/SF6228_S311_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz
wildcards: sample=SF6228_S311, read=R1
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz > output/qc/merged/SF6228_S311_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6228_S311.R1.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Thu Jan 25 16:13:56 2024]
Finished job 12.
2 of 62 steps (3%) done
Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz.
[Thu Jan 25 16:13:57 2024]
Finished job 5.
3 of 62 steps (5%) done
Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz.
[Thu Jan 25 16:13:57 2024]
Finished job 10.
4 of 62 steps (6%) done
Select jobs to execute...
Execute 2 jobs...

[Thu Jan 25 16:13:57 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz
output: output/qc/merged/SF6230_S331_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6230_S331.R2.log
jobid: 14
benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R2.txt
reason: Missing output files: output/qc/merged/SF6230_S331_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz
wildcards: sample=SF6230_S331, read=R2
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz > output/qc/merged/SF6230_S331_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6230_S331.R2.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Thu Jan 25 16:13:57 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz
output: output/qc/merged/SF6230_S331_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6230_S331.R1.log
jobid: 9
benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R1.txt
reason: Missing output files: output/qc/merged/SF6230_S331_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz
wildcards: sample=SF6230_S331, read=R1
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz > output/qc/merged/SF6230_S331_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6230_S331.R1.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Thu Jan 25 16:13:59 2024]
Finished job 8.
5 of 62 steps (8%) done
Select jobs to execute...
Execute 2 jobs...

[Thu Jan 25 16:13:59 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz
output: output/qc/merged/SF6229_S321_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6229_S321.R2.log
jobid: 13
benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R2.txt
reason: Missing output files: output/qc/merged/SF6229_S321_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz
wildcards: sample=SF6229_S321, read=R2
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz > output/qc/merged/SF6229_S321_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6229_S321.R2.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Thu Jan 25 16:14:00 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz
output: output/qc/merged/SF6229_S321_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6229_S321.R1.log
jobid: 7
benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R1.txt
reason: Missing output files: output/qc/merged/SF6229_S321_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz
wildcards: sample=SF6229_S321, read=R1
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz > output/qc/merged/SF6229_S321_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6229_S321.R1.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Thu Jan 25 16:14:00 2024]
Finished job 14.
6 of 62 steps (10%) done
Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz.
[Thu Jan 25 16:14:01 2024]
Finished job 9.
7 of 62 steps (11%) done
Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz.
[Thu Jan 25 16:14:08 2024]
Finished job 7.
8 of 62 steps (13%) done
Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz.
[Thu Jan 25 16:14:08 2024]
Finished job 4.
9 of 62 steps (15%) done
Select jobs to execute...
Execute 2 jobs...

[Thu Jan 25 16:14:08 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz
output: output/qc/merged/SF6227_S301_R2.fq.gz
log: output/logs/qc/merge_fastqs/SF6227_S301.R2.log
jobid: 11
benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R2.txt
reason: Missing output files: output/qc/merged/SF6227_S301_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz
wildcards: sample=SF6227_S301, read=R2
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz > output/qc/merged/SF6227_S301_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6227_S301.R2.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_

[Thu Jan 25 16:14:08 2024]
localrule merge_fastqs:
input: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz
output: output/qc/merged/SF6227_S301_R1.fq.gz
log: output/logs/qc/merge_fastqs/SF6227_S301.R1.log
jobid: 3
benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R1.txt
reason: Missing output files: output/qc/merged/SF6227_S301_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz
wildcards: sample=SF6227_S301, read=R1
resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz > output/qc/merged/SF6227_S301_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6227_S301.R1.log
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_
[Thu Jan 25 16:14:08 2024]
Finished job 13.
10 of 62 steps (16%) done
Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz.
[Thu Jan 25 16:14:13 2024]
Finished job 11.
11 of 62 steps (18%) done
Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz.
[Thu Jan 25 16:14:13 2024]
Finished job 3.
12 of 62 steps (19%) done
Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz.
Select jobs to execute...
Execute 1 jobs...

[Thu Jan 25 16:14:13 2024]
localrule megahit:
input: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz
output: output/assembly/megahit/coassembly/coassembly.contigs.fa
log: output/logs/assembly/megahit/coassembly.log
jobid: 2
benchmark: output/benchmarks/assembly/megahit/coassembly.txt
reason: Missing output files: output/assembly/megahit/coassembly/coassembly.contigs.fa; Input files updated by another job: output/qc/merged/SF6228_S311_R2.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz, output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz, output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz
wildcards: group=coassembly
threads: 64
resources: tmpdir=/tmp, mem_mb=1000000, mem_mib=953675

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/coassembly

    megahit -1 output/qc/merged/SF6227_S301_R1.fq.gz,output/qc/merged/SF6228_S311_R1.fq.gz,output/qc/merged/SF6229_S321_R1.fq.gz,output/qc/merged/SF6230_S331_R1.fq.gz -2 output/qc/merged/SF6227_S301_R2.fq.gz,output/qc/merged/SF6228_S311_R2.fq.gz,output/qc/merged/SF6229_S321_R2.fq.gz,output/qc/merged/SF6230_S331_R2.fq.gz                       -o output/assembly/megahit/coassembly                       --presets meta-large                                   --out-prefix coassembly                              --min-contig-len 1000                    -t 64                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/coassembly.log

    rm -rf output/assembly/megahit/coassembly/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_
[Thu Jan 25 18:55:47 2024]
Finished job 2.
13 of 62 steps (21%) done
Select jobs to execute...
Execute 1 jobs...

[Thu Jan 25 18:55:47 2024]
localrule rename_contigs:
input: output/assembly/megahit/coassembly/coassembly.contigs.fa
output: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
log: output/logs/assembly/rename_contigs_coassembly.log
jobid: 1
benchmark: output/logs/assembly/rename_contigs_coassembly.txt
reason: Missing output files: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs.fa
wildcards: group=coassembly
resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "", $0); print $0; next}{print}' output/assembly/megahit/coassembly/coassembly.contigs.fa > output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb
[Thu Jan 25 18:55:49 2024]
Finished job 1.
14 of 62 steps (23%) done
Removing temporary output output/assembly/megahit/coassembly/coassembly.contigs.fa.
Select jobs to execute...
Execute 2 jobs...

[Thu Jan 25 18:55:49 2024]
localrule concatenate_contigs:
input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz
log: output/logs/mapping/concatenate_contigs/cobinning.log
jobid: 19
benchmark: output/benchmarks/mapping/concatenate_contigs/cobinning.txt
reason: Missing output files: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
wildcards: group=cobinning
resources: tmpdir=/tmp, mem_mb=800000, mem_mib=762940

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_

[Thu Jan 25 18:55:49 2024]
localrule assembly_report:
input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv
log: output/logs/assembly/assembly_report.log
jobid: 15
benchmark: output/benchmarks/assembly/assembly_report.txt
reason: Missing output files: output/assembly/assembly_report/assembly_report.tsv; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
resources: tmpdir=/tmp

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_
[Thu Jan 25 18:55:49 2024]
Error in rule concatenate_contigs:
jobid: 19
input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz
log: output/logs/mapping/concatenate_contigs/cobinning.log (check log file(s) for error details)
conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_
shell:

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log
    
    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_
Traceback (most recent call last):
File "/fs04/rp24/fra/analyses/deep_sea_corals/genome_resolved_analysis_coassembly_cobinning/.snakemake/scripts/tmp5p_p4byu.assembly_report.py", line 11, in
import pandas as pd
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/init.py", line 46, in
from pandas.core.api import (
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/core/api.py", line 1, in
from pandas._libs import (
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/_libs/init.py", line 16, in
import pandas.libs.pandas_parser # noqa: E501 # isort: skip # type: ignore[reportUnusedImport]
ModuleNotFoundError: No module named 'pandas.libs.pandas_parser'
RuleException:
CalledProcessError in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/assembly.smk, line 134:
Command 'source /home/fricci/rp24/fra/software/miniconda3/bin/activate '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a'; set -euo pipefail; python /fs04/rp24/fra/analyses/deep_sea_corals/genome_resolved_analysis_coassembly_cobinning/.snakemake/scripts/tmp5p_p4byu.assembly_report.py' returned non-zero exit status 1.
[Thu Jan 25 18:55:51 2024]
Error in rule assembly_report:
jobid: 15
input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa
output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv
log: output/logs/assembly/assembly_report.log (check log file(s) for error details)
conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-25T161219.079538.snakemake.log
WorkflowError:
At least one job did not complete successfully.
Traceback (most recent call last):
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in
sys.exit(main())
^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main
args.func(args)
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main
retcode = run_cmd(cmd)
^^^^^^^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd
retcode = check_call(cmd.split())
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', 'metaphor_settings.yaml', '--cores', '64', '--printshellcmds', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'samples=samples.csv', '--config', 'max_mb=1000000']' returned non-zero exit status 1.

Answer 9 · 2024-01-26T05:45:56.000Z

Looks like an issue with the version of Pandas. I can submit a fix by Monday.

Answer 10 · 2024-01-26T23:21:26.000Z

Thanks Vini!On 26 Jan 2024, at 16:46, Vini Salazar ***@***.***> wrote: Looks like an issue with the version of Pandas. I can submit a fix by Monday. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 11 · 2024-01-29T03:19:59.000Z

@francesco-ricci could you please confirm the Pandas and Python version in your metaphor conda environment?

Thank you

Answer 12 · 2024-01-29T03:39:13.000Z

Pandas 2.1.4 (I just updated it to 2.2.0); python 3.10.13 Thanks Francesco *Dr Francesco Ricci* Postdoctoral Research Fellow, Greening Lab <https://streaklinks.com/B1QGa0IR0WtvIhzTbApbDp80/http%3A%2F%2Fwww.greeninglab.com%2F> Department of Microbiology Monash University - Address: 19 Innovation Walk, Office 135, Clayton VIC 3800 Australia ᐧ

…

On Mon, 29 Jan 2024 at 14:20, Vini Salazar ***@***.***> wrote: @francesco-ricci <https://github.com/francesco-ricci> could you please confirm the Pandas and Python version in your metaphor conda environment? Thank you — Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANALJT2D7QB33SK7JIICDFDYQ4IOTAVCNFSM6AAAAABB5SH5YKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTHA4TGNRUHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 13 · 2024-02-02T02:16:30.000Z

Hi @francesco-ricci,

I looked into this one but couldn't find any obvious solutions.

Could you please try:

Updating Pandas in your Metaphor conda environment (I see you've done that already)
Check the Pandas version in this environment: conda activate /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_ && conda list | grep pandas
Delete this directory: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_ and resume your analysis

The last step should force the recreation of the conda environment used in the offending rule (assembly_report)

I see that the import error that is happening: ModuleNotFoundError: No module named 'pandas.libs.pandas_parser' is expected in Pandas<=2.0.*, but if the version is 2.1.* or above, I wouldn't expect it to happen.

Let me know how it goes and if it doesn't work, I can submit a patch with an option to turn off the assembly report.

Thanks,
Vini

Answer 14 · 2024-02-08T23:01:07.000Z

Hi VIni,

this worked! thank you!