cormhap error
hidvegin opened this issue · 18 comments
I use the latest canu
release under Red Hat Enterprise Linux release 8.6 with slurm 22.05.7. I used this script:
/home/fk8jybr/program/canu-2.2/bin/canu -p 0.070 -d /project/denolen/output/canu-2.2/0.070/ genomeSize=4g gridOptions="--time=7-00:00:00 --partition=cpu --account denolen" batMemory=200 gridEngineArrayOption="-a ARRAY_JOBS%30" correctedErrorRate=0.070 -pacbio /project/denolen/input/LC001pacbio.fastq.gz
The script is crashed somewhere but I do not know exactly where. I re-run the same script but when canu started the jobs between 1-495 at cormhap phase the canu is failed with this error:
----------------------------------------
-- Starting command on
cd /project/denolen/output/canu-2.2/0.070
sbatch \
--depend=afterany:cormhap_0.070:cormhap_0.070:cormhap_0.070:cormhap_0.070:cormhap_0.070:cormhap_0.070 \
--cpus-per-task=1 \
--mem-per-cpu=4g \
--time=7-00:00:00 \
--partition=cpu \
--account denolen \
-D `pwd` \
-J 'canu_0.070' \
-o canu-scripts/canu.18.out canu-scripts/canu.18.sh
sbatch: error: Batch script is empty!
-- Finished on ----------------------------------------
ERROR:
ERROR: Failed with exit code ERROR:
-- Failed to submit Canu executive. Giving up after two tries.
How can I resolve this error?
The depend string definitely looks incorrect, like a previous job submission failed and/or returned an invalid message. What is the contents of canu.out that is in your folder?
Thank @skoren for your answer. The canu.out is also empty. In the canu-scripts folder, the canu.17.sh and canu.18.sh also empty. This it the canu.16.sh:
#!/bin/sh
# Path to Canu.
bin="/project/home/fk8jybr/program/canu-2.2/bin"
# Report paths.
echo ""
echo "Found perl:"
echo " " `which perl`
echo " " `perl --version | grep version`
echo ""
echo "Found java:"
echo " " `which java`
echo " " `java -showversion 2>&1 | head -n 1`
echo ""
echo "Found canu:"
echo " " $bin/canu
echo " " `$bin/canu -version`
echo ""
# Environment for any object storage.
export CANU_OBJECT_STORE_CLIENT=
export CANU_OBJECT_STORE_CLIENT_UA=
export CANU_OBJECT_STORE_CLIENT_DA=
export CANU_OBJECT_STORE_NAMESPACE=
export CANU_OBJECT_STORE_PROJECT=
rm -f canu.out
ln -s canu-scripts/canu.16.out canu.out
/usr/bin/env perl \
$bin/canu -p '0.070' 'genomeSize=4g' 'gridOptions=--time=7-00:00:00 --partition=cpu --account denolen' 'batMemory=200' 'gridEngineArrayOption=-a ARRAY_JOBS%30' 'correctedErrorRate=0.070' -pacbio '/project/denolen/input/LC001pacbio.fastq.gz canuIteration=1
In the canu-logs folder also the last two files are empty. The last not empty file contains this:
###
### Reading options from '/project/home/fk8jybr/program/canu-2.2/bin/canu.defaults'
###
# Add site specific options (for setting up Grid or limiting memory/threads) here.
###
### Reading options from the command line.
###
genomeSize=4g
gridOptions=--time=7-00:00:00 --partition=cpu --account denolen
batMemory=200
gridEngineArrayOption=-a ARRAY_JOBS%30
correctedErrorRate=0.070
canuIteration=1
I do not know what happened with the canu job but when I re-run canu, canu start all of the cormhap_1-499 files and the canu dependency job is faild.
What's the recursive file listing in your canu folder (ls -lhR /project/denolen/output/canu-2.2/0.070
)? Can you upload all the canu.*.out
files in the folder to see which is the last one that did any work? Also, any out/err files in the correction/1-overlapper as well as any *jobSubmit*.sh
files there.
I uploaded here the files:
masodik.zip
files.log
When I re-run the canu script, canu start the faild files in cormhap phase but the dependancy canu script is faild and after the finished cormhap phase, canu stop working.
Is this a different run? The files don't seem to match the run you originally mentioned in the issue. I don't see any files that say "Failed to submit" nor are there any canu.*.out jobs past 03 when the original one you posted said canu.18.out. The file canu.out is also not empty. Everything here looks OK and a subset of mhap jobs are failing which get retried and fail again at which point canu stops. This is how it should work and I don't see any weird submit commands or dependencies.
The question from these logs is why did the subset of jobs fail to complete. I'd guess insufficient time on the grid but you should be able to check the job history to see why these jobs failed. For example, check the history of job 2912743 and how much time/memory it used vs requested.
Yes, it is a new run. I restarted canu with the same script, which I mentioned earlier. But now, I get the same error. This is the files about the error:
canu_error.zip
In the slurm file, you will see the empty canu.out file and "Failed to submit".
I wouldn't worry about the second error then, the question is why did the original jobs fail. None of the error logs show a failure from the job so I'd guess you either ran out of space or they were killed by your grid manager. Check the history of the job I mentioned above on your grid and see how much memory/time it requested/used and confirm you're not at your space quota.
Thank @skoren for your answer. You were right. The problem was the run out of space. I asked more space from the HPC administrator. I hope 40 TB will be enough. Are there any option for decrease the space usage?
Yes, the FAQ has suggestions to reduce disk usage here: https://canu.readthedocs.io/en/latest/faq.html#my-assembly-is-running-out-of-space-is-too-slow. However, those options require a restart from scratch and I suspect 40tb will be sufficient without them.
Thanks @skoren for your answer.
In the cns phase I got an error again. I see in the slurm output file htis:
-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction and consensus use:
-- Šošic M, Šikic M.
-- Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
-- Bioinformatics. 2017 May 1;33(9):1394-1395.
-- http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
-- Berlin K, et al.
-- Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
-- Nat Biotechnol. 2015 Jun;33(6):623-30.
-- http://doi.org/10.1038/nbt.3238
--
-- Myers EW, et al.
-- A Whole-Genome Assembly of Drosophila.
-- Science. 2000 Mar 24;287(5461):2196-204.
-- http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
-- Chin CS, et al.
-- Phased diploid genome assembly with single-molecule real-time sequencing.
-- Nat Methods. 2016 Dec;13(12):1050-1054.
-- http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
-- Chin CS, et al.
-- Phased diploid genome assembly with single-molecule real-time sequencing.
-- Nat Methods. 2016 Dec;13(12):1050-1054.
-- http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '19.0.1' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command 'gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
--
-- Detected 48 CPUs and 48000 gigabytes of memory on the local machine.
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '19.0.1' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command 'gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
--
-- Detected 48 CPUs and 48000 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with task IDs up to 999 allowed.
--
-- Slurm support detected. Resources available:
-- 1 host with 288 cores and 11903 GB memory.
-- 58 hosts with 64 cores and 249 GB memory.
-- 184 hosts with 128 cores and 249 GB memory.
-- 4 hosts with 128 cores and 501 GB memory.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Detected Slurm with task IDs up to 999 allowed.
--
-- Slurm support detected. Resources available:
-- 1 host with 288 cores and 11903 GB memory.
-- 58 hosts with 64 cores and 249 GB memory.
-- 184 hosts with 128 cores and 249 GB memory.
-- 4 hosts with 128 cores and 501 GB memory.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Grid: meryl 24.000 GB 8 CPUs (k-mer counting)
-- Grid: hap 16.000 GB 32 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 31.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: obtovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: utgovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: cor -.--- GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 32.000 GB 1 CPU (overlap store sorting)
-- Grid: red 33.000 GB 8 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 200.000 GB 32 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 8 CPUs (consensus)
--
-- Grid: hap 16.000 GB 32 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 31.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: obtovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: utgovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: cor -.--- GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 32.000 GB 1 CPU (overlap store sorting)
-- Grid: red 33.000 GB 8 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 200.000 GB 32 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 8 CPUs (consensus)
--
-- Found PacBio CLR reads in '0.070.seqStore':
-- Libraries:
-- PacBio CLR: 1
-- Reads:
-- Raw: 130133008970
-- Corrected: 95373203011
-- Corrected and Trimmed: 92179573151
--
--
-- Generating assembly '0.070' in '/project/denolen/output/canu-2.2/0.070':
-- genomeSize:
-- 4000000000
--
-- Libraries:
-- PacBio CLR: 1
-- Reads:
-- Raw: 130133008970
-- Corrected: 95373203011
-- Corrected and Trimmed: 92179573151
--
--
-- Generating assembly '0.070' in '/project/denolen/output/canu-2.2/0.070':
-- genomeSize:
-- 4000000000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.2400 ( 24.00%)
-- obtOvlErrorRate 0.0700 ( 7.00%)
-- utgOvlErrorRate 0.0700 ( 7.00%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.2500 ( 25.00%)
-- obtErrorRate 0.0700 ( 7.00%)
-- utgErrorRate 0.0700 ( 7.00%)
-- cnsErrorRate 0.0700 ( 7.00%)
--
-- Stages to run:
-- assemble corrected and trimmed reads.
-- corOvlErrorRate 0.2400 ( 24.00%)
-- obtOvlErrorRate 0.0700 ( 7.00%)
-- utgOvlErrorRate 0.0700 ( 7.00%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.2500 ( 25.00%)
-- obtErrorRate 0.0700 ( 7.00%)
-- utgErrorRate 0.0700 ( 7.00%)
-- cnsErrorRate 0.0700 ( 7.00%)
--
-- Stages to run:
-- assemble corrected and trimmed reads.
--
--
-- Correction skipped; not enabled.
--
-- Trimming skipped; not enabled.
--
-- BEGIN ASSEMBLY
-- Using slow alignment for consensus (iteration '0').
-- Configured 280 consensus jobs.
--
-- Grid: cns 1.250 GB 8 CPUs (consensus)
--
--
--
-- Correction skipped; not enabled.
--
-- Trimming skipped; not enabled.
--
-- BEGIN ASSEMBLY
-- Using slow alignment for consensus (iteration '0').
-- Configured 280 consensus jobs.
--
-- Grid: cns 1.250 GB 8 CPUs (consensus)
--
--
-- Running jobs. First attempt out of 2.
--
-- 'consensus.jobSubmit-01.sh' -> job 3129351 task 7.
-- 'consensus.jobSubmit-02.sh' -> job 3129352 task 189.
-- 'consensus.jobSubmit-03.sh' -> job 3129353 task 202.
-- 'consensus.jobSubmit-04.sh' -> job 3129354 task 204.
-- 'consensus.jobSubmit-05.sh' -> job 3129355 task 206.
-- 'consensus.jobSubmit-06.sh' -> job 3129356 tasks 208-209.
-- 'consensus.jobSubmit-07.sh' -> job 3129357 task 211.
-- 'consensus.jobSubmit-08.sh' -> job 3129358 tasks 213-215.
-- 'consensus.jobSubmit-09.sh' -> job 3129359 tasks 217-280.
--
----------------------------------------
-- Starting command on Sun Nov 5 10:18:09 2023 with 17905.432 GB free disk space
cd /project/denolen/output/canu-2.2/0.070
sbatch \
--depend=afterany:3129351:3129352:3129353:3129354:3129355:3129356:3129357:3129358:3129359 \
--cpus-per-task=1 \
--mem-per-cpu=4g \
--time=7-00:00:00 \
--partition=cpu \
--account denolen \
-D `pwd` \
-J 'canu_0.070' \
-o canu-scripts/canu.33.out canu-scripts/canu.33.sh
Submitted batch job 3129360
-- Finished on Sun Nov 5 10:18:09 2023 (lickety-split) with 17905.432 GB free disk space
----------------------------------------
The canu.out file mention that some of the cns files are faild:
Found perl:
/usr/bin/perl
This is perl 5, version 26, subversion 3 (v5.26.3) built for x86_64-linux-thread-multi
Found java:
/opt/software/packages/jdk/19.0.1/bin/java
java version "19.0.1" 2022-10-18
Found canu:
/project/home/fk8jybr/program/canu-2.2/bin/canu
canu 2.2
-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction and consensus use:
-- Šošic M, Šikic M.
-- Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
-- Bioinformatics. 2017 May 1;33(9):1394-1395.
-- http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
-- Berlin K, et al.
-- Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
-- Nat Biotechnol. 2015 Jun;33(6):623-30.
-- http://doi.org/10.1038/nbt.3238
--
-- Myers EW, et al.
-- A Whole-Genome Assembly of Drosophila.
-- Science. 2000 Mar 24;287(5461):2196-204.
-- http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
-- Chin CS, et al.
-- Phased diploid genome assembly with single-molecule real-time sequencing.
-- Nat Methods. 2016 Dec;13(12):1050-1054.
-- http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '19.0.1' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command 'gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
--
-- Detected 3 CPUs and 5121 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with task IDs up to 999 allowed.
--
-- Slurm support detected. Resources available:
-- 1 host with 288 cores and 11903 GB memory.
-- 58 hosts with 64 cores and 249 GB memory.
-- 4 hosts with 128 cores and 501 GB memory.
-- 184 hosts with 128 cores and 249 GB memory.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Grid: meryl 24.000 GB 8 CPUs (k-mer counting)
-- Grid: hap 16.000 GB 32 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 31.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: obtovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: utgovl 24.000 GB 16 CPUs (overlap detection)
-- Grid: cor -.--- GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 32.000 GB 1 CPU (overlap store sorting)
-- Grid: red 33.000 GB 8 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 200.000 GB 32 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 8 CPUs (consensus)
--
-- Found PacBio CLR reads in '0.070.seqStore':
-- Libraries:
-- PacBio CLR: 1
-- Reads:
-- Raw: 130133008970
-- Corrected: 95373203011
-- Corrected and Trimmed: 92179573151
--
--
-- Generating assembly '0.070' in '/project/denolen/output/canu-2.2/0.070':
-- genomeSize:
-- 4000000000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.2400 ( 24.00%)
-- obtOvlErrorRate 0.0700 ( 7.00%)
-- utgOvlErrorRate 0.0700 ( 7.00%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.2500 ( 25.00%)
-- obtErrorRate 0.0700 ( 7.00%)
-- utgErrorRate 0.0700 ( 7.00%)
-- cnsErrorRate 0.0700 ( 7.00%)
--
-- Stages to run:
-- assemble corrected and trimmed reads.
--
--
-- Correction skipped; not enabled.
--
-- Trimming skipped; not enabled.
--
-- BEGIN ASSEMBLY
-- Using slow alignment for consensus (iteration '2').
-- Configured 280 consensus jobs.
--
-- Grid: cns 1.250 GB 8 CPUs (consensus)
--
--
-- Consensus jobs failed, tried 2 times, giving up.
-- job ctgcns/0007.cns FAILED.
-- job ctgcns/0189.cns FAILED.
-- job ctgcns/0202.cns FAILED.
-- job ctgcns/0204.cns FAILED.
-- job ctgcns/0206.cns FAILED.
-- job ctgcns/0208.cns FAILED.
-- job ctgcns/0209.cns FAILED.
-- job ctgcns/0211.cns FAILED.
-- job ctgcns/0213.cns FAILED.
-- job ctgcns/0214.cns FAILED.
-- job ctgcns/0215.cns FAILED.
-- job ctgcns/0217.cns FAILED.
-- job ctgcns/0218.cns FAILED.
-- job ctgcns/0219.cns FAILED.
-- job ctgcns/0220.cns FAILED.
-- job ctgcns/0221.cns FAILED.
-- job ctgcns/0222.cns FAILED.
-- job ctgcns/0223.cns FAILED.
-- job ctgcns/0224.cns FAILED.
-- job ctgcns/0225.cns FAILED.
-- job ctgcns/0226.cns FAILED.
-- job ctgcns/0227.cns FAILED.
-- job ctgcns/0228.cns FAILED.
-- job ctgcns/0229.cns FAILED.
-- job ctgcns/0230.cns FAILED.
-- job ctgcns/0231.cns FAILED.
-- job ctgcns/0232.cns FAILED.
-- job ctgcns/0233.cns FAILED.
-- job ctgcns/0234.cns FAILED.
-- job ctgcns/0235.cns FAILED.
-- job ctgcns/0236.cns FAILED.
-- job ctgcns/0237.cns FAILED.
-- job ctgcns/0238.cns FAILED.
-- job ctgcns/0239.cns FAILED.
-- job ctgcns/0240.cns FAILED.
-- job ctgcns/0241.cns FAILED.
-- job ctgcns/0242.cns FAILED.
-- job ctgcns/0243.cns FAILED.
-- job ctgcns/0244.cns FAILED.
-- job ctgcns/0245.cns FAILED.
-- job ctgcns/0246.cns FAILED.
-- job ctgcns/0247.cns FAILED.
-- job ctgcns/0248.cns FAILED.
-- job ctgcns/0249.cns FAILED.
-- job ctgcns/0250.cns FAILED.
-- job ctgcns/0251.cns FAILED.
-- job ctgcns/0252.cns FAILED.
-- job ctgcns/0253.cns FAILED.
-- job ctgcns/0254.cns FAILED.
-- job ctgcns/0255.cns FAILED.
-- job ctgcns/0256.cns FAILED.
-- job ctgcns/0257.cns FAILED.
-- job ctgcns/0258.cns FAILED.
-- job ctgcns/0259.cns FAILED.
-- job ctgcns/0260.cns FAILED.
-- job ctgcns/0261.cns FAILED.
-- job ctgcns/0262.cns FAILED.
-- job ctgcns/0263.cns FAILED.
-- job ctgcns/0264.cns FAILED.
-- job ctgcns/0265.cns FAILED.
-- job ctgcns/0266.cns FAILED.
-- job ctgcns/0267.cns FAILED.
-- job ctgcns/0268.cns FAILED.
-- job ctgcns/0269.cns FAILED.
-- job ctgcns/0270.cns FAILED.
-- job ctgcns/0271.cns FAILED.
-- job ctgcns/0272.cns FAILED.
-- job ctgcns/0273.cns FAILED.
-- job ctgcns/0274.cns FAILED.
-- job ctgcns/0275.cns FAILED.
-- job ctgcns/0276.cns FAILED.
-- job ctgcns/0277.cns FAILED.
-- job ctgcns/0278.cns FAILED.
-- job ctgcns/0279.cns FAILED.
-- job ctgcns/0280.cns FAILED.
--
ABORT:
ABORT: canu 2.2
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting. If that doesn't work, ask for help.
ABORT:
When I re-run canu, I got the same error as above. How can I resolve this error?
Given that a lot of sequential jobs are failing and they're near the end of the array of jobs, I'd suspect disk space issues again. What does one of the jobs report in it's logs (something like unitigging/5-consensus/consensus.*.out
)?
Thank @skoren for your answer. This is the consensus.3129359_234.out file:
Found perl:
/usr/bin/perl
This is perl 5, version 26, subversion 3 (v5.26.3) built for x86_64-linux-thread-multi
Found java:
/opt/software/packages/jdk/19.0.1/bin/java
java version "19.0.1" 2022-10-18
Found canu:
/project/home/fk8jybr/program/canu-2.2/bin/canu
canu 2.2
Running job 234 based on SLURM_ARRAY_TASK_ID=234 and offset=0.
-- Using seqFile '../0.070.ctgStore/partition.0234'.
-- Opening tigStore '../0.070.ctgStore' version 1.
-- Opening output results file './ctgcns/0234.cns.WORKING'.
--
-- Computing consensus for b=0 to e=2179996 with errorRate 0.0700 (max 0.4000) and minimum overlap 500
--
Loading corrected-trimmed reads from seqFile '../0.070.ctgStore/partition.0234'
----------CONTAINED READS---------- -DOVETAIL READS-
tigID length reads used coverage ignored coverage used coverage
------- --------- ------- -------- -------- -------- -------- -------- --------
890 27066 46 37 11.59x 0 0.00x 9 3.93x
1512 18478 67 60 20.85x 0 0.00x 7 3.45x
1576 18918 66 55 17.93x 0 0.00x 11 4.81x
1635 16677 74 62 30.39x 0 0.00x 12 7.25x
1670 21221 58 44 15.13x 0 0.00x 14 5.55x
2734 21021 59 47 15.71x 0 0.00x 12 4.77x
2822 22973 54 42 14.07x 0 0.00x 12 4.49x
2923 14932 83 56 37.48x 23 12.32x 4 2.89x
3902 18617 67 56 20.68x 0 0.00x 11 4.23x
4934 30219 41 33 7.13x 0 0.00x 8 2.39x
5364 18412 67 56 19.90x 0 0.00x 11 5.65x
5369 17106 72 65 31.81x 0 0.00x 7 4.29x
5557 24718 51 36 11.44x 0 0.00x 15 5.17x
5795 18946 65 54 25.11x 0 0.00x 11 6.58x
5839 19357 65 56 18.10x 0 0.00x 9 3.67x
5869 27654 45 36 8.43x 0 0.00x 9 3.01x
7159 28311 44 29 6.50x 0 0.00x 15 3.92x
7453 21214 58 40 13.68x 0 0.00x 18 7.60x
7858 17587 71 64 21.98x 0 0.00x 7 3.62x
7925 21783 58 49 14.83x 0 0.00x 9 3.16x
8092 25788 49 39 8.34x 0 0.00x 10 3.33x
8097 15750 80 77 30.45x 0 0.00x 3 1.91x
8791 22871 54 45 16.01x 0 0.00x 9 4.79x
8891 19511 63 54 21.72x 0 0.00x 9 5.11x
9463 10809 114 59 37.28x 51 20.25x 4 3.21x
9551 27131 46 31 6.92x 0 0.00x 15 4.02x
9593 19210 65 52 18.15x 0 0.00x 13 6.15x
9766 19389 64 55 19.32x 0 0.00x 9 3.82x
9854 15200 81 72 34.57x 1 0.17x 8 5.45x
9883 25159 49 36 12.63x 0 0.00x 13 4.99x
10039 16995 74 65 28.06x 0 0.00x 9 5.35x
10242 20958 60 47 18.34x 0 0.00x 13 5.87x
11182 24841 50 37 10.84x 0 0.00x 13 3.99x
11460 18479 67 55 22.47x 0 0.00x 12 5.75x
12501 17731 70 62 27.50x 0 0.00x 8 4.56x
12632 24878 50 37 10.34x 0 0.00x 13 4.39x
13146 21807 58 51 15.77x 0 0.00x 7 3.28x
13168 18366 68 56 18.98x 0 0.00x 12 4.90x
13342 22204 56 48 14.11x 0 0.00x 8 3.66x
13359 27442 45 34 7.77x 0 0.00x 11 3.15x
14068 32635 38 28 5.71x 0 0.00x 10 3.41x
14359 27475 45 33 8.20x 0 0.00x 12 3.88x
14428 21324 58 45 15.46x 0 0.00x 13 5.09x
14467 14991 82 52 36.51x 25 13.85x 5 4.04x
14803 20995 60 51 17.63x 0 0.00x 9 3.81x
14907 18826 67 52 24.26x 0 0.00x 15 6.30x
15409 26960 46 36 10.06x 0 0.00x 10 4.50x
15728 24197 51 37 10.68x 0 0.00x 14 5.08x
15761 23690 52 44 16.14x 0 0.00x 8 3.96x
15762 28046 44 30 8.88x 0 0.00x 14 5.83x
15907 19022 65 50 20.16x 0 0.00x 15 6.97x
16088 20356 61 48 17.06x 0 0.00x 13 5.67x
16181 25006 50 40 11.35x 0 0.00x 10 3.35x
16818 15683 79 73 31.78x 0 0.00x 6 3.52x
16892 25123 50 40 13.46x 0 0.00x 10 6.40x
16946 22581 55 50 15.81x 0 0.00x 5 2.48x
17033 24986 50 41 15.72x 0 0.00x 9 3.51x
17398 14749 84 63 36.63x 16 5.20x 5 3.62x
17687 14589 85 74 36.18x 3 0.72x 8 4.04x
17860 25479 49 35 7.29x 0 0.00x 14 3.90x
17993 19083 65 61 25.76x 0 0.00x 4 2.57x
18124 26470 47 37 9.75x 0 0.00x 10 3.00x
18289 21714 57 46 15.49x 0 0.00x 11 5.01x
18425 16226 77 71 32.04x 0 0.00x 6 3.33x
18447 25862 48 42 10.44x 0 0.00x 6 2.44x
18529 25248 50 44 17.24x 0 0.00x 6 2.98x
19417 13868 90 74 37.47x 12 3.99x 4 2.54x
19526 27331 45 38 10.99x 0 0.00x 7 3.63x
19557 23360 54 48 14.69x 0 0.00x 6 2.50x
20041 19591 64 54 19.88x 0 0.00x 10 4.50x
21091 21719 57 50 12.41x 0 0.00x 7 3.22x
21399 22780 54 43 12.11x 0 0.00x 11 4.92x
21965 28702 44 37 8.68x 0 0.00x 7 2.66x
22117 25786 48 41 8.05x 0 0.00x 7 2.39x
22312 22585 55 44 14.72x 0 0.00x 11 4.78x
22969 15937 79 61 25.94x 0 0.00x 18 9.43x
23146 25705 49 40 12.90x 0 0.00x 9 4.53x
23275 18625 66 60 27.53x 0 0.00x 6 3.66x
23583 19631 63 51 19.44x 0 0.00x 12 5.51x
24238 28241 44 38 9.12x 0 0.00x 6 2.32x
24307 16234 76 72 36.24x 0 0.00x 4 2.67x
24376 13453 93 86 36.46x 1 0.08x 6 3.55x
25454 29390 43 31 6.55x 0 0.00x 12 3.85x
25472 29000 43 25 7.84x 0 0.00x 18 6.53x
25509 15142 82 69 32.83x 1 0.23x 12 7.27x
25872 15986 79 74 30.38x 0 0.00x 5 3.07x
25935 22520 55 48 13.77x 0 0.00x 7 2.68x
26447 13882 91 88 24.60x 0 0.00x 3 2.86x
26494 22332 56 47 15.28x 0 0.00x 9 3.35x
26686 21330 59 50 18.22x 0 0.00x 9 4.09x
27383 26922 46 37 15.36x 0 0.00x 9 4.60x
27428 27396 45 31 8.26x 0 0.00x 14 5.36x
27757 25750 48 35 11.18x 0 0.00x 13 5.94x
27882 18952 66 54 20.63x 0 0.00x 12 6.08x
27890 24749 51 40 10.91x 0 0.00x 11 4.07x
27935 35895 35 26 8.69x 0 0.00x 9 3.26x
28193 28879 43 29 5.72x 0 0.00x 14 /scratch/slurm/SlurmdSpoolDir/x1000c1s3b1n1/job3127620/slurm_script: line 93: 2270585 Killed $bin/utgcns -R ../0.070.ctgStore/partition.$jobid -T ../0.070.ctgStore 1 -P $jobid -O ./ctgcns/$jobid.cns.WORKING -maxcoverage 40 -e 0.070 -pbdagcon -edlib -threads 8
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=3127620.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
As I see, this is a kind of memory issue. Maybe, the memory is not enough?
Correct, slurm killed it for exceeding the requested memory limit. Canu has computed that it requires only 1.25GB memory per job. This is very small, but plausible given the short contigs. You can increase the memory limit with Canu option cnsMemory=8g
(or more). No need to clean up anything, just add the option and restart.
However, this assembly looks quite poor (unless I'm missing something). The contigs listed above are very short, yet there are 2.2 million contigs. Computing 2.2 million contigs * 20 Kbp average contig size = 4.4 Gbp assembled size. Is this expected?
Thanks @brianwalenz for your answer. I add cnsMemory=8g
to my canu script and now it looks good.
The assembled genome size is 4.4 Gbp, so it is correct. Yes, the contigs are very short but I do not know why. This is a diploid, very repetitive plant genome. Can I optimise somehow the canu
options for better assembly? I tried change the correctedErrorRate
between 0.045 and 0.095. Now, I do the 0.070.
If the issue is repeats, I don't think the correctedErrorRate would help, it would only potentially collapse more diverse haplotypes. What kind of data and coverage do you have? Post the report file canu outputs from its run.
Thanky @skoren for your answer. I have got a diploid plant genome with about 30x coverage from PacBio. The expected genome size is 4.4 Gbp. I attached the report file from canu
. 0.070.report.txt
Also, I have got 30x coverage from 10x Genomics data, 150x coverage from Illumiona PE150 data, and 100x coverage from BGI PE150 data. After canu, I would like to use this datasets for scaffolding.
I didn't realize your coverage is that low. From the report, the reads are also very short, almost all of them are 4-8kb. These are CLR reads and not HiFi? With that low of a coverage and short reads, I'm not sure you can do much to improve the assembly. You can use the FAQ settings for low coverage data but I'm assuming you already are. I would recommend getting HiFi for this genome if possible, you'd likely get a much better assembly than what you have currently.
Idle