Assembly with example data on slurm failed: {Node(0-rawreads/build)}
Closed this issue · 6 comments
sadikmu commented
Any feedback please?
I have been struggling for a while to setup Falcon (falcon-kit 1.8.1) on slum to get it run example F1_bull_test data before using Falcon configuration for my Pacbio data.
I have got the F1_bull_test data from here and sadly I have not succeeded it getting it successfully assembly the this F1_bull_test data.
Can you please help me?
Here is the detail below:
pb-assembly suite installed in a separate an environment called 'falcon'
conda create -n falcon
source activate falcon
conda install pb-assembly
Lists of packages within falcon environment
# packages in environment at /home/miniconda3/envs/falcon:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_llvm conda-forge
bedtools 2.29.2 hc088bd4_0 bioconda
blas 1.0 mkl
blasr 5.3.3 h707fff8_0 bioconda
blasr_libcpp 5.3.3 h707fff8_0 bioconda
bwa 0.7.17 hed695b0_7 bioconda
bzip2 1.0.8 h516909a_2 conda-forge
ca-certificates 2020.1.1 0 anaconda
certifi 2020.4.5.1 py37_0 anaconda
curl 7.69.1 hbc83047_0
decorator 4.4.2 py_0 conda-forge
falcon-kit 1.8.1 pypi_0 pypi
falcon-phase 1.2.0 pypi_0 pypi
falcon-unzip 1.3.7 pypi_0 pypi
future 0.18.2 py37hc8dfbb8_1 conda-forge
hdf5 1.10.4 nompi_h3c11f04_1106 conda-forge
htslib 1.9 ha228f0b_7 bioconda
intel-openmp 2020.0 166 anaconda
krb5 1.17.1 h2fd8d38_0 conda-forge
ld_impl_linux-64 2.34 h53a641e_0 conda-forge
libblas 3.8.0 15_mkl conda-forge
libcblas 3.8.0 15_mkl conda-forge
libcurl 7.69.1 h20c2e04_0
libdeflate 1.0 h14c3975_1 bioconda
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.2.0 h24d8f2e_2 conda-forge
libgfortran-ng 7.3.0 hdf63c60_5 conda-forge
libgomp 9.2.0 h24d8f2e_2 conda-forge
liblapack 3.8.0 15_mkl conda-forge
libopenblas 0.3.9 h5ec1e0e_0 conda-forge
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge
llvm-openmp 10.0.0 hc9558a2_0 conda-forge
minimap2 2.17 h8b12597_1 bioconda
mkl 2020.0 166 conda-forge
mkl-service 2.3.0 py37he904b0f_0
mkl_fft 1.0.15 py37ha843d7b_0
mkl_random 1.1.0 py37hd6b4f25_0
mummer4 4.0.0beta2 pl526he1b5a44_5 bioconda
ncurses 6.1 hf484d3e_1002 conda-forge
networkx 2.4 py_1 conda-forge
nim-falcon 1.1.0 h2f8a211_0 bioconda
numpy 1.18.1 py37h8960a57_1 conda-forge
numpy-base 1.18.1 py37hde5b4d6_1
openssl 1.1.1 h7b6447c_0 anaconda
pb-assembly 0.0.8 0 bioconda
pb-dazzler 0.0.1 h516909a_0 bioconda
pb-falcon 2.2.4 py37hed50d52_0 bioconda
pb-falcon-phase 0.1.0 h3889886_0 bioconda
pbbam 0.23.0 h88345a9_0 bioconda
pbgcpp 1.9.0 0 bioconda
pbmm2 1.2.1 h56fc30b_0 bioconda
pcre 8.44 he1b5a44_0 conda-forge
perl 5.26.2 h516909a_1006 conda-forge
pip 20.0.2 py_2 conda-forge
pysam 0.15.3 py37hda2845c_1 bioconda
python 3.7.6 h0371630_2
python-edlib 1.3.8.post1 py37hc9558a2_0 bioconda
python-intervaltree 3.0.2 py_0 bioconda
python-msgpack 0.6.1 py37hc9558a2_1 bioconda
python-sortedcontainers 2.1.0 py_0 bioconda
python_abi 3.7 1_cp37m conda-forge
racon 1.4.12 he513fc3_0 bioconda
readline 7.0 h7b6447c_5
samtools 1.9 h10a08f8_12 bioconda
setuptools 46.1.3 py37hc8dfbb8_0 conda-forge
six 1.14.0 py37_0
sqlite 3.31.1 h7b6447c_0
tk 8.6.10 hed695b0_0 conda-forge
wheel 0.34.2 py_1 conda-forge
xz 5.2.5 h516909a_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
Contents of fc_run.cfg
[job.defaults]
JOB_QUEUE = bigmem
MB = 30000
NPROC = 6
job_type=slurm
submit = srun --wait=0 -p myqueue \
-J ${JOB_NAME} \
-o ${JOB_STDOUT} \
-e ${JOB_STDERR} \
--mem-per-cpu=${MB}M \
--cpus-per-task=${NPROC} \
${JOB_SCRIPT}
"${CMD}"
[General]
input_type = raw
#input_type = preads
input_fofn = subreads.fasta.fofn
# The length cutoff used for seed reads used for initial mapping
genome_size=20000000
seed_coverage = 40
length_cutoff = -1
# The length cutoff for preads
length_cutoff_pr = 12000
falcon_greedy = False
falcon_sense_greedy=False
# overlapping options for Daligner
pa_daligner_option = -e0.76 -l1200 -k18 -h480 -w8 -s100
ovlp_daligner_option = -k24 -h480 -e.95 -l1800 -s100
pa_HPCdaligner_option = -v -B128 -M24
ovlp_HPCdaligner_option = -v -B128 -M24
pa_HPCTANmask_option = -k18 -h480 -w8 -e.8 -s100
pa_HPCREPmask_option = -k18 -h480 -w8 -e.8 -s100
#pa_REPmask_code=1,20;10,15;50,10
pa_DBsplit_option = -x500 -s200
ovlp_DBsplit_option = -s400
# error correction consensus option
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24
# overlap filtering options
overlap_filtering_setting = --max_diff 120 --max_cov 120 --min_cov 4 --n_core 24
[job.step.da]
NPROC=4
MB=32000
njobs=300
[job.step.la]
NPROC=8
MB=64000
njobs=200
[job.step.cns]
NPROC=8
MB=64000
njobs=200
[job.step.pda]
NPROC=8
MB=64000
njobs=200
[job.step.pla]
NPROC=4
MB=32000
njobs=300
[job.step.asm]
NPROC=24
MB=192000
njobs=1
contents of fc.246025.err
`"falcon_sense_option=--output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24". Those should be "-", as in
"falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24". Auto-replaced.
[WARNING]Option contains flags with "_":
"overlap_filtering_setting=--max_diff 120 --max_cov 120 --min_cov 4 --n_core 24". Those should be "-", as in
"overlap_filtering_setting=--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24". Auto-replaced.
[WARNING]Unexpected keys in input config: {'falcon_greedy'}
[WARNING]Please supply a default for "njobs" (aka concurrency) in section [job.defaults]. For now, we will use 8
[INFO]cfg=
{
"General": {
"LA4Falcon_preload": false,
"avoid_text_file_busy": true,
"bestn": 12,
"dazcon": false,
"falcon_greedy": "False",
"falcon_sense_greedy": false,
"falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
"falcon_sense_skip_contained": false,
"fc_ovlp_to_graph_option": " --min-len 12000",
"genome_size": "20000000",
"input_fofn": "subreads.fasta.fofn",
"input_type": "raw",
"length_cutoff": "-1",
"length_cutoff_pr": "12000",
"overlap_filtering_setting": "--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24",
"ovlp_DBdust_option": "",
"ovlp_DBsplit_option": "-s400",
"ovlp_HPCdaligner_option": "-v -B128 -M24",
"ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
"pa_DBdust_option": "",
"pa_DBsplit_option": "-x500 -s200",
"pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCdaligner_option": "-v -B128 -M24",
"pa_REPmask_code": "0,300/0,300/0,300",
"pa_daligner_option": "-e0.76 -l1200 -k18 -h480 -w8 -s100",
"pa_dazcon_option": "-j 4 -x -l 500",
"pa_fasta_filter_option": "streamed-internal-median",
"pa_subsample_coverage": 0,
"pa_subsample_random_seed": 12345,
"pa_subsample_strategy": "random",
"seed_coverage": "40",
"skip_checks": false,
"target": "assembly"
},
"job.defaults": {
"JOB_QUEUE": "bigmem",
"MB": "30000",
"NPROC": "6",
"job_type": "slurm",
"njobs": 8,
"pwatcher_type": "fs_based",
"submit": "srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem
-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n\"${CMD}\"",
"use_tmpdir": false
},
"job.step.asm": {
"MB": "192000",
"NPROC": "24",
"njobs": "1"
},
"job.step.cns": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
},
"job.step.da": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
},
"job.step.dust": {},
"job.step.la": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pda": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pla": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
}
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py'>
[INFO]job_type='slurm', (default)job_defaults={'JOB_QUEUE': 'bigmem', 'MB': '30000', 'NPROC': '6', 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO] starting job Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}) w/ job_type=SLURM
[INFO]$ srun --wait=0 -p myqueue \
-J Pac163ff68e5c9f \
-o stdout \
-e stderr \
--mem-per-cpu=4000M \
--cpus-per-task=1 \
/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash
"/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash" >
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobSlurm(MetaJob(job=Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}), lang_exe='/bin/bash'))
Traceback (most recent call last):
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 474, in cmd_run
state.submit_background(bjob)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 121, in submit_background
bjob.submit(self, exe, script_fn) # Can raise
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 325, in submit
self.submit_capture = capture(sge_cmd)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/io.py", line 50, in capture
raise Exception(msg)
Exception: 126 <- 'srun --wait=0 -p myqueue \\\n-J Pac163ff68e5c9f \\\n-o stdout \\\n-e stderr \\\n--mem-per-cpu=4000M \\\n--cpus-per-task=1 \\\n/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash\n"/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash"':
srun: error: cnode84: tasks 0-1: Exited with exit code 13
srun: Terminating job step 246025.0
/bin/sh: line 7: /home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash: Permission denied
[ERROR]Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
[INFO]CALL:
scancel -n Pac163ff68e5c9f
Traceback (most recent call last):
File "/home/miniconda3/envs/falcon/bin/fc_run", line 11, in <module>
load_entry_point('falcon-kit==1.8.1', 'console_scripts', 'fc_run')()
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
main1(argv[0], args.config, args.logger)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
input_fofn_fn=input_fofn_fn,
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
wf.refreshTargets()
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
self._refreshTargets(updateFreq, exitOnFailure)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 323, in _refreshTargets
raise Exception(msg)
Exception: Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
`
Thank you,
sadikmu commented
Thanks Bas Tolhuis from from PacBio techsupport @PacificBiosciences
I got feedback to check 0-rawreads/build for stderr file and the following error message noticed
2020-04-13 11:31:50,050 - root - ERROR - Was waiting for '../../subreads.fasta.fofn'
Checking my files again I found that subreads.fasta.fofn file was missing.
After creating the subreads.fasta.fofn file containing absolute path to the input tutorial F1_bull_test.subreads.fasta.gz file job now submitted to slurm. Progress will be updated.
sadikmu commented
resubmitted job failed again.
I couldn’t see stderr file in … 0-rawreads/build path.
List of files in in …0-rawreads/build
lrwxrwxrwx 1 sadikm sadikm 108 Apr 20 23:27 pwatcher.dir -> /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/jobs/Pac163ff68e5c9f
-rw-rw-r-- 1 sadikm sadikm 269 Apr 20 23:27 run.sh
-rw-rw-r-- 1 sadikm sadikm 610 Apr 20 23:27 task.sh
-rw-rw-r-- 1 sadikm sadikm 368 Apr 20 23:27 task.json
-rw-rw-r-- 1 sadik sadikm 296 Apr 20 23:27 template.sh
contents of fc.2467347.err
falcon-kit 1.8.1 (pip thinks "falcon-kit 1.8.1")
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data >
[INFO]Apparently '/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data' is not in lustre filesystem, which is fine.
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]Option contains flags with "_":
"falcon_sense_option=--output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24". Those should be "-", as in
"falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24". Auto-replaced.
[WARNING]Option contains flags with "_":
"overlap_filtering_setting=--max_diff 120 --max_cov 120 --min_cov 4 --n_core 24". Those should be "-", as in
"overlap_filtering_setting=--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24". Auto-replaced.
[WARNING]Unexpected keys in input config: {'falcon_greedy'}
[WARNING]Please supply a default for "njobs" (aka concurrency) in section [job.defaults]. For now, we will use 8
[INFO]cfg=
{
"General": {
"LA4Falcon_preload": false,
"avoid_text_file_busy": true,
"bestn": 12,
"dazcon": false,
"falcon_greedy": "False",
"falcon_sense_greedy": false,
"falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
"falcon_sense_skip_contained": false,
"fc_ovlp_to_graph_option": " --min-len 12000",
"genome_size": "20000000",
"input_fofn": "subreads.fasta.fofn",
"input_type": "raw",
"length_cutoff": "-1",
"length_cutoff_pr": "12000",
"overlap_filtering_setting": "--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24",
"ovlp_DBdust_option": "",
"ovlp_DBsplit_option": "-s400",
"ovlp_HPCdaligner_option": "-v -B128 -M24",
"ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
"pa_DBdust_option": "",
"pa_DBsplit_option": "-x500 -s200",
"pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCdaligner_option": "-v -B128 -M24",
"pa_REPmask_code": "0,300/0,300/0,300",
"pa_daligner_option": "-e0.76 -l1200 -k18 -h480 -w8 -s100",
"pa_dazcon_option": "-j 4 -x -l 500",
"pa_fasta_filter_option": "streamed-internal-median",
"pa_subsample_coverage": 0,
"pa_subsample_random_seed": 12345,
"pa_subsample_strategy": "random",
"seed_coverage": "40",
"skip_checks": false,
"target": "assembly"
},
"job.defaults": {
"JOB_QUEUE": "bigmem",
"MB": "30000",
"NPROC": "6",
"job_type": "slurm",
"njobs": 8,
"pwatcher_type": "fs_based",
"submit": "srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n\"${CMD}\"",
"use_tmpdir": false
},
"job.step.asm": {
"MB": "192000",
"NPROC": "24",
"njobs": "1"
},
"job.step.cns": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.da": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
},
"job.step.dust": {},
"job.step.la": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pda": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pla": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
}
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py'>
[INFO]job_type='slurm', (default)job_defaults={'JOB_QUEUE': 'bigmem', 'MB': '30000', 'NPROC': '6', 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO] starting job Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}) w/ job_type=SLURM
[INFO]$ srun --wait=0 -p myqueue \
-J Pac163ff68e5c9f \
-o stdout \
-e stderr \
--mem-per-cpu=4000M \
--cpus-per-task=1 \
/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash
"/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash" >
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobSlurm(MetaJob(job=Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue \\\n-J ${JOB_NAME} \\\n-o ${JOB_STDOUT} \\\n-e ${JOB_STDERR} \\\n--mem-per-cpu=${MB}M \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}), lang_exe='/bin/bash'))
Traceback (most recent call last):
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 474, in cmd_run
state.submit_background(bjob)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 121, in submit_background
bjob.submit(self, exe, script_fn) # Can raise
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 325, in submit
self.submit_capture = capture(sge_cmd)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/io.py", line 50, in capture
raise Exception(msg)
Exception: 126 <- 'srun --wait=0 -p myqueue \\\n-J Pac163ff68e5c9f \\\n-o stdout \\\n-e stderr \\\n--mem-per-cpu=4000M \\\n--cpus-per-task=1 \\\n/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash\n"/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash"':
srun: error: cnode10: tasks 0-1: Exited with exit code 13
srun: Terminating job step 247347.0
/bin/sh: line 7: /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash: Permission denied
[ERROR]Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
[INFO]CALL:
scancel -n Pac163ff68e5c9f
Traceback (most recent call last):
File "/home/miniconda3/envs/falcon/bin/fc_run", line 11, in <module>
load_entry_point('falcon-kit==1.8.1', 'console_scripts', 'fc_run')()
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
main1(argv[0], args.config, args.logger)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
input_fofn_fn=input_fofn_fn,
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
wf.refreshTargets()
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
self._refreshTargets(updateFreq, exitOnFailure)
File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 323, in _refreshTargets
raise Exception(msg)
Exception: Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
shannonekj commented
Hi @sadikmu this appears to be a problem I'm running into as well. I believe it is due to the permissions of the .bash scripts that are created in the mypwatcher/wrappers/ directory.
If you change the permissions of the file to be executable and then submit your run script again it will work for that particular step (pertaining to the bash script you just made into an executable). However, it seems we'll have to do this for each individual step & I'd like it to just run. Do any of the PacBio folks have any suggestions/workarounds?
sadikmu commented
hi @shannonekj,
It would be unrealistic to rerun on slurm queue every time it fails modifying in the file permission to executable which should be autogenerated executable the job is submitted.
The F1_bull_test data (~462Mb) and the fc_run.cfg script here worked successfully on local server. However, running falcon on actual data on the local sever after modifying genome size in fc_run.cfg used above just failed.
I tried to used various parameter options within fc_run.cfg including parameters in #708 -unfortunately it did not work at all and throws the same error shown below. I have been struggling on this for a while and could not figure it out.
Unable to move forward on this and highly appreciate anyones suggestion.
[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /home/assembly/falcon >
[INFO]Apparently '/home/assembly/falcon' is not in lustre filesystem, which is fine.
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]Unexpected keys in input config: set(['ovlp_concurrent_jobs', 'falcon_greedy'])
[INFO]cfg=
{
"General": {
"LA4Falcon_preload": false,
"avoid_text_file_busy": true,
"bestn": 12,
"dazcon": false,
"falcon_greedy": "False",
"falcon_sense_greedy": false,
"falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
"falcon_sense_skip_contained": false,
"fc_ovlp_to_graph_option": " --min-len 10000",
"genome_size": "570000000",
"input_fofn": "subreads.fasta.fofn",
"input_type": "raw",
"length_cutoff": "-1",
"length_cutoff_pr": "10000",
"overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 4 --n-core 24",
"ovlp_DBdust_option": "",
"ovlp_DBsplit_option": "-s400",
"ovlp_HPCdaligner_option": "-v -B128 -M24 -k24 -h600 -e.95 -l1800 -s100",
"ovlp_concurrent_jobs": "288 -s50",
"ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
"pa_DBdust_option": "",
"pa_DBsplit_option": "-x500 -s50",
"pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
"pa_HPCdaligner_option": "-v -B128 -e0.75 -M24 -l1200 -k14 -h256 -w8 -s100 -t16",
"pa_REPmask_code": "0,300/0,300/0,300",
"pa_daligner_option": "-e0.75 -l1200 -k14 -h256 -w8 -s100",
"pa_dazcon_option": "-j 4 -x -l 500",
"pa_fasta_filter_option": "streamed-internal-median",
"pa_subsample_coverage": 0,
"pa_subsample_random_seed": 12345,
"pa_subsample_strategy": "random",
"seed_coverage": "40",
"skip_checks": false,
"target": "assembly"
},
"job.defaults": {
"JOB_QUEUE": "default",
"MB": "32768",
"NPROC": "6",
"job_type": "local",
"njobs": "32",
"pwatcher_type": "blocking",
"submit": "/bin/bash -c \"${JOB_SCRIPT}\" > \"${JOB_STDOUT}\" 2> \"${JOB_STDERR}\"",
"use_tmpdir": false
},
"job.step.asm": {
"MB": "192000",
"NPROC": "24",
"njobs": "1"
},
"job.step.cns": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.da": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
},
"job.step.dust": {},
"job.step.la": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pda": {
"MB": "64000",
"NPROC": "8",
"njobs": "200"
},
"job.step.pla": {
"MB": "32000",
"NPROC": "4",
"njobs": "300"
}
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pwatcher/blocking.pyc'>
[INFO]job_type='local', (default)job_defaults={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': '32768', 'job_type': 'local', 'submit': '/bin/bash -c "${JOB_SCRIPT}" > "${JOB_STDOUT}" 2> "${JOB_STDERR}"', 'NPROC': '6', 'njobs': '32'}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 32; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO]Popen: '/bin/bash -c "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pwatcher/mains/job_start.sh" > "/home/assembly/falcon/0-rawreads/build/run-P593808baa38b0f.bash.stdout" 2> "/home/assembly/falcon/0-rawreads/build/run-P593808baa38b0f.bash.stderr"'
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO](slept for another 0.3s -- another 2 loop iterations)
[INFO](slept for another 1.2s -- another 3 loop iterations)
[INFO](slept for another 3.0s -- another 4 loop iterations)
[INFO](slept for another 6.0s -- another 5 loop iterations)
[INFO](slept for another 10.5s -- another 6 loop iterations)
[INFO](slept for another 16.8s -- another 7 loop iterations)
[INFO](slept for another 25.2s -- another 8 loop iterations)
[INFO](slept for another 36.0s -- another 9 loop iterations)
[INFO](slept for another 49.5s -- another 10 loop iterations)
[INFO](slept for another 66.0s -- another 11 loop iterations)
[INFO](slept for another 85.8s -- another 12 loop iterations)
[INFO](slept for another 109.2s -- another 13 loop iterations)
[INFO](slept for another 135.6s -- another 14 loop iterations)
[INFO](slept for another 150.0s -- another 15 loop iterations)
[ERROR]Task Node(0-rawreads/build) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
[ERROR]ready: set([])
submitted: set([])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
File "/home/sadik.conda/envs/falcon/bin/fc_run", line 11, in <module>
load_entry_point('falcon-kit==1.3.0', 'console_scripts', 'fc_run')()
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
main1(argv[0], args.config, args.logger)
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
input_fofn_fn=input_fofn_fn,
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/pype.py", line 109, in gen_parallel_tasks
wf.refreshTargets()
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
self._refreshTargets(updateFreq, exitOnFailure)
File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
pb-cdunn commented
We've been concentrating on pbipa, which now works really well if you have HiFi data (i.e. ccs reads).
Try using an absolute path for
input_fofn = subreads.fasta.fofn
I think it's having trouble finding that.