OSError: [Errno 28] No space left on device
Opened this issue · 6 comments
Ask away!
Hi i encountered a new error related to I/O that might be specific to my system and i wonder if you could help me out with the issue. Tried troubleshooting by changing SINGULARITY_CACHEDIR but didn't seem to work. I'm running it on a HPC as a SLURM job using singularity with custom config:
nextflow run epi2me-labs/wf-single-cell
-profile singularity
-r v2.0.3
--ref_genome_dir /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample1/epi2me/refdata-gex-GRCh38-2020-A
-c /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample1/epi2me/my_config.cfg
--threads 32
--fastq /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/data.dir/BM_Sample2.fastq.gz
--kit_name 3prime
--kit_version v3
--expected_cells 2000
--full_length_only True
--out_dir /ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/epi2me/output
-resume
Config file:
my_config_20240617.txt
my_config_20240617.txt
my_config_20240617.txt
Error message:
executor > local (324)
[90/929b23] process > fastcat (1) [100%] 1 of 1 ✔
[11/41ed10] process > parse_kit_metadata (1) [100%] 1 of 1 ✔
[4e/cae50f] process > pipeline:getVersions [100%] 1 of 1 ✔
[c0/3ab1a5] process > pipeline:getParams [100%] 1 of 1 ✔
[4d/31dc07] process > pipeline:preprocess:call_pa... [100%] 1 of 1 ✔
[bb/a811f1] process > pipeline:preprocess:get_chr... [100%] 1 of 1 ✔
[ae/f06a9f] process > pipeline:preprocess:build_m... [100%] 1 of 1 ✔
[65/747683] process > pipeline:preprocess:call_ad... [100%] 75 of 75 ✔
[16/975768] process > pipeline:process_bams:split... [100%] 1 of 1 ✔
[68/5203d5] process > pipeline:process_bams:gener... [100%] 1 of 1 ✔
[37/392382] process > pipeline:process_bams:assig... [100%] 75 of 75 ✔
[66/cfc009] process > pipeline:process_bams:cat_t... [100%] 1 of 1 ✔
[17/c839fd] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔
[b0/4572b7] process > pipeline:process_bams:strin... [100%] 40 of 40 ✔
[f9/124217] process > pipeline:process_bams:align... [100%] 40 of 40 ✔
[c2/8ca75b] process > pipeline:process_bams:assig... [100%] 39 of 39 ✔
[df/b2946b] process > pipeline:process_bams:creat... [100%] 39 of 39 ✔
[a1/76432a] process > pipeline:process_bams:proce... [100%] 2 of 2, failed: 2 ✘
[5b/d0a07b] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔
[40/044f2f] process > pipeline:process_bams:combi... [ 0%] 0 of 1
[45/e19f76] process > pipeline:process_bams:tag_b... [ 0%] 0 of 1
[- ] process > pipeline:process_bams:umi_g... -
[- ] process > pipeline:process_bams:pack_... -
[- ] process > pipeline:prepare_report_data -
[- ] process > pipeline:makeReport -
ERROR ~ Error executing process > 'pipeline:process_bams:process_matrix (2)'
Caused by:
Process pipeline:process_bams:process_matrix (2) terminated with an error exit status (1)
Command executed:
export NUMBA_NUM_THREADS=1
workflow-glue process_matrix inputs/matrix*.hdf --feature transcript --raw transcript_raw_feature_bc_matrix --processed transcript_processed_feature_bc_matrix --per_cell_mito transcript.expression.mito-per-cell.tsv --per_cell_expr transcript.expression.mean-per-cell.tsv --umap_tsv transcript.expression.umap.tsv --enable_filtering --min_features 200 --min_cells 3 --max_mito 20 --mito_prefixes MT- --norm_count 10000 --enable_umap --replicates 3
Command exit status:
1
Command output:
(empty)
Command error:
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
Traceback (most recent call last):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 469, in save
data_name = overloads[key]
KeyError: ('f4(f4[::1],f4[::1])', ('x86_64-unknown-linux-gnu', 'znver3', '+64bit,+adx,+aes,-amx-bf16,-amx-int8,-amx-tile,+avx,+avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512fp16,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,+bmi,+bmi2,-cldemote,+clflushopt,+clwb,+clzero,+cmov,+crc32,+cx16,+cx8,-enqcmd,+f16c,+fma,-fma4,+fsgsbase,+fxsr,-gfni,-hreset,+invpcid,-kl,-lwp,+lzcnt,+mmx,+movbe,-movdir64b,-movdiri,+mwaitx,+pclmul,-pconfig,+pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,+rdpid,+rdrnd,+rdseed,-rtm,+sahf,-serialize,-sgx,+sha,+shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+sse4a,+ssse3,-tbm,-tsxldtrk,-uintr,+vaes,+vpclmulqdq,-waitpkg,+wbnoinvd,-widekl,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'), ('0c41297f1f6a9ad7b5e76ac3455427cf5bda8e992eb29997e17af36f3f4e8fbb', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in
cli()
File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 66, in cli
components = get_components(allowed_components=[sys.argv[1]])
File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 29, in get_components
mod = importlib.import_module(f"{_package_name}.{name}")
File "/home/epi2melabs/conda/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in load_unlocked
File "", line 843, in exec_module
File "", line 219, in call_with_frames_removed
File "/home/d/dloi/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/process_matrix.py", line 8, in
import umap
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/init.py", line 2, in
from .umap import UMAP
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap.py", line 41, in
from umap.layouts import (
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py", line 40, in
def rdist(x, y):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/decorators.py", line 241, in wrapper
disp.compile(sig)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/dispatcher.py", line 972, in compile
self._cache.save_overload(sig, cres)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 652, in save_overload
self._save_overload(sig, data)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 662, in _save_overload
self._cache_file.save(key, data)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 478, in save
self._save_index(overloads)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 525, in _save_index
f.write(data)
File "/home/epi2melabs/conda/lib/python3.8/contextlib.py", line 120, in exit
next(self.gen)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 562, in _open_for_write
yield f
OSError: [Errno 28] No space left on device
Work dir:
/ceph/project/cribbslab/shared/proj048/analyses/BM_Sample2/epi2me/work/fa/7398278c39a3cfe22a95f9e307e34f
Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run
-- Check '.nextflow.log' file for details
Log file:
nextflow_20240617.log
nextflow_20240617.log
nextflow_20240617.log
Many thanks!
Hi @lscdanson,
Can you trying the advice in the Troubleshooting section of the documentation. We've often seen odd filesystem errors when using this workflow with singularity.
Hi @cjw85, thanks for your suggestion. Yes I've read through the troubleshooting guide but there's no information related to this particular error. While it might not be relevant or useful, I've submitted another job increasing the process memory:
process {
withName:process_bams:process_matrix {
memory = “1000 GB”
}
}
I will keep you posted.
Sorry, I meant simply that you should apply what is mentioned there regardless of the fact that it discusses a different issue.
@lscdanson Any updates on how that last run went?
I am encountering the same error when running with Singularity on my campus HPC. Has this bug been addressed in any recent updates? A colleague of mine was able to bypass the issue by modifying the source code to disable the UMAP feature, which I don't need for my work. It would be very helpful if there was an argument or flag that could be passed to disable the UMAP features directly. Any guidance or updates on this would be greatly appreciated. Thank you!