cannot cache function 'rdist'
Closed this issue · 1 comments
davidebolo1993 commented
Operating System
CentOS 7
Other Linux
No response
Workflow Version
v2.0.3
Workflow Execution
Command line (Cluster)
Other workflow execution
No response
EPI2ME Version
No response
CLI command run
nextflow run epi2me-labs/wf-single-cell \
-r v2.0.3 \
-w $outdir/$base/process \
-profile singularity \
-c wf-single-cell.config \
--fastq $line/$base.fastq \
--kit_name 5prime \
--kit_version v1 \
--expected_cells 20000 \
--ref_genome_dir $reference \
--out_dir $outdir/$base/result \
--max_threads 16Workflow Execution - CLI Execution Profile
None
What happened?
Hi guys,
for 4 different runs I've submitted to the cluster, the workflow ends with an error in step:
ERROR ~ Error executing process > 'pipeline:process_bams:process_matrix (2). The error looks always the same:
RuntimeError: cannot cache function 'rdist': no locator available for file '/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py'
Relevant log output
executor > local (356)
[69/3d5e8a] process > fastcat (1) [100%] 1 of 1 ✔
[4f/ec80a4] process > parse_kit_metadata (1) [100%] 1 of 1 ✔
[50/1aff65] process > pipeline:getVersions [100%] 1 of 1 ✔
[47/0b5326] process > pipeline:getParams [100%] 1 of 1 ✔
[2b/e4c6bb] process > pipeline:preprocess:call_pa... [100%] 1 of 1 ✔
[de/bbebbb] process > pipeline:preprocess:get_chr... [100%] 1 of 1 ✔
[55/e94f26] process > pipeline:preprocess:build_m... [100%] 1 of 1 ✔
[f8/b2ef22] process > pipeline:preprocess:call_ad... [100%] 91 of 91 ✔
[2f/19fb66] process > pipeline:process_bams:split... [100%] 1 of 1 ✔
[6d/6a7a47] process > pipeline:process_bams:gener... [100%] 1 of 1 ✔
[cf/d1c9a0] process > pipeline:process_bams:assig... [100%] 91 of 91 ✔
[05/452ab7] process > pipeline:process_bams:cat_t... [100%] 1 of 1 ✔
[11/5f134f] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔
[54/7b61c3] process > pipeline:process_bams:strin... [100%] 40 of 40 ✔
[b6/b5faad] process > pipeline:process_bams:align... [100%] 40 of 40 ✔
[6b/8bd78d] process > pipeline:process_bams:assig... [100%] 39 of 39 ✔
[80/1b8962] process > pipeline:process_bams:creat... [100%] 39 of 39 ✔
[20/cc3b9a] process > pipeline:process_bams:proce... [ 50%] 1 of 2, failed: 1
[4b/a9a1bd] process > pipeline:process_bams:merge... [100%] 1 of 1 ✔
[39/e70b32] process > pipeline:process_bams:combi... [ 0%] 0 of 1
[04/4a1340] process > pipeline:process_bams:tag_b... [ 0%] 0 of 1
[- ] process > pipeline:process_bams:umi_g... -
[- ] process > pipeline:process_bams:pack_... -
[- ] process > pipeline:prepare_report_data -
[- ] process > pipeline:makeReport -
ERROR ~ Error executing process > 'pipeline:process_bams:process_matrix (2)'
Caused by:
Process `pipeline:process_bams:process_matrix (2)` terminated with an error exit status (1)
Command executed:
export NUMBA_NUM_THREADS=1
workflow-glue process_matrix inputs/matrix*.hdf --feature transcript --raw transcript_raw_feature_bc_matrix --processed transcript_processed_feature_bc_matrix --per_cell_mito transcript.expression.mito-per-cell.tsv --per_cell_expr transcript.expression.mean-per-cell.tsv --umap_tsv transcript.expression.umap.tsv --enable_filtering --min_features 200 --min_cells 3 --max_mito 20 --mito_prefixes MT- --norm_count 10000 --enable_umap --replicates 3
Command exit status:
1
Command output:
(empty)
Command error:
[20:06:25 - workflow_glue] Bootstrapping CLI.
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
Traceback (most recent call last):
File "/home/davide.bolognini/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in <module>
cli()
File "/home/davide.bolognini/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 66, in cli
components = get_components(allowed_components=[sys.argv[1]])
File "/home/davide.bolognini/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 29, in get_components
mod = importlib.import_module(f"{_package_name}.{name}")
File "/home/epi2melabs/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/davide.bolognini/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/process_matrix.py", line 8, in <module>
import umap
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/__init__.py", line 2, in <module>
from .umap_ import UMAP
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py", line 41, in <module>
from umap.layouts import (
File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py", line 40, in <module>
def rdist(x, y):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/decorators.py", line 234, in wrapper
disp.enable_caching()
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/dispatcher.py", line 863, in enable_caching
self._cache = FunctionCache(self.py_func)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 601, in __init__
self._impl = self._impl_class(py_func)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 337, in __init__
raise RuntimeError("cannot cache function %r: no locator available "
RuntimeError: cannot cache function 'rdist': no locator available for file '/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py'
Work dir:
/project/immune_variation/processed_data/ont_scrna/CCH_005_1/process/20/cc3b9a2d33db926e8490d210c0178c
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
-- Check '.nextflow.log' file for details
WARN: Killing running tasks (3)
slurmstepd: error: _cgroup_procs_check: failed on path /sys/fs/cgroup/memory/slurm/uid_3722/job_12405166/step_batch/cgroup.procs: No such file or directory
slurmstepd: error: unable to read '/sys/fs/cgroup/memory/slurm/uid_3722/job_12405166/step_batch/cgroup.procs'Application activity log entry
No response
Were you able to successfully run the latest version of the workflow with the demo data?
yes
Other demo data information
No response
davidebolo1993 commented
My bad, just checked the README.