Error executing process > 'parse_kit_metadata (1)' with singularity

Question

Error executing process > 'parse_kit_metadata (1)' with singularity

Closed this issue 3 months ago · 5 comments

ktpolanski commented 5 months ago

Operating System

Other Linux (please specify below)

Other Linux

18.04.4

Workflow Version

v1.1.0

Workflow Execution

Command line

EPI2ME Version

No response

CLI command run

~/nextflow-23.12.0-edge-all run epi2me-labs/wf-single-cell \
    --fastq fastq/ \
    --kit_name multiome \
    --kit_version v1 \
    --expected_cells 5000 \
    --ref_genome_dir /home/ubuntu/cellranger/GRCh38-2020-A/ \
    --sample $SAMPLE \
    --max_threads 20 \
    -c openstack.cfg \
    -profile singularity \
    -resume

Workflow Execution - CLI Execution Profile

singularity

What happened?

I'm trying to use the singularity container on the off chance that somehow fixes #81 (which I don't expect it to, but figured I should give it a chance). Instead I explode very early on in the kit metadata parsing.

I encounter the same problem when trying to do the demo:

~/nextflow-23.12.0-edge-all run epi2me-labs/wf-single-cell \
    --fastq wf-single-cell-demo/chr17.fq.gz \
    --kit_name 3prime \
    --kit_version v3 \
    --expected_cells 100 \
    --ref_genome_dir wf-single-cell-demo/ \
    --plot_umaps \
    -profile singularity

I'm on Singularity 3.6.3 in case that matters.

Relevant log output

[14:41:15 - matplotlib] Matplotlib created a temporary cache directory at /tmp/matplotlib-r1ta80n4 because the default path (/home/ubuntu/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Fontconfig error: No writable cache directories
[14:41:16 - matplotlib.font_manager] generated new fontManager
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
Traceback (most recent call last):
  File "/home/ubuntu/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in <module>
    cli()
  File "/home/ubuntu/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 61, in cli
    f'{_package_name}.{comp}' for comp in get_components()]
  File "/home/ubuntu/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 26, in get_components
    mod = importlib.import_module(f"{_package_name}.{name}")
  File "/home/epi2melabs/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/umap_reduce.py", line 7, in <module>
    import umap
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/__init__.py", line 2, in <module>
    from .umap_ import UMAP
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py", line 41, in <module>
    from umap.layouts import (
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py", line 40, in <module>
    def rdist(x, y):
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/decorators.py", line 234, in wrapper
    disp.enable_caching()
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/dispatcher.py", line 863, in enable_caching
    self._cache = FunctionCache(self.py_func)
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 601, in __init__
    self._impl = self._impl_class(py_func)
  File "/home/epi2melabs/conda/lib/python3.8/site-packages/numba/core/caching.py", line 337, in __init__
    raise RuntimeError("cannot cache function %r: no locator available "
RuntimeError: cannot cache function 'rdist': no locator available for file '/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py'

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

no

Other demo data information

No response

Answer 1 · 2024-03-22T16:04:07.000Z

For whatever reason I'm not getting the error while quickly checking the demo data on a cluster head node with nextflow-23.04.1-all and singularity 3.11.5. Think it's also a Bionic Ubuntu so it's not that. Interesting.

Answer 2 · 2024-03-26T18:23:25.000Z

Hi @ktpolanski,

I would stay clear of the edge releases of Nextflow in the first instance, and stick with one of the longer term support versions. I don't know that I can comment on that error internal to the umap library.

One thing to note however is that the README is misleading. The correct command to use with the demo data is:

  nextflow run epi2me-labs/wf-single-cell \
	--expected_cells 100 \
	--fastq 'wf-single-cell-demo/chr17.fq.gz' \
	--kit_name '3prime' \
	--kit_version 'v3' \
	--ref_genome_dir 'wf-single-cell-demo' \
	--umap_plot_genes 'wf-single-cell-demo/umap_plot_genes.csv'

I apologise the README is incorrect, updating that is currently performed manually. The text if you run

nextflow run epi2me-labs/wf-single-cell -r v1.1.0 --help

is however maintained automatically and in line with our internal automated tests which ensure the demo does successfully complete.

Answer 3 · 2024-05-02T09:46:04.000Z

Hi @cjw85,

Running the command you posted, I get the same error with nextflow version 23.10.1 and wf-single-cell v1.1.0 with -profile singularity.
RuntimeError: cannot cache function 'rdist': no locator available for file '/home/epi2melabs/conda/lib/python3.8/site-packages/umap/layouts.py'

Any thoughts?
Best,
Barbara

Answer 4 · 2024-05-08T17:58:31.000Z

We've seen this error only whilst using singularity. A solution is to provide a configuration file to Nextflow with the following:

 profiles {
     singularity {
         singularity {
             enabled = true
             autoMounts = true
             runOptions = '--writable-tmpfs'
         }
     }
 }

env {
    NUMBA_CACHE_DIR='/path/to/use/for/cache'
}

Not the runOptions declaration here which is not in the workflows included config. You will need to provide a path in the env section.

Answer 5 · 2024-05-10T08:24:35.000Z

Dear @cjw85,

this solved the issue for me.

Many thanks!
Barbara