pipeline:makeReport (1)` terminated with an error exit status (1)

Question

pipeline:makeReport (1)` terminated with an error exit status (1)

Closed this issue 5 months ago · 12 comments

cihaterdogan commented 7 months ago

Operating System

Windows 10

Other Linux

Red Hat Enterprise 8.x

Workflow Version

v1.0.1

Workflow Execution

EPI2ME Desktop application

EPI2ME Version

No response

CLI command run

nextflow run epi2me-labs/wf-single-cell -resume
-r prerelease
-w ${OUTPUT}/workspace
-profile singularity
--max_threads 32
--fastq Chrm_290_8/
--kit_name 3prime
--kit_version v3
--expected_cells 45000
--ref_genome_dir refdata-gex-GRCh38-2020-A/
--out_dir ${OUTPUT}
--merge_bam

Workflow Execution - CLI Execution Profile

singularity

What happened?

I was running the wf-single-cell pipeline as a job (with Slurm) on my institution's clusters with the following input:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=0-72:00:00
#SBATCH --mem=500gb

module load singularity/3.7.2
module load java/15.0.2

OUTPUT=Chrm_290_8_results

./nextflow run epi2me-labs/wf-single-cell -resume
-r prerelease
-w ${OUTPUT}/workspace
-profile singularity
--max_threads 32
--fastq Chrm_290_8/
--kit_name 3prime
--kit_version v3
--expected_cells 45000
--ref_genome_dir refdata-gex-GRCh38-2020-A/
--out_dir ${OUTPUT}
--merge_bam

And the pipeline got terminated with the following error:

executor > local (350)
[98/dc1b6d] process > fastcat (1) [100%] 1 of 1 ✔
[94/48090a] process > parse_kit_metadata (1) [100%] 1 of 1 ✔
[15/f08730] process > pipeline:getVersions [100%] 1 of 1 ✔
[ae/339199] process > pipeline:getParams [100%] 1 of 1 ✔
[90/96c5e1] process > pipeline:summariseCatChunkR... [100%] 1 of 1 ✔
[4c/238272] process > pipeline:stranding:call_ada... [100%] 32 of 32 ✔
[a5/96ff1a] process > pipeline:stranding:combine_... [100%] 1 of 1 ✔
[ec/40172b] process > pipeline:stranding:summariz... [100%] 1 of 1 ✔
[9a/648471] process > pipeline:align:call_paftools [100%] 1 of 1 ✔
[ef/a663e3] process > pipeline:align:get_chrom_sizes [100%] 1 of 1 ✔
[2e/4233a2] process > pipeline:align:align_to_ref... [100%] 1 of 1 ✔
[2c/f7fd41] process > pipeline:process_bams:split... [100%] 1 of 1 ✔
[71/ed8628] process > pipeline:process_bams:get_c... [100%] 1 of 1 ✔
[ba/565adc] process > pipeline:process_bams:extra... [100%] 40 of 40 ✔
[b7/a15deb] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[7f/2fc59c] process > pipeline:process_bams:gener... [100%] 1 of 1 ✔
[5a/a50ffd] process > pipeline:process_bams:assig... [100%] 40 of 40 ✔
[93/68d179] process > pipeline:process_bams:strin... [100%] 40 of 40 ✔
[00/88b60f] process > pipeline:process_bams:align... [100%] 40 of 40 ✔
[2e/22b70a] process > pipeline:process_bams:assig... [100%] 40 of 40 ✔
[ff/8a9950] process > pipeline:process_bams:clust... [100%] 40 of 40 ✔
[85/6bc2ec] process > pipeline:process_bams:tag_b... [100%] 40 of 40 ✔
[5f/7c6266] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[5c/a22c96] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[de/363000] process > pipeline:process_bams:umi_g... [100%] 1 of 1 ✔
[a1/55bd6e] process > pipeline:process_bams:const... [100%] 1 of 1 ✔
[a3/725e61] process > pipeline:process_bams:proce... [100%] 1 of 1 ✔
[08/cdab80] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[29/e613f4] process > pipeline:process_bams:pack_... [100%] 1 of 1 ✔
[92/50195f] process > pipeline:prepare_report_dat... [100%] 1 of 1 ✔
[1c/460092] process > pipeline:makeReport (1) [ 0%] 0 of 1
[aa/fde863] process > output (13) [100%] 15 of 15 ✔
[- ] process > output_report -
ERROR ~ Error executing process > 'pipeline:makeReport (1)'

Caused by:
Process pipeline:makeReport (1) terminated with an error exit status (1)

Command executed:

workflow-glue report --read_stats read_stats.csv --params params.csv --versions versions --survival survival.tsv --wf_summary wf_summary.tsv --output wf-single-cell-report.html --umap_dirs Chrm_290_8_umap --images images_Chrm_290_8 --umap_genes umap_plot_genes.csv

Command exit status:
1

Command output:
(empty)

Command error:
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
[05:13:35 - workflow_glue] Starting entrypoint.
[05:13:35 - Report ] Building report
Traceback (most recent call last):
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in
cli()
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 72, in cli
args.func(args)
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/report.py", line 176, in main
SeqSummary(args.read_stats)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 47, in init
df_all = load_stats(seq_summary, format='fastcat')
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 375, in load_stats
df['start_time'] = df.start_time.dt.tz_localize(None)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in getattr
return object.getattribute(self, name)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/accessor.py", line 224, in get
accessor_obj = self._accessor(obj)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/accessors.py", line 580, in new
raise AttributeError("Can only use .dt accessor with datetimelike values")
AttributeError: Can only use .dt accessor with datetimelike values

Work dir:
/N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

The wf-single cell pipeline worked smoothly on different data sets. Do you have any suggestions?

Relevant log output

Caused by:
  Process `pipeline:makeReport (1)` terminated with an error exit status (1)

Command executed:

  workflow-glue report         --read_stats read_stats.csv         --params params.csv         --versions versions         --survival survival.tsv         --wf_summary wf_summary.tsv         --output wf-single-cell-report.html         --umap_dirs Chrm_290_8_umap         --images images_Chrm_290_8         --umap_genes umap_plot_genes.csv

Command exit status:
  1

Command output:
  (empty)

Command error:
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
    @numba.jit()
  [05:13:35 - workflow_glue] Starting entrypoint.
  [05:13:35 - Report    ] Building report
  Traceback (most recent call last):
    File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in <module>
      cli()
    File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 72, in cli
      args.func(args)
    File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/report.py", line 176, in main
      SeqSummary(args.read_stats)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 47, in __init__
      df_all = load_stats(seq_summary, format='fastcat')
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 375, in load_stats
      df['start_time'] = df.start_time.dt.tz_localize(None)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in __getattr__
      return object.__getattribute__(self, name)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/accessor.py", line 224, in __get__
      accessor_obj = self._accessor(obj)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/accessors.py", line 580, in __new__
      raise AttributeError("Can only use .dt accessor with datetimelike values")
  AttributeError: Can only use .dt accessor with datetimelike values

Work dir:
  /N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Jan-19 05:17:11.536 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-19 05:17:11.538 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `pipeline:makeReport (1)` terminated with an error exit status (1)
Jan-19 05:17:12.101 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-19 05:17:12.102 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jan-19 05:17:12.241 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=349; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=21d 18h 44m 4s; failedDuration=4m 8s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=29; peakCpus=32; peakMemory=105.3 GB; ]
Jan-19 05:17:12.241 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jan-19 05:17:12.247 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jan-19 05:17:12.806 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jan-19 05:17:13.274 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-19 05:17:13.292 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Application activity log entry

No response

Answer 1 · 2024-01-19T13:52:58.000Z

Please also find the log file here:
nextflow.log

Answer 2 · 2024-01-19T14:42:10.000Z

Hi @cihaterdogan

Sorry that you're having issues with the workflow.

Would you be able to post a few lines from the following file? /N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9/read_stats.csv

Answer 3 · 2024-01-19T15:29:49.000Z

Thank you for the quick response. These are a few lines

Answer 4 · 2024-01-19T17:07:14.000Z

Thanks. That part of the file looks OK. There's possibly a malformed start_time entry somewhere in the read stats file. Have you had this issue with any other data sets?

I will update the container with the latest version of the code that generates the stats files to see if that fixes the issue. And will get back to you.

Answer 5 · 2024-01-19T17:18:13.000Z

No, I haven't experienced this problem with other datasets. But this dataset was the largest one I have ever worked with the wf-single-cell pipeline.

Thanks.

Answer 6 · 2024-01-30T07:53:15.000Z

Hi @cihaterdogan

Sorry for the delay. It would be good if I could get hold of the whole read stats file. I've emailed you a link to share it with me, if you'er able to.

Thanks

Answer 7 · 2024-01-30T09:34:24.000Z

I have just been informed of a possibly related issue. When start_time in the read_stats.csv file contain entries with mixed time zones, an error is raised. Do you happen to have mixed timezone data in the sample in question, maybe from combining samples run at different times.

For example, if you had these two start time entries in read_stats.csv, it will fail (The second entry is offset by an hour - +01:00)
2022-06-01T19:38:06.299792+00:00
2022-02-15T21:10:45.415802+01:00

This issue will be fixed shortly.

Answer 8 · 2024-01-30T15:01:12.000Z

Hi Neil, I've started to upload the read_stats.csv file to the OneDrive folder. The file size is more than 18 GB. Actually, to fix the issue I removed the time information from the start_time column and re-run the wf-singlecell pipeline last week. Finally, the pipeline was completed successfully. Thank you for the information. My Best, Cihat

…

On Tue, Jan 30, 2024 at 4:34 AM Neil Horner ***@***.***> wrote: I have just been informed of a possibly related issue. When start_time in the read_stats.csv file contain entries with mixed time zones, an error is raised. Do you happen to have mixed timezone data in the sample in question, maybe from combining samples run at different times. For example, if you had these two start time entries in read_stats.csv, it will fail (The second entry is offset by an hour - +01:00) 2022-06-01T19:38:06.299792+00:00 2022-02-15T21:10:45.415802+01:00 This issue will be fixed shortly. — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH5KMENA7QP4WQ6OMYNACMDYRC5CXAVCNFSM6AAAAABCB7OWNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJWGQZDONRTGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Cihat ERDOĞAN

Answer 9 · 2024-01-30T15:02:56.000Z

cihaterdogan commented 7 months ago

Answer 10 · 2024-01-30T15:41:45.000Z

Hi Chiat, thanks for the update. I'll let you know when we have a fix in place for this.

Neil

Answer 11 · 2024-01-30T16:41:09.000Z

Sounds good, thanks!

…

On Tue, Jan 30, 2024 at 10:41 AM Neil Horner ***@***.***> wrote: Hi Chiat, thanks for the update. I'll let you know when we have a fix in place for this. Neil — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH5KMEMZN6DAA6KZEGI7R4TYREIEJAVCNFSM6AAAAABCB7OWNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGI3TGOJWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 12 · 2024-03-13T15:39:28.000Z

Hi @cihaterdogan this issue should hopefully be fixed in v1.1.0. Will close this ticket now, thanks again for you help on this.