pipeline:makeReport (1)` terminated with an error exit status (1)
Closed this issue · 12 comments
Operating System
Windows 10
Other Linux
Red Hat Enterprise 8.x
Workflow Version
v1.0.1
Workflow Execution
EPI2ME Desktop application
EPI2ME Version
No response
CLI command run
nextflow run epi2me-labs/wf-single-cell -resume
-r prerelease
-w ${OUTPUT}/workspace
-profile singularity
--max_threads 32
--fastq Chrm_290_8/
--kit_name 3prime
--kit_version v3
--expected_cells 45000
--ref_genome_dir refdata-gex-GRCh38-2020-A/
--out_dir ${OUTPUT}
--merge_bam
Workflow Execution - CLI Execution Profile
singularity
What happened?
I was running the wf-single-cell pipeline as a job (with Slurm) on my institution's clusters with the following input:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=0-72:00:00
#SBATCH --mem=500gb
module load singularity/3.7.2
module load java/15.0.2
OUTPUT=Chrm_290_8_results
./nextflow run epi2me-labs/wf-single-cell -resume
-r prerelease
-w ${OUTPUT}/workspace
-profile singularity
--max_threads 32
--fastq Chrm_290_8/
--kit_name 3prime
--kit_version v3
--expected_cells 45000
--ref_genome_dir refdata-gex-GRCh38-2020-A/
--out_dir ${OUTPUT}
--merge_bam
And the pipeline got terminated with the following error:
executor > local (350)
[98/dc1b6d] process > fastcat (1) [100%] 1 of 1 ✔
[94/48090a] process > parse_kit_metadata (1) [100%] 1 of 1 ✔
[15/f08730] process > pipeline:getVersions [100%] 1 of 1 ✔
[ae/339199] process > pipeline:getParams [100%] 1 of 1 ✔
[90/96c5e1] process > pipeline:summariseCatChunkR... [100%] 1 of 1 ✔
[4c/238272] process > pipeline:stranding:call_ada... [100%] 32 of 32 ✔
[a5/96ff1a] process > pipeline:stranding:combine_... [100%] 1 of 1 ✔
[ec/40172b] process > pipeline:stranding:summariz... [100%] 1 of 1 ✔
[9a/648471] process > pipeline:align:call_paftools [100%] 1 of 1 ✔
[ef/a663e3] process > pipeline:align:get_chrom_sizes [100%] 1 of 1 ✔
[2e/4233a2] process > pipeline:align:align_to_ref... [100%] 1 of 1 ✔
[2c/f7fd41] process > pipeline:process_bams:split... [100%] 1 of 1 ✔
[71/ed8628] process > pipeline:process_bams:get_c... [100%] 1 of 1 ✔
[ba/565adc] process > pipeline:process_bams:extra... [100%] 40 of 40 ✔
[b7/a15deb] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[7f/2fc59c] process > pipeline:process_bams:gener... [100%] 1 of 1 ✔
[5a/a50ffd] process > pipeline:process_bams:assig... [100%] 40 of 40 ✔
[93/68d179] process > pipeline:process_bams:strin... [100%] 40 of 40 ✔
[00/88b60f] process > pipeline:process_bams:align... [100%] 40 of 40 ✔
[2e/22b70a] process > pipeline:process_bams:assig... [100%] 40 of 40 ✔
[ff/8a9950] process > pipeline:process_bams:clust... [100%] 40 of 40 ✔
[85/6bc2ec] process > pipeline:process_bams:tag_b... [100%] 40 of 40 ✔
[5f/7c6266] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[5c/a22c96] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[de/363000] process > pipeline:process_bams:umi_g... [100%] 1 of 1 ✔
[a1/55bd6e] process > pipeline:process_bams:const... [100%] 1 of 1 ✔
[a3/725e61] process > pipeline:process_bams:proce... [100%] 1 of 1 ✔
[08/cdab80] process > pipeline:process_bams:combi... [100%] 1 of 1 ✔
[29/e613f4] process > pipeline:process_bams:pack_... [100%] 1 of 1 ✔
[92/50195f] process > pipeline:prepare_report_dat... [100%] 1 of 1 ✔
[1c/460092] process > pipeline:makeReport (1) [ 0%] 0 of 1
[aa/fde863] process > output (13) [100%] 15 of 15 ✔
[- ] process > output_report -
ERROR ~ Error executing process > 'pipeline:makeReport (1)'
Caused by:
Process pipeline:makeReport (1)
terminated with an error exit status (1)
Command executed:
workflow-glue report --read_stats read_stats.csv --params params.csv --versions versions --survival survival.tsv --wf_summary wf_summary.tsv --output wf-single-cell-report.html --umap_dirs Chrm_290_8_umap --images images_Chrm_290_8 --umap_genes umap_plot_genes.csv
Command exit status:
1
Command output:
(empty)
Command error:
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
[05:13:35 - workflow_glue] Starting entrypoint.
[05:13:35 - Report ] Building report
Traceback (most recent call last):
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in
cli()
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/init.py", line 72, in cli
args.func(args)
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/report.py", line 176, in main
SeqSummary(args.read_stats)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 47, in init
df_all = load_stats(seq_summary, format='fastcat')
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 375, in load_stats
df['start_time'] = df.start_time.dt.tz_localize(None)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in getattr
return object.getattribute(self, name)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/accessor.py", line 224, in get
accessor_obj = self._accessor(obj)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/accessors.py", line 580, in new
raise AttributeError("Can only use .dt accessor with datetimelike values")
AttributeError: Can only use .dt accessor with datetimelike values
Work dir:
/N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9
Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run
-- Check '.nextflow.log' file for details
The wf-single cell pipeline worked smoothly on different data sets. Do you have any suggestions?
Relevant log output
Caused by:
Process `pipeline:makeReport (1)` terminated with an error exit status (1)
Command executed:
workflow-glue report --read_stats read_stats.csv --params params.csv --versions versions --survival survival.tsv --wf_summary wf_summary.tsv --output wf-single-cell-report.html --umap_dirs Chrm_290_8_umap --images images_Chrm_290_8 --umap_genes umap_plot_genes.csv
Command exit status:
1
Command output:
(empty)
Command error:
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
@numba.jit()
/home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: �[1mThe 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.�[0m
@numba.jit()
[05:13:35 - workflow_glue] Starting entrypoint.
[05:13:35 - Report ] Building report
Traceback (most recent call last):
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in <module>
cli()
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 72, in cli
args.func(args)
File "/N/scratch/cerdogan/nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/report.py", line 176, in main
SeqSummary(args.read_stats)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 47, in __init__
df_all = load_stats(seq_summary, format='fastcat')
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 375, in load_stats
df['start_time'] = df.start_time.dt.tz_localize(None)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in __getattr__
return object.__getattribute__(self, name)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/accessor.py", line 224, in __get__
accessor_obj = self._accessor(obj)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/accessors.py", line 580, in __new__
raise AttributeError("Can only use .dt accessor with datetimelike values")
AttributeError: Can only use .dt accessor with datetimelike values
Work dir:
/N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Jan-19 05:17:11.536 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-19 05:17:11.538 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `pipeline:makeReport (1)` terminated with an error exit status (1)
Jan-19 05:17:12.101 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-19 05:17:12.102 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jan-19 05:17:12.241 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=349; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=21d 18h 44m 4s; failedDuration=4m 8s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=29; peakCpus=32; peakMemory=105.3 GB; ]
Jan-19 05:17:12.241 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jan-19 05:17:12.247 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jan-19 05:17:12.806 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jan-19 05:17:13.274 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-19 05:17:13.292 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Application activity log entry
No response
Please also find the log file here:
nextflow.log
Sorry that you're having issues with the workflow.
Would you be able to post a few lines from the following file? /N/scratch/cerdogan/nextflow/Chrm_290_8_results/workspace/1c/4600923fd2c05d30f0c876a8abfad9/read_stats.csv
Thanks. That part of the file looks OK. There's possibly a malformed start_time
entry somewhere in the read stats file. Have you had this issue with any other data sets?
I will update the container with the latest version of the code that generates the stats files to see if that fixes the issue. And will get back to you.
No, I haven't experienced this problem with other datasets. But this dataset was the largest one I have ever worked with the wf-single-cell pipeline.
Thanks.
Sorry for the delay. It would be good if I could get hold of the whole read stats file. I've emailed you a link to share it with me, if you'er able to.
Thanks
I have just been informed of a possibly related issue. When start_time
in the read_stats.csv file contain entries with mixed time zones, an error is raised. Do you happen to have mixed timezone data in the sample in question, maybe from combining samples run at different times.
For example, if you had these two start time entries in read_stats.csv
, it will fail (The second entry is offset by an hour - +01:00)
2022-06-01T19:38:06.299792+00:00
2022-02-15T21:10:45.415802+01:00
This issue will be fixed shortly.
Hi Chiat, thanks for the update. I'll let you know when we have a fix in place for this.
Neil
Hi @cihaterdogan this issue should hopefully be fixed in v1.1.0. Will close this ticket now, thanks again for you help on this.