Issue with test profile
DPI-NZ opened this issue · 11 comments
Description of the bug
I was testing the test profile, and it fails at two points:
-
Parameter --wfmash_sparse_map supplied as string "1.0" fails (picture not attached)
-> solution: should be supplied as 1.0 (without quotation marks) -
Step NFCORE_PANGENOME:PANGENOME:PGGB:WFMASH_MAP_ALIGN fails (picture attached)
-> solution: parameter --wfmash_block_length needs to be a float and replacing with "25000.0" (as string) worked; not sure how to do that with the multiplier setting currently there
Command used and terminal output
nextflow run 'https://github.com/nf-core/pangenome'
-name nice_colden
-params-file 'https://api.tower.nf/ephemeral/4YM4VhfA8XZPm86w10bIdg.json'
-with-tower
-r 1.0.0
-profile testRelevant files
No response
System information
awsbatch
Hi @DPI-NZ,
I don't have AWS, so I have to use my local laptop. Which Nextflow version are you using?
I am trying to follow your command line
nextflow run nf-core/pangenome -name nice_colden -r 1.0.0 -profile test,docker
N E X T F L O W ~ version 23.04.1
Launching `https://github.com/nf-core/pangenome` [nice_colden] DSL2 - revision: 9fc8ccd2ee [1.0.0]
ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
The following invalid input values have been detected:
* Missing required parameter: --outdir
As you can see, it should already have failed earlier. The following works for me.
nextflow run nf-core/pangenome -name nice_colden -r 1.0.0 -profile test,docker --outdir test
N E X T F L O W ~ version 23.04.1
Launching `https://github.com/nf-core/pangenome` [nice_colden] DSL2 - revision: 9fc8ccd2ee [1.0.0]
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/pangenome v1.0.0-g9fc8ccd
------------------------------------------------------
Core Nextflow options
revision : 1.0.0
runName : nice_colden
containerEngine : docker
launchDir : /home/heumos/Downloads/DPI-NZ
workDir : /home/heumos/Downloads/DPI-NZ/work
projectDir : /home/heumos/.nextflow/assets/nf-core/pangenome
userName : heumos
profile : test,docker
configFiles :
Input/output options
input : https://github.com/nf-core/test-datasets/raw/pangenome/data/DRB1-3123/DRB1-3123.fa.gz
n_haplotypes : 12
outdir : test
Wfmash Options
wfmash_block_length : null
wfmash_sparse_map : null
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/pangenome for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/pangenome/blob/master/CITATIONS.md
executor > local (23)
[f6/dc9e78] process > NFCORE_PANGENOME:PANGENOME:INPUT_CHECK:SAMTOOLS_FAIDX (DRB1-3123.fa.gz) [100%] 1 of 1 ✔
[2e/2e14dd] process > NFCORE_PANGENOME:PANGENOME:PGGB:WFMASH_MAP_ALIGN (DRB1-3123.fa.gz) [100%] 1 of 1 ✔
[9d/517f09] process > NFCORE_PANGENOME:PANGENOME:PGGB:SEQWISH (DRB1-3123.fa.gz) [100%] 1 of 1 ✔
[69/e4966a] process > NFCORE_PANGENOME:PANGENOME:PGGB:SMOOTHXG (DRB1-3123.fa.gz) [100%] 1 of 1 ✔
[88/2a9b63] process > NFCORE_PANGENOME:PANGENOME:PGGB:GFAFFIX (DRB1-3123.fa.gz) [100%] 1 of 1 ✔
[52/49d534] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_BUILD (DRB1-3123.fa.gz.gfaffix) [100%] 2 of 2 ✔
[53/52b2ba] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_UNCHOP (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[e2/300dde] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_SORT (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[19/1a0689] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_VIEW (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[0a/53329f] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_STATS (DRB1-3123.fa.gz.gfaffix) [100%] 2 of 2 ✔
[80/651e0a] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_COLOR (DRB1-3123.fa.gz.gfaffix.viz) [100%] 1 of 1 ✔
[dd/197ba5] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_POS (DRB1-3123.fa.gz.gfaffix.viz_pos) [100%] 1 of 1 ✔
[e8/31c9fc] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_DEPTH (DRB1-3123.fa.gz.gfaffix.viz_depth) [100%] 1 of 1 ✔
[95/582db4] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_INV (DRB1-3123.fa.gz.gfaffix.viz_inv) [100%] 1 of 1 ✔
[bd/106bd2] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_COMPR (DRB1-3123.fa.gz.gfaffix.viz_O) [100%] 1 of 1 ✔
[4b/7a1959] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_VIZ_UNCALLED (DRB1-3123.fa.gz.gfaffix.viz_uncalled) [100%] 1 of 1 ✔
[f5/3f6350] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_LAYOUT (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[59/a2dbfa] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_DRAW_MULTIQC (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[ac/eeaa85] process > NFCORE_PANGENOME:PANGENOME:PGGB:ODGI_QC:ODGI_DRAW_HEIGHT (DRB1-3123.fa.gz.gfaffix) [100%] 1 of 1 ✔
[58/617b38] process > NFCORE_PANGENOME:PANGENOME:CUSTOM_DUMPSOFTWAREVERSIONS (1) [100%] 1 of 1 ✔
[06/66211b] process > NFCORE_PANGENOME:PANGENOME:MULTIQC [100%] 1 of 1 ✔
-[nf-core/pangenome] Pipeline completed successfully-
So my suspicion is that your Nextflow version is too old.
Hi @subwaystation, thank you for looking into this. I used Nextflow version 23.04.2.
When I follow your setting with "wfmash_block_length : null", the test also now runs smoothly, but that's not the case for the default test settings that appear in my Nextflow Tower when I try to launch. There it's set to the default, i.e. "default": "wfmash_segment_length * 5" from pangenome/nextflow_schema.json file (line 69).
Is your test not robust, and would break if you set "wfmash_block_length: wfmash_segment_length * 5"? Or is that something special for Tower+AWS execution that I am running?
Thanks!
Hi @DPI-NZ,
I don't know how Nextflow Tower is working in detail. But I do have access to an instance, so maybe @FriederikeHanssen or @mirpedrol can help me here?
The uniqueness with e.g. the wfmash_block_length parameter is that by default, it is calculated automatically after the pipeline has started. So I don't know the actual default value before the pipeline runs. I tried to investigate how to deal with this at https://nfcore.slack.com/archives/CE6SDBX2A/p1689605084916579 and #135 (comment), but to no avail.
You can set wfmash_block_length only to a specific value. If you think the block length should differ from what is currently calculated, just add the number you want to have as a whole.
wfmash_block_length: 50000
To get a better idea of the default calculation take a look at
Line 75 in fc6afeb
Could you please also attach your -params-file?
Hi @subwaystation,
Below are the parameters from the failed test run. When I change wfmash_block_length to 25000.0 or null, the test runs OK. I hope that helps.
Damian
custom_config_base = https://raw.githubusercontent.com/nf-core/configs/master
skip_smoothxg = false
plaintext_email = false
wfmash_only = false
monochrome_logs = false
wfmash_sparse_map = 1
max_cpus = 2
smoothxg_max_edge_jump = 0
smoothxg_max_path_jump = 0
smoothxg_poa_length = 700,900,1100
validationFailUnrecognisedParams = false
seqwish_sparse_factor = 0
smoothxg_poa_params = 1,19,39,3,81,1
max_multiqc_email_size = 25.MB
max_time = 6.h
wfmash_segment_length = 5000
seqwish_min_match_length = 19
validate_params = true
smoothxg_poa_padding = 0.001
validationShowHiddenParams = false
smoothxg_poa_cpus = 0
smoothxg_run_global_poa = false
version = false
publish_dir_mode = copy
input = https://github.com/nf-core/test-datasets/raw/pangenome/data/DRB1-3123/DRB1-3123.fa.gz
config_profile_name = Test profile
validationSkipDuplicateCheck = false
validation-show-hidden-params = false
validationSchemaIgnoreParams = genomes
validation-lenient-mode = false
wfmash_map_pct_id = 90
smoothxg_consensus_prefix = Consensus_
validation-skip-duplicate-check = false
wfmash_no_splits = false
smoothxg_block_ratio_min = 0
custom_config_version = master
max_memory = 6.GB
wfmash_mash_kmer = 19
wfmash_merge_segments = false
schema_ignore_params = igenomes_base
smoothxg_run_abpoa = false
wfmash_chunks = 1
config_profile_description = Minimal test dataset to check pipeline function
wfmash_block_length = wfmash_segment_length * 5.0
smoothxg_pad_max_depth = 100
smoothxg_write_maf = false
validationLenientMode = false
outdir = s3://nz-tower-bucket/results/
seqwish_transclose_batch = 10000000
n_haplotypes = 12
help = false
validation-fail-unrecognised-params = false
smoothxg_keep_intermediate_files = false
show_hidden_params = false
wfmash_mash_kmer_thres = 0.001
validation-schema-ignore-params = genomes
communities = false
You can't enter wfmash_segment_length * 5.0 like this. You need an explicit value here, or don't touch it at all.
Using a double does not make sense to me. Just try 25000. Anyhow, the idea is that you only have to set wfmash_segment_length. The automatic calculation of wfmash_block_length should work very well.
I see - it's just that is what appears once one imports the pipeline to Tower (Add pipeline -> "Pipeline to launch": nf-core/pangenome + "Config profiles": test) and set the "test" profile. It populates the variables with various things for the launch, including:
Am importing the pipeline incorrectly, hence experiencing this behaviour?
Aaah. So Tower is the guilty one. In my opinion.
Sorry, I don't have experience how to fix this. Neither on the Tower side nor on my side. Could you please try in the #help Slack channel? Thanks!
Were you able to solve this @DPI-NZ? Else I would wait for some colleagues to come back from vacation.
If there are no news here within this week, I will close the issue.
For tower related issues, see the nextflow slack, they have a tower-community help channel. the public tower instance can probably also be used for testing how the parameters are filled out.
Please re-open, if this is still a problem.