[BUG] Include statement is not allowed within a workflow definition
Opened this issue · 9 comments
Describe the bug
I am proceeding through the steps as described in the 10k PBMC tutorial:
https://vsn-pipelines-examples.readthedocs.io/en/latest/PBMC10k.html
The following error is encountered when attempting to process a single sample:
`Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e [master]
Include statement is not allowed within a workflow definition
-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`
To Reproduce
Steps to reproduce the behavior:
Follow the configuration steps as described here on Macbook.
- Run using this entry point:
`Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e [master]
Include statement is not allowed within a workflow definition
-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`
- See error:
`N E X T F L O W ~ version 21.10.5
Launching vib-singlecell-nf/vsn-pipelines [confident_legentil] - revision: 96cff8e [master]
Include statement is not allowed within a workflow definition
-- Check script '/Users/mattsdwatson/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details`
Screenshots
The full stack trace output is attached here describing the error in further detail.
Please complete the following information:
- OS: Macbook
- Nextflow Version: 21.10.5.5658
- vsn-pipelines Version: 0.27.0
Note: that Singularity is not installed or used for this test case.
+1 on:
- OS: Ubuntu 18.04.1
- Nextflow Version: 21.10.6
- vsn-pipelines Version: 0.27.0
and on:
- CentOS Linux release 7.7.1908
Hey @matt-sd-watson and @roncoronimiguel, could you try again using Nextflow version 21.04.3 and let me know if you have the same issue?
Cheers,
Kris
@KrisDavie no issues when using Nexflow 21.04.3
Thanks, looking into it, there was indeed a bug which has been fixed that affects our code on newer versions of nextflow. For now, the solution is to use 21.04.3, and we'll fix this for the next release.
Hi @KrisDavie,
I am still having the issue and it is does not change whatever version of nextflow I use,
`me@gauss07$ ./nextflow-21.10.3-all -C pbmc10k.vsn-pipelines.complete.config run vib- singlecell-nf/vsn-pipelines -entry single_sample
N E X T F L O W ~ version 21.10.3
Launching `vib-singlecell-nf/vsn-pipelines` [disturbed_mayer] - revision: 96cff8e597 [master]
Missing process or function with name 'include'
-- Check script '/home/me/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details
me@gauss07:/mnt/DOSI/MDALAB/USERS/Active/Ammar_Sabir_me/scenic_analysis/example$ vim /home/me/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf
me@gauss07$ nextflow -C pbmc10k.vsn-pipelines.complete.config run vib-singlecell-nf/vsn-pipelines -entry single_sample
N E X T F L O W ~ version 21.10.6
Launching `vib-singlecell-nf/vsn-pipelines` [sleepy_gates] - revision: 96cff8e597 [master]
Include statement is not allowed within a workflow definition
-- Check script '~/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details
me@gauss07 $ ./nextflow-21.10.4-all -C pbmc10k.vsn-pipelines.complete.config run vib-singlecell- nf/vsn-pipelines -entry single_sample
N E X T F L O W ~ version 21.10.4
Launching `vib-singlecell-nf/vsn-pipelines` [romantic_nobel] - revision: 96cff8e597 [master]
Include statement is not allowed within a workflow definition
-- Check script '/home/cheema/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/main.nf' at line: 237 or see '.nextflow.log' file for more details
I tried using version 21.10.6, 21.10.4 and 21.10.3 and it does not remove the error. My config file is given below,
I don't know if this is due to some problem in my config file, please have a look on the config file given below,
manifest {
name = 'vib-singlecell-nf/vsn-pipelines'
description = 'A repository of pipelines for single-cell data in Nextflow DSL2'
homePage = 'https://github.com/vib-singlecell-nf/vsn-pipelines'
version = '0.27.0'
mainScript = 'main.nf'
defaultBranch = 'master'
nextflowVersion = '!>=21.04.3'
}
params {
global {
project_name = 'Test'
outdir = 'out'
species = 'human'
genome {
assembly = 'hg38'
}
}
misc {
test {
enabled = false
}
}
utils {
container = 'vibsinglecellnf/utils:0.4.0'
file_converter {
off = 'h5ad'
tagCellWithSampleId = true
remove10xGEMWell = false
useFilteredMatrix = true
makeVarIndexUnique = false
}
publish {
compressionLevel = 6
annotateWithBatchVariableName = false
mode = 'copy'
}
scope {
genome = ''
tree {
level_1 = ''
level_2 = ''
level_3 = ''
}
}
}
tools {
scanpy {
container = 'vibsinglecellnf/scanpy:1.8.1'
report {
annotations_to_plot = []
}
feature_selection {
report_ipynb = '/src/scanpy/bin/reports/sc_select_variable_genes_report.ipynb'
flavor = 'seurat'
minMean = 0.0125
maxMean = 3
minDisp = 0.5
off = 'h5ad'
}
feature_scaling {
method = 'zscore_scale'
maxSD = 10
off = 'h5ad'
}
neighborhood_graph {
nPcs = 50
off = 'h5ad'
}
dim_reduction {
report_ipynb = '/src/scanpy/bin/reports/sc_dim_reduction_report.ipynb'
pca {
method = 'pca'
nComps = 50
off = 'h5ad'
}
umap {
method = 'umap'
off = 'h5ad'
}
tsne {
method = 'tsne'
off = 'h5ad'
}
}
clustering {
preflight_checks = true
report_ipynb = '/src/scanpy/bin/reports/sc_clustering_report.ipynb'
method = 'louvain'
resolution = 0.8
off = 'h5ad'
}
marker_genes {
method = 'wilcoxon'
ngenes = 0
groupby = 'louvain'
off = 'h5ad'
}
filter {
report_ipynb = '/src/scanpy/bin/reports/sc_filter_qc_report.ipynb'
cellFilterStrategy = 'fixedthresholds'
cellFilterMinNGenes = 200
cellFilterMaxNGenes = 4000
cellFilterMaxPercentMito = 0.15
geneFilterMinNCells = 3
off = 'h5ad'
outdir = 'out'
}
data_transformation {
method = 'log1p'
off = 'h5ad'
}
normalization {
method = 'cpx'
targetSum = 10000
off = 'h5ad'
}
}
scenic {
container = 'aertslab/pyscenic:0.10.4'
scenicoutdir = 'out/scenic/'
filteredLoom = '/ddn1/vol1/staging/leuven/stg_00002/lcb/cflerin/testruns/scenic-nf_testing /expr_mat.loom'
scenicOutputLoom = 'SCENIC_output.loom'
scenicScopeOutputLoom = 'SCENIC_SCope_output.loom'
mode = 'dask_multiprocessing'
client_or_address = ''
cell_id_attribute = 'CellID'
gene_attribute = 'Gene'
report_ipynb = '/src/scenic/bin/reports/scenic_report.ipynb'
skipReports = false
grn {
algorithm = 'grnboost2'
tfs = 'allTFs_mm.txt'
}
cistarget {
adj = 'adj.tsv'
type = ''
rank_threshold = 5000
auc_threshold = 0.05
nes_threshold = 3.0
min_orthologous_identity = 0.0
max_similarity_fdr = 0.001
annotations_fname = ''
thresholds = '0.75,0.90'
top_n_targets = 50
top_n_regulators = '5,10,50'
min_genes = 20
all_modules = false
motifsDb = '/ddn1/vol1/staging/leuven/stg_00002/lcb/cflerin/testruns/scenic-nf_testing/genome-ranking.feather'
motifsAnnotation = 'motifs-v9-nr.mgi-m0.001-o0.0.tbl'
tracksDb = '/staging/leuven/res_00001/databases/cistarget/databases/homo_sapiens/hg38/refseq_r80/tc_v1/gene_based/mm10__refseq-r80__*feather'
}
aucell {
output = 'aucell_output.loom'
rank_threshold = 5000
auc_threshold = 0.05
nes_threshold = 3.0
min_genes_regulon = 5
min_regulon_gene_occurrence = 5
}
numRuns = 100
aggregate_features {
use_chunking = true
output_format = 'csv'
compression = 'gzip'
}
}
}
data {
tenx {
cellranger_mex = 'data/raw_gene_expression_df.csv'
}
}
}
process {
executor = 'local'
cpus = 2
memory = '60 GB'
clusterOptions = '-A cluster_account'
withLabel:compute_resources__default {
time = '1h'
}
withLabel:compute_resources__minimal {
cpus = 1
memory = '1 GB'
}
withLabel:compute_resources__mem {
cpus = 4
memory = '160 GB'
}
withLabel:compute_resources__cpu {
cpus = 20
memory = '80 GB'
}
withLabel:compute_resources__report {
maxForks = 2
cpus = 1
memory = '160 GB'
}
withLabel:compute_resources__24hqueue {
time = '24h'
}
withLabel:'compute_resources__scenic.*' {
cpus = 4
memory = '60 GB'
time = '24h'
maxForks = 1
}
withLabel:compute_resources__scenic_grn {
cpus = 4
memory = '120 GB'
time = '24h'
}
withLabel:compute_resources__scenic_cistarget {
cpus = 4
memory = '80 GB'
}
withLabel:compute_resources__scenic_aucell {
cpus = 4
memory = '80 GB'
}
withLabel:'compute_resources__scenic_multiruns.*' {
cpus = 4
memory = '60 GB'
time = '24h'
}
withLabel:compute_resources__scenic_multiruns_motifs2regulons {
memory = '60 GB'
}
}
timeline {
enabled = true
file = 'out/nextflow_reports/execution_timeline.html'
}
report {
enabled = true
file = 'out/nextflow_reports/execution_report.html'
}
trace {
enabled = true
file = 'out/nextflow_reports/execution_trace.txt'
}
dag {
enabled = true
file = 'out/nextflow_reports/pipeline_dag.svg'
}
min {
enabled = false
}
vsc {
enabled = false
}memory = '1 GB'
}
withLabel:compute_resources__mem {
cpus = 4
memory = '160 GB'
}
withLabel:compute_resources__cpu {
cpus = 20
memory = '80 GB'
}
withLabel:compute_resources__report {
maxForks = 2
cpus = 1
memory = '160 GB'
}
withLabel:compute_resources__24hqueue {
time = '24h'
}
withLabel:'compute_resources__scenic.*' {
cpus = 4
memory = '60 GB'
time = '24h'
maxForks = 1
}
withLabel:compute_resources__scenic_grn {
cpus = 4
memory = '120 GB'
time = '24h'
}
withLabel:compute_resources__scenic_cistarget {
cpus = 4
memory = '80 GB'
}
withLabel:compute_resources__scenic_aucell {
cpus = 4
memory = '80 GB'
}
withLabel:'compute_resources__scenic_multiruns.*' {
cpus = 4
memory = '60 GB'
time = '24h'
}
withLabel:compute_resources__scenic_multiruns_motifs2regulons {
memory = '60 GB'
}
}
timeline {
enabled = true
file = 'out/nextflow_reports/execution_timeline.html'
}
report {
enabled = true
file = 'out/nextflow_reports/execution_report.html'
}
trace {
enabled = true
file = 'out/nextflow_reports/execution_trace.txt'
}
dag {
enabled = true
file = 'out/nextflow_reports/pipeline_dag.svg'
}
min {
enabled = false
}
vsc {
enabled = false
}
singularity {
enabled = true
autoMounts = true
runOptions = '--cleanenv -H $PWD -B ${HOME}'
}
singularity {
enabled = true
autoMounts = true
runOptions = '--cleanenv -H $PWD -B ${HOME}'
}
Thanks,
Hi @MarcDalod, you must use version 21.04.3, anything above 21.04.3 will not work until we update VSN unfortunately, and the versions you mentioned were all 21.10 or higher.
Can you try 21.04.3 and report back?
Cheers,
Kris
Hi @KrisDavie ,
Thanks for your response, I have tried with version 21.04.3 but it is giving the problem of java, but java is already installed,
cheemaa@gauss05$ ./nextflow-21.04.3-all
ERROR: Cannot find Java or it's a wrong version -- please make sure that Java 8 or later is installed
NOTE: Nextflow is trying to use the Java VM defined by the following environment variables:
JAVA_CMD: /usr/bin/java
JAVA_HOME:
$ java --version
openjdk 17.0.2 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 17.0.2+8-Ubuntu-120.04, mixed mode, sharing)
cheemaa@gauss05$ which java
/usr/bin/java
Can you please let me know if there is some solution to this problem ?
Hey @MarcDalod,
That version of Java (17) is much to high for the older version of nextflow, you should use something between 8 and 12 as described here: nextflow-io/nextflow#1662
This won't be an issue any longer once we update VSN to use newer nextflow versions as higher java versions are supported in nextflow 21.05 and higher.
Cheers,
Kris
Hi @KrisDavie ,
Then is it possible that I make a docker image with the right version of java, nexflow and the pyscenic container?
Do you think it could work this way?
Thanks,
Ammar