Running with test parameters, how to specify readPaths

Question

Running with test parameters, how to specify readPaths

PhilPalmer opened this issue 6 years ago · 8 comments

Hi,

When I try running hlatyping with the test profile it works. However if I try passing what I think should be the same inputs as the test profile (for both the bam & fastq) it fails. I'm probably doing something wrong with passing the input parameters (esepcially readPaths) but any help would be much appreciated.

BAM

I'm not exactly sure why but it seems like the input_data is not set correctly from the mapping. If I print the value of the input_data chanel I get [h, [/Users/phil/Documents/GitHub/nf-core/hlatyping/l]]

nextflow run nf-core/hlatyping --readPaths hlatyping/testdata/example_pe.bam --bam --singleEnd false --index hlatyping/testdata/indicies/yara/hla_reference_dna -profile standard,docker
Unable to find any JVMs matching version "1.8".
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/hlatyping` [festering_euler] - revision: fc2d6c06af [master]
WARN: The config file defines settings for an unknown process: fastqc
=========================================
WARN: Access to undefined parameter `version` -- Initialise it to a default value eg. `params.version = some_value`
 nf-core/hlatyping vnull
=========================================
Run Name       : festering_euler
Reads          : hlatyping/testdata/example_pe.bam
Data Type      : Paired-End
File Type      : BAM
IP solver      : glpk
Enumerations   : 1
Beta           : 0.009
Prefix         : hla_run
Max Memory     : 128 GB
Max CP Us      : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /Users/phil/Documents/GitHub/nf-core/hlatyping/work
Container      : nfcore/hlatyping:1.1.1
Pipeline Release: master
Current home   : /Users/phil
Current user   : phil
Current path   : /Users/phil/Documents/GitHub/nf-core/hlatyping
Script dir     : /Users/phil/.nextflow/assets/nf-core/hlatyping
Config Profile : standard,docker
=========================================
WARN: Access to undefined parameter `nf_required_version` -- Initialise it to a default value eg. `params.nf_required_version = some_value`
ERROR ~ ====================================================
  Nextflow version null required! You are running v0.32.0.
  Pipeline execution will continue, but things may break.
  Please run `nextflow self-update` to update Nextflow.
============================================================

 -- Check '.nextflow.log' file for details
BAM file format detected. Initiate remapping to HLA alleles with yara mapper.
[warm up] executor > local
[25/576e76] Submitted process > make_ot_config
[7e/ceb4d6] Submitted process > remap_to_hla (1)
ERROR ~ Error executing process > 'remap_to_hla (1)'

Caused by:
  Process `remap_to_hla (1)` terminated with an error exit status (1)

Command executed:

  samtools view -h -f 0x40 l > output_1.bam
  samtools view -h -f 0x80 l > output_2.bam
  samtools bam2fq output_1.bam > output_1.fastq
  samtools bam2fq output_2.bam > output_2.fastq
  yara_mapper -e 3 -t 16 -f bam /Users/phil/.nextflow/assets/nf-core/hlatyping/hlatyping/testdata/indicies/yara/hla_reference_dna output_1.fastq output_2.fastq > output.bam
  samtools view -h -F 4 -f 0x40 -b1 output.bam > mapped_1.bam
  samtools view -h -F 4 -f 0x80 -b1 output.bam > mapped_2.bam

Command exit status:
  1

Command output:
  (empty)

Command error:
  [E::hts_open_format] Failed to open file l
  samtools view: failed to open "l" for reading: No such file or directory

Work dir:
  /Users/phil/Documents/GitHub/nf-core/hlatyping/work/7e/ceb4d618fabcd48745ac0b394bcf84

FASTQ

Not sure if I have specified the value of readPaths correctly here. Looks like it's a problem with the reads again that is causing the issue here

nextflow run nf-core/hlatyping --readPaths "testdata/NA11995_SRR766010_*" --index testdata/indicies/yara/hla_reference_dna
Unable to find any JVMs matching version "1.8".
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/hlatyping` [admiring_lavoisier] - revision: fc2d6c06af [master]
WARN: The config file defines settings for an unknown process: fastqc
=========================================
WARN: Access to undefined parameter `version` -- Initialise it to a default value eg. `params.version = some_value`
 nf-core/hlatyping vnull
=========================================
Run Name       : admiring_lavoisier
Reads          : testdata/NA11995_SRR766010_*
Data Type      : Paired-End
File Type      : Other (fastq, fastq.gz, ...)
IP solver      : glpk
Enumerations   : 1
Beta           : 0.009
Prefix         : hla_run
Max Memory     : 128 GB
Max CP Us      : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /Users/phil/Documents/GitHub/nf-core/hlatyping/work
Container      : nfcore/hlatyping:1.1.1
Pipeline Release: master
Current home   : /Users/phil
Current user   : phil
Current path   : /Users/phil/Documents/GitHub/nf-core/hlatyping
Script dir     : /Users/phil/.nextflow/assets/nf-core/hlatyping
Config Profile : standard
=========================================
WARN: Access to undefined parameter `nf_required_version` -- Initialise it to a default value eg. `params.nf_required_version = some_value`
ERROR ~ ====================================================
  Nextflow version null required! You are running v0.32.0.
  Pipeline execution will continue, but things may break.
  Please run `nextflow self-update` to update Nextflow.
============================================================

 -- Check '.nextflow.log' file for details
ERROR ~ begin 1, end 2, length 1

 -- Check script 'main.nf' at line: 138 or see '.nextflow.log' file for more details
[warm up] executor > local

Answer 1 · 2018-12-01T11:02:43.000Z

Hm, could you try again with Nextflow 18.10.1.? This looks a bit like the nf_required_version is not defined but it should be...

Answer 2 · 2018-12-01T22:53:56.000Z

I retried it with nextflow v18.10.1, but unfortauntely I got the same error.
I think there may be a couple of problems:

The readPaths are not being specifed properly. I think this is a problem caused from the mapping as I mentioned above, although I'm still not 100% sure I've specified them correctly.
The nf_required_version doesn't seem to be working properly. This is likely a more minor error

nextflow run nf-core/hlatyping --readPaths hlatyping/testdata/example_pe.bam --bam --singleEnd false --index hlatyping/testdata/indicies/yara/hla_reference_dna -profile standard,docker
Unable to find any JVMs matching version "1.8".
N E X T F L O W  ~  version 18.10.1
WARN: Invalid config manifest attribute `pipelineVersion`
Launching `nf-core/hlatyping` [nauseous_stallman] - revision: fc2d6c06af [master]
WARN: Invalid config manifest attribute `pipelineVersion`
WARN: The config file defines settings for an unknown process: fastqc
WARN: Access to undefined parameter `reads` -- Initialise it to a default value eg. `params.reads = some_value`
=========================================
WARN: Access to undefined parameter `version` -- Initialise it to a default value eg. `params.version = some_value`
 nf-core/hlatyping vnull
=========================================
Run Name       : nauseous_stallman
Reads          : hlatyping/testdata/example_pe.bam
Data Type      : Paired-End
File Type      : BAM
IP solver      : glpk
Enumerations   : 1
Beta           : 0.009
Prefix         : hla_run
Max Memory     : 128 GB
Max CP Us      : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /Users/phil/Documents/GitHub/nf-core/work
Container      : nfcore/hlatyping:1.1.1
Pipeline Release: master
Current home   : /Users/phil
Current user   : phil
Current path   : /Users/phil/Documents/GitHub/nf-core
Script dir     : /Users/phil/.nextflow/assets/nf-core/hlatyping
Config Profile : standard,docker
=========================================
WARN: Access to undefined parameter `nf_required_version` -- Initialise it to a default value eg. `params.nf_required_version = some_value`
ERROR ~ ====================================================
  Nextflow version null required! You are running v18.10.1.
  Pipeline execution will continue, but things may break.
  Please run `nextflow self-update` to update Nextflow.
============================================================

 -- Check '.nextflow.log' file for details
BAM file format detected. Initiate remapping to HLA alleles with yara mapper.
[warm up] executor > local
[30/addb6c] Submitted process > remap_to_hla (1)
[67/c51569] Submitted process > make_ot_config
ERROR ~ Error executing process > 'remap_to_hla (1)'

Caused by:
  Process `remap_to_hla (1)` terminated with an error exit status (1)

Command executed:

  samtools view -h -f 0x40 l > output_1.bam
  samtools view -h -f 0x80 l > output_2.bam
  samtools bam2fq output_1.bam > output_1.fastq
  samtools bam2fq output_2.bam > output_2.fastq
  yara_mapper -e 3 -t 16 -f bam /Users/phil/.nextflow/assets/nf-core/hlatyping/hlatyping/testdata/indicies/yara/hla_reference_dna output_1.fastq output_2.fastq > output.bam
  samtools view -h -F 4 -f 0x40 -b1 output.bam > mapped_1.bam
  samtools view -h -F 4 -f 0x80 -b1 output.bam > mapped_2.bam

Command exit status:
  1

Command output:
  (empty)

Command error:
  [E::hts_open_format] Failed to open file l
  samtools view: failed to open "l" for reading: No such file or directory

Work dir:
  /Users/phil/Documents/GitHub/nf-core/work/30/addb6ce86753e06cd1140dbc498c51

nextflow.log

Answer 3 · 2018-12-07T07:26:30.000Z

Hey @PhilPalmer thanks for reporting this. I will look into it asap! Sorry for the late response

Answer 4 · 2018-12-12T12:15:53.000Z

Hey @PhilPalmer,

i think the correct command has to be

nextflow run nf-core/hlatyping --reads hlatyping/testdata/example_pe.bam --bam --index hlatyping/testdata/indicies/yara/hla_reference_dna -profile standard,docker

Can you confirm this please?

Best, Sven

Answer 5 · 2018-12-19T09:02:40.000Z

Dear @PhilPalmer would you mind confirming the corrected command line execution? So I can close this issue ;)

Answer 6 · 2018-12-20T10:56:20.000Z

@sven1103 sorry for taking so long to reply.

Unfortunately, I'm afraid I'm still having issues:

nextflow run nf-core/hlatyping --reads testdata/example_pe.bam --bam --index data/indices/yara/hla_reference_dna -profile standard,docker
N E X T F L O W  ~  version 18.10.1
Pulling nf-core/hlatyping ...
 downloaded from https://github.com/nf-core/hlatyping.git
Launching `nf-core/hlatyping` [mad_woese] - revision: 5a56879580 [master]
WARN: The config file defines settings for an unknown process: fastqc
=========================================
WARN: Access to undefined parameter `version` -- Initialise it to a default value eg. `params.version = some_value`
 nf-core/hlatyping vnull
=========================================
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
Run Name       : mad_woese
Reads          : testdata/example_pe.bam
Data Type      : Paired-End
File Type      : BAM
Index Location : data/indices/yara/hla_reference_dna
IP solver      : glpk
Enumerations   : 1
Beta           : 0.009
Prefix         : hla_run
Max Memory     : 128 GB
Max CP Us      : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /phil/hlatyping/work
Container      : nfcore/hlatyping:1.1.2
Pipeline Release: master
Current home   : /root
Current user   : root
Current path   : /phil/hlatyping
Script dir     : /root/.nextflow/assets/nf-core/hlatyping
Config Profile : standard,docker
=========================================
WARN: Access to undefined parameter `nf_required_version` -- Initialise it to a default value eg. `params.nf_required_version = some_value`
ERROR ~ ====================================================
  Nextflow version null required! You are running v18.10.1.
  Pipeline execution will continue, but things may break.
  Please run `nextflow self-update` to update Nextflow.
============================================================

 -- Check '.nextflow.log' file for details
ERROR ~ No signature of method: nextflow.Channel$_fromFilePairs0_closure6.call() is applicable for argument types: (sun.nio.fs.UnixPath) values: [/phil/hlatyping/testdata/example_pe.bam]
Possible solutions: any(), any(), tap(groovy.lang.Closure), each(groovy.lang.Closure), any(groovy.lang.Closure), tap(groovy.lang.Closure)

 -- Check '.nextflow.log' file for details
ERROR ~ Cannot find any reads matching: testdata/example_pe.bam
NB: Path needsto be enclosed in quotes!
NB: Path requires at least one * wildcard!
If this is single-end data, please specify --singleEnd on the command line.

 -- Check '.nextflow.log' file for details

Here's the log file: nextflow.log
I also tried on the dev branch but had the same problem there as well.

It looks like it's a problem with the fromFilePairs but I'm not exactly sure whats causing it

.fromFilePairs( params.reads, size: params.singleEnd || params.bam ? 1 : 2 )

Answer 7 · 2019-01-08T15:30:08.000Z

I'll take care of this as the next open issue ....

Answer 8 · 2019-01-18T12:05:08.000Z

Dear @PhilPalmer did @apeltzer's commit fix your issue?