lambda expression in `resources` fails to recognize `threads` during dry run

[fanhuan]

Warning: I'm new to snakemake. This is not an issue but something I don't understand and is dying for help.

Hi,

Thanks for the tool. I was following the instructions without major problems until here,

when I was trying to do a dry-run with the suggested command:

$ snakemake --use-conda -j 1 -Fqn

The error message I received:

Using output directory: some_out_dir_I_specified/output_GTDBr95_n10/ 
Number of skipped sample table entries: 0
Using temporary directory: /tmp/Struo2/user/Struo2_18628754/ 
--Running db-create pipeline--
* Creating kraken2 database
* Creating bracken database
* Creating genes database
* Creating humann3 database
  * Using "mmseqs search" for annotating genes
InputFunctionException in line 125 of /home/hfan/build/struo2/bin/db_create/humann3/db_create/Snakefile:
TypeError: <lambda>() missing 1 required positional argument: 'threads'
Wildcards:
uniref=uniref90

When I looked in the snakefile mentioned above, it is referring to the rule starting on line 125:

rule humann3_bowtie2_build:
    """
    Running bowtie2 build on combined, annotated genes (nucleotide)
    """
    input:
        fna = config['tmp_dir'] + '{uniref}/genome_reps_filt_annot.fna'
    output:
        touch(os.path.join(humann3_dir, '{uniref}/bowtie2_build.done'))
    conda:
        '../../../envs/humann3.yaml'
    threads:
        12
    resources:
        time = lambda wildcards, attempt: attempt * 2 * 60 * 24,
        n = lambda wildcards, attempt, threads: threads,
        mem_gb_pt = lambda wildcards, attempt: attempt ** 2 * 2 + 8,
        lg_idx = lambda wildcards, attempt: '--large-index' if attempt > 1 else ''
    log:
        log_dir + 'humann3_bowtie2_build/{uniref}.log'
    benchmark:
        benchmark_dir + 'humann3_bowtie2_build/{uniref}.txt'
    shell:
        """
        OUTDIR=`dirname {output}`
        PREFIX="$OUTDIR/all_genes_annot"
        bowtie2-build --threads {threads} {resources.lg_idx} \
          {input.fna} $PREFIX 2> {log} 1>&2

        # check that output exists
        IDX_FILES=`find $OUTDIR -maxdepth 1 -name "*.bt2*"`
        IDX_FILES=`echo $IDX_FILES | perl -pe 's/ +/\n/g' | wc -l`
        if [ $IDX_FILES -lt 1 ]; then
          echo "ERROR: no bowtie2 index files found!" >> {log}
          exit 1
        fi
        touch {output} 2>> {log}
        """    

As you can see, the error occurred in the resources part on line:

n = lambda wildcards, attempt, threads: threads,

To me, it seems like this line of code, or the resource block failed to acknowledge that we've assigned some value to threads within this rule, 4 lines above:

threads:
     12

But in the doc of snakemake on the resources keyword, there is an example that is almost doing the same thing (the callable there is a defined function instead of a lambda expression).

def get_mem_mb(wildcards, threads):
    return threads * 150

rule b:
    input:     ...
    output:    ...
    threads: 8
    resources:
        mem_mb=get_mem_mb
    shell:
        "..."

I'm out of my depth here on what actually went wrong. Also, I don't see this n callable used anywhere. Any help would be highly appreciated.

[nick-youngblut]

The line wrapping doesn't matter for threads: or any other rule in a snakefile, so that shouldn't be the problem. Which version of snakemake are you running? Do you still get the error if you use >1 thread (eg., snakemake -j 2)?

[fanhuan]

Version: 4.3.1

Yes the same error with snakemake -j 2.

[nick-youngblut]

You are using snakemake v4.3.1? Try updating snakemake to v6.7.0

[fanhuan]

Updating snakemake to v6.* solved the problem. Thank you Nick. (and I feel like an idiot.)

[nick-youngblut]

No worries. Easy fixes are the best!