Mark duplicates sometimes overflows memory

[cnluzon]

I have seen that mark_pe_duplicates rule sometimes gets killed due to memory overflow. It may be the repeated ID dictionary that is in memory, since the files that caused this last time were pooled BAM, 12GB in size, and this job has only one core allocated (6GB mem) but I need to look at it in more detail.

For now, I think we should allocate at least one core more to make sure this does not happen often.

[marcelm]

When using the SLURM profile, this should be specified as a memory requirement. Something like this I think (not tested):

rule ...:
    resources:
        mem_mb=12000

Requesting two cores will also work at the moment, but that transfer to other clusters so well that might have different memory allocations per core. Also, one might get an impression that duplicate marking actually uses two threads, which would not be correct.

[cnluzon]

I agree that if memory is what is required, it should be allocated as such (plus, we get charged for 2 cores even if we only use one). I was suggesting allocating one more core because in Uppmax documentation it kind of implies that you need to request memory through core numbers when you specify core (I guess if you're going to allocate the full node, this does not apply anyway, since you will get the full node with all its resources):

Example, on Rackham: If a job needs half of the RAM, i.e. 64 GB, you need to reserve also at least half of the cores on the node, i.e. ten cores, with the "-n 10" flag

I know that Uppmax's SLURM system does not have all SLURM options available to users. But I haven't actually tried to ask for memory and not for cores, maybe it does work regardless.

[marcelm]

Interesting, just reading through the page (which I haven’t looked at in a while) it appears you are right, but I think it is outdated.

SLURM is smarter than what the UPPMAX text implies: I ran sbatch -n 1 --mem 12 GB job.sh and actually got a job where I had two cores available. So it is correct that SLURM gives out resources in units of "1 core and 6.4 GB memory", but how many of those units you get is determined by both -n and --mem.

This also agrees with the way I have controlled SLURM resource usage from Nextflow. So mem_mb should work just fine!

[cnluzon]

Then I agree and we should definitely ask for memory, not cores.

As a follow up on this, I have seen that there is an attempt directive that can be used when allocating resources: in a rule. So it could be a good solution if we assess that this is the unusual case for our data size (which, so far, it's my perception, it's rare that we get such large files I think). This way we avoid getting charged for two cores each time. The downside is that the demanding job will fail after running for a while so effectively will run twice. But I think considering the amount of BAM files that are processed in a pipeline run, this should be way smaller waste than asking always for 2 cores.

One solution would be to ask for the memory allocated for one core by default, and upon failure ask for more memory. But since the memory required is proportional to the size of the BAM files, I could put more accurate numbers on this. If we are accurate with the estimation, we can minimize the attempts.

(or I could find a more memory-efficient way to deduplicate 😅)