nipreps/fmriprep

Tag nodes with virtual memory footprints

effigies opened this issue · 3 comments

effigies commented

Right now we're tagging high-memory usage nodes with their maximum demands for resident memory, which is appropriate for systems that permit the kernel to overcommit memory. However, in strict overcommit mode, the kernel counts all virtual memory against its total limits.

For non-strict mode, using resident memory requirements will allow us to better utilize a system's resources, while for strict mode, using virtual memory requirements will prevent us from crashing. So it seems sensible to allow both options, which will adjust the tagging.

I suspect we'll want a function or an object to handle the memory options. If we make some rough assumptions of VM/RSS ratios, we can simply keep the numbers we have in the case of non-strict, and apply a scaling factor for strict. e.g.

MEMORY_MODE = 'strict'
VM_RSS_RATIO = 2

def scale_mem(rss, vm=None):
    if MEMORY_MODE == 'nonstrict':
        return rss
    if vm is not None:
        return vm
    return rss * VM_RSS_RATIO

# Profiled RSS only
node = pe.Node(Interface(), mem_gb=scale_mem(3))

# Profiled RSS AND VM
node = pe.Node(Interface(), mem_gb=scale_mem(3, 5))
oesteban commented

We can check for /proc/sys/vm/overcommit_memory < 2 to see if overcommit is allowed at all (default would be yes, in case the file can't be read). And a default RSS_VM_RATIO (I switched the order, so ratios will be <1.0 in that case) can be read from /proc/sys/vm/overcommit_ratio. A default of 0.5 makes sense.

oesteban commented

Actually, this is probably something we may want to integrate directly into nipype, provide nipype config options etc.