Unexpected swapping with COBS
karel-brinda opened this issue · 6 comments
It seems that when COBS is used with 2 threads, it somehow uses twice as much virtual memory (with --load-complete).
Is it possible @leoisl ? See
E.coli 17 uses 16 GB of virtual memory.
But the size after decompression is the following: 8.1G Sep 21 12:42 escherichia_coli__17.cobs_classic
In consequence, the system seems to be massively swapping (see the red parts), even though it shouldn't be.
Also, this can be just my misunderstanding of what's going on...
@leoisl Is it theoretically possible that COBS with --load-complete uses at the same time memory mapping for the file and its loaded image, which would explain the doubling?
This is weird and very important, debugging...
@leoisl Is it theoretically possible that COBS with --load-complete uses at the same time memory mapping for the file and its loaded image, which would explain the doubling?
As far as I know this should not be happening, but I will debug some COBS runs and try to understand
@leoisl Is it theoretically possible that COBS with --load-complete uses at the same time memory mapping for the file and its loaded image, which would explain the doubling?
Hey @karel-brinda , I have checked the code in details. My understanding of the code, which I executed step by step in a debug environment, shows that the RAM consumption does not depend on the number of threads with --load-complete. This was further confirmed by two sets of benchmarks with two different indexes, varying threads in [1, 2, 4, 8]. All runs for the same indexes kept the same RAM usage. However, I also would like to remember that COBS was designed and implemented for linux, not Mac OS X. It has not been extensively tested and benchmarked in Mac OS X. This could be potentially an issue happening with Mac OS X only. Could you please do some further benchmark on this index, for example, with varying number of threads to see if RAM increases and how it increases?
Interesting. I will keep it open and we should look at this at some point.
Seems to be fixed by #210