Memory reporting is inconsistent with "free"

Description

The total memory amount is consistent but the free amount does not add up to what 'free' reports using the same units. I suspect that this is due to the handling of 'shared memory' that simply should be ignored, and is accounted for in the other fields already.

The reference to how the mess in /proc/meminfo should be interpreted is undoubtedly 'free.c' from procps.

https://gitlab.com/procps-ng/procps/-/tree/master?ref_type=heads

Neofetch version

7.1.0

Screenshot

Config file

nil

Verbose log

https://pastebin.com/GAcSZZWc

free is consistent with what the kernel developers suggest user space to do:

MemTotal = MemAvailable + "Used"

where "used" is non-volatile, allocated memory in various ways, but not directly defined by /proc/meminfo. With basic algebra we can derive the total lump of "used" memory from the total and the kernel reported available memory:

"Used" = MemTotal - MemAvailable

This is what free effectively does:

We do not need to break any 2.6 kernel users out there we can simply check the kernel for existance of 'MemoryAvailable'. It is important to get these things right because other tools look at random other tools and errors propagate in the ecosystem. Just look at the number of references to this issue: KittyKatt/screenFetch#386 (comment)

We can happily leave the taxonomy details of how memory is micro managed by the kernel to the kernel API, and achieve less work, and consistency with the reference report by 'free'.

I patched my neofetch locally and with a green light I can submit a PR to share it.

@tobbez Torbjörn, do you care to comment on this things or have you moved on to more important things than bit-picking on memory reporting? ;-)

Using MemTotal - MemAvailable would be reasonable, and that's also what procps-ng does (falling back to MemTotal - MemFree if the result is negative).

Problem is, neofetch already uses MemAvailable if it is available (since 106a53c):

neofetch/neofetch

Lines 2689 to 2701 in ccd5d9f

    
                   # Available since Linux 3.14rc (34e431b0ae398fc54ea69ff85ec700722c9da773). 
        
                   # If detected this will be used over the above calculation for mem_used. 
        
                   "MemAvailable") 
        
                       mem_avail=${b/kB} 
        
                   ;; 
        
               esac 
        
           done < /proc/meminfo 
        
           if [[ $mem_avail ]]; then 
        
               mem_used=$(((mem_total - mem_avail) / 1024)) 
        
           else 
        
               mem_used="$((mem_used / 1024))" 
        
           fi

In addition, the handling of Shmem in the fallback path should still be correct.

The logic behind including Shmem becomes clearer if you look at the conky code from which it originates:

  /* Reclaimable memory: does not include shared memory, which is part of cached
     but unreclaimable. Includes the reclaimable part of the Slab cache though.
     Note: when shared memory is swapped out, shmem decreases and swapfree
     decreases - we want this.
  */
  curbufmem = (info.cached - info.shmem) + info.buffers + sreclaimable;

So the calculation currently used:

used = MemTotal + Shmem - MemFree - Buffers - Cached - SReclaimable

was simplified from:

used = MemTotal - MemFree - Buffers - (Cached - Shmem) - SReclaimable

That Shmem should indeed be excluded from Cached is confirmed by posts on the Linux kernel mailing list from 2016 (discussing a patch that proposed excluding shmem and driver pages from Cached in /proc/meminfo, precisely because of this confusion):

Even before we added MemAvailable, users knew that page cache is
easily convertible to free memory on pressure, and estimated their
"available" memory by looking at the sum of MemFree, Cached, Buffers.
However, "Cached" is calculated using NR_FILE_PAGES, which includes
shmem and random driver pages inserted into the page tables; neither
of which are easily reclaimable, or reclaimable at all. Reclaiming
shmem requires swapping, which is slow. And unlike page cache, which
has fairly conservative dirty limits, all of shmem needs to be written
out before becoming evictable. Without swap, shmem is not evictable at
all. And driver pages certainly never are.

Calling these pages "Cached" is misleading and has resulted in broken
formulas in userspace. They misrepresent the memory situation and
cause either waste or unexpected OOM kills.

-- https://lore.kernel.org/lkml/1455827801-13082-1-git-send-email-hannes@cmpxchg.org/

We have to assume that Cached has been useful to some people, and that
they've learnt to subtract Shmem from it, if slow or no swap concerns them.

-- https://lore.kernel.org/lkml/alpine.LSU.2.11.1602181422550.2289@eggly.anvils/

I think everything will ok. Subtraction of shmem isn't widespread practice,
more like secret knowledge. This wasn't documented and people who use
this should be aware that this might stop working at any time. So, ACK.

-- https://lore.kernel.org/lkml/CALYGNiMHAtaZfGovYeud65Eix8v0OSWSx8F=4K+pqF6akQah0A@mail.gmail.com/

(it appears the patch under discussion was not subsequently committed)

Thank you for the input!

Problem is, neofetch already uses MemAvailable if it is available (since 106a53c):

Look at that! Great to see that I was not alone in thinking that this could be improved.

No action required here then. Well, we want that next release soon. 7.1.0 is 2 years old! ;-)

	# Available since Linux 3.14rc (34e431b0ae398fc54ea69ff85ec700722c9da773).
	# If detected this will be used over the above calculation for mem_used.
	"MemAvailable")
	mem_avail=${b/kB}
	;;
	esac
	done < /proc/meminfo

	if [[ $mem_avail ]]; then
	mem_used=$(((mem_total - mem_avail) / 1024))
	else
	mem_used="$((mem_used / 1024))"
	fi