Force high memory for default C heap
Closed this issue · 3 comments
kLIBC implements a voting mechanism when deciding which heap to use by default: in high memory (above 512 M) or in low memory. The EXE and each kLIBC DLL statically linked to this EXE may participate by voting for high memory (via -Zhigh-mem in LDFLAGS) or for low memory (when -Zhigh-mem is absent). If there is at least one DLL (or the EXE itself) which votes against high memory, low memory is used by default for operations like malloc and calloc.
It seems that this approach was introduced in kLIBC for safety reasons — to make sure that high memory buffers are not passed to functions in DLLs which don't expect high memory. However, it makes little sense in the real life. First, there aren't kLIBC DLLs around that would actually need low memory buffers on purpose (at least, not to my knowledge). Second, this voting may be easily tricked by supplying a high memory buffer (allocated explicitly with _hmalloc or via DosAllocMem(OBJ_ANY)) even if the DLL votes for low memory. And this may happen even unintentionally (this detail is hidden inside the DLL code so it's impossible to tell which one you have unless you try to load it). Moreover, all cases when we observed real problems with high memory passed to low memory APIs have nothing to do with kLIBC DLLs and their voting: it was a directly allocated high mem buffer passed to a DLL which isn't kLIBC based and for which it was not known that it doesn't support low memory.
It turns out that this voting approach does more evil than good because most kLIBC DLLs work perfectly well with high memory, it's just their developer didn't care about which memory to use and didn't participate in voting with -Zhigh-mem. And this results in the whole application using low memory just because of that, even if the app EXE itself was built with -Zhigh-mem. It's even conceptually wrong — a single DLL should not override the application's decision to use a particular kind of memory by default. There is at least one real life example when it was the case: bitwiseworks/mozilla-os2#250 (comment).
All in all, it's much more efficient to handle such memory conflicts on a DLL by DLL basis. I.e., if it's a kLIBC DLL with the source code which deliberately wants low memory, it should use _lmalloc and friends directly (or it should use Safe Dos wrappers in kLIBC if it wants to call Dos APIs which are not high memory aware). If it's an EXE calling some DLL for which it's uncertain if it works with high memory, then the EXE should use _lmalloc directly (or use Safe Dos wrappers if it's about unsafe Dos calls).
This leads us to a conclusion that any application should use high memory for the default C heap when high memory is available at all. The only way to make it so w/o changing each application is to put code forcing high memory for the default heap in LIBCx (where we already have pre-main() hooks). This way, just linking against LIBCx will be enough for defaulting to the high memory heap — no matter how linked DLL vote.
Note that there is a respective kLIBC ticket by @komh that suggests to make -Zhigh-mem a default choice when not specified, but it's left w/o attention for 3 years now so there are little chances it will be made this way (and if it did, it would still be problematic as a DLL could still force the application's heap type with -Zlow-mem or such). Hence overriding it on the LIBCx level looks like the best solution.
This, however, may be desirable to disable this high memory override for testing purposes or in some special cases. For this I'm going to use an environment variable called LIBCX_HIGHMEM. This variable will take 4 possible values:
- 0: Low memory will be forced for the default C heap regardless of voting.
- 1: High memory will be forced for the default C heap regardless of voting.
- 2: Normal kLIBC voting will be used to detect the type of the default C heap (i.e. the old behavior, w/o LIBCx intervention).
- 3: Same as 1 but LIBCx will terminate the application at startup with a respective message to the console if there is any DLL that votes for low memory. This is a safety mode to ensure there are no "suspicious" DLLs among the statically linked ones.
The default value (i.e. if no LIBCX_HIGHMEM is present in the environment) will be 1 (force high memory). As this will work in most cases for most users.
BTW, there is one more proof that this voting system is useless. Actually in case of Firefox (bitwiseworks/mozilla-os2#250 (comment)) only the JS Shell (js.exe) was affected by the ICU libraries voting for low memory (because they were lacking -Zhigh-mem). The Firefox process itself (started by firefox.exe) was not affected and used high memory even with ICU libs voting for low mem — because ICU libs are statically linked to XUL.DLL, not to firefox.exe. And firefox.exe is not linked to XUL.DLL itself — it loads it dynamically. So this is yet another valid approach that will trick voting.
Done; seems to work. I will give it more local testing before finally releasing.