runtime: mmap(0xc000000000, ...) fails on arm64
davecheney opened this issue · 10 comments
The first action of the runtime when it starts up is to mmap 64k at the start of the heap. This appears to fail, causing the runtime to spend a lot of time fretting over this allocation. I've verified on the other 64bit platforms that this first mmap always succeeds and believe that the failure path and fallback code may not be well tested.
arm64:
mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f81eb3000 // not the allocation it was expecting
mincore(0xc000000000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory) // runtime fishes around, thinking the allocation was close, but not exact
mincore(0xc000001000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000002000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000003000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000004000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000005000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000006000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000007000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000008000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc000009000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000a000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000b000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000c000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000d000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000e000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0xc00000f000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory) // nope, really wasn't what it wanted
munmap(0x7f81eb3000, 65536) = 0 // gives up, returns the memory
mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) // trys to allocate the memory with MAP_FIXED, still fails
mmap(0x1c000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f81eb3000 // tries a little higher in the VA range, still no dice
mincore(0x1c000000000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
mincore(0x1c000001000, 4096, 0x54b32f) = -1 ENOMEM (Cannot allocate memory)
Long story short, @mwhudson kindly reminded me that arm64 kernels that use 4k pages are limited to a VA heap of 512gb, or a 39 bit heap. 0xc000000000, needs 40 bits ...
This is where I put the heap for gccgo: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;a=commitdiff;h=ab46d63c29339633929988bcc8d92eb1117c7c0a
For the record, this address space limit is lifted in version 3.17 of the kernel. So possibly we'll only really have to care about this for a fairly short amount of time.
I'm going to apply your fix for the moment.
- I can't control the kernel on rugby, nor can @4ad on his machine
- saying, run a newer kernel, won't really fly
- we can just bodge this until we have to integrate upstream.
Oh, sure, I didn't mean to imply we could ignore this, more that bodging it sounds perfectly fine to me.
Fixed in efc7898
On Fri, Feb 13, 2015 at 3:19 PM, Michael Hudson-Doyle <
notifications@github.com> wrote:
Oh, sure, I didn't mean to imply we could ignore this, more that bodging
it sounds perfectly fine to me.—
Reply to this email directly or view it on GitHub
https://github.com/4ad/go/issues/104#issuecomment-74203148.
For the record, this address space limit is lifted in version 3.17 of the kernel.
Actually it's more complicated than that. When using 4k pages, the available the address space is determined by the numbers of translation buffers, 3 or 4. With 3 translation buffers we get 40 bits, with 4 translation buffers we get 48 bits. More translation buffers means less performance, so I expect no distro will ever ship with 4 translation buffers enabled.
When using 64k pages, we can only have 2 or 3 translation buffers giving 43 bits or 48 bits.
64k pages are nice, and we must make sure Go works with 64 k pages.
When using 64k pages, we can only have 2 or 3 translation buffers giving
43 bits or 48 bits.64k pages are nice, and we must make sure Go works with 64 k pages.
The good news there is Minux and Austin (and Doku) did all the hard work
there because ppc64le only comes in 64kb flavours.
I can't control the kernel on rugby, nor can @4ad on his machine
Oh yes I can, I have both a patched kernel and a patched bootloader. I run a patched kernel with uprobes and other fixes.
Unfortunately running my tracer/ktap/systemtap require custom kernel, custom ktap, newer binutils.
The rest of us are sadly just mortals ...
On Fri, Feb 13, 2015 at 9:00 PM, Aram Hăvărneanu notifications@github.com
wrote:
I can't control the kernel on rugby, nor can @4ad https://github.com/4ad
on his machineOh yes I can, I have both a patched kernel and a patched bootloader. I run
a patched kernel with uprobes and other fixes.Unfortunately running my tracer/ktap/systemtap require custom kernel,
custom ktap, newer binutils.—
Reply to this email directly or view it on GitHub
https://github.com/4ad/go/issues/104#issuecomment-74230445.
Just as an aside. This machine support big-endian too, and I made and booted a big-endian kernel! Unfortunately, there is no big-endian userspace availabe from nobody.
yawn byte order is just a frame of mind.
On Fri, Feb 13, 2015 at 9:02 PM, Aram Hăvărneanu notifications@github.com
wrote:
Just as an aside. This machine support big-endian too, and I made and
booted a big-endian kernel! Unfortunately, there is no big-endian userspace
availabe from nobody.—
Reply to this email directly or view it on GitHub
https://github.com/4ad/go/issues/104#issuecomment-74230630.