Runtime Error: Out of Bounds Memory Access

[torch2424]

Hello! So at Fastly we're currently debugging some code from a user of Compute@Edge, and we have determined that we found a bug in the AS runtime.

We’re seeing a crash in the collector, and this is the stack trace that we get with ASC_RTRACE=1:

WebAssembly trapped: wasm trap: out of bounds memory access
wasm backtrace:
    0:  0x890 - <unknown>!~lib/rt/itcms/__visit
    1:  0xcb8 - <unknown>!~lib/rt/itcms/step
    2: 0x1100 - <unknown>!~lib/rt/itcms/__new
    3: 0x6fbb - <unknown>!~lib/string/String.UTF8.encode
    4: 0x13a6 - <unknown>!~lib/as-wasi/as-wasi/Descriptor#writeString
    5: 0x141a - <unknown>!~lib/as-wasi/as-wasi/Console.log
    6: 0x5a82 - <unknown>!assembly/[Omitted for privacy, essentially is a logging function 😄 ]
    7: 0x5fb5 - <unknown>!assembly/index/main
    8: 0x644e - <unknown>!start:assembly/index
    9: 0x313e - <unknown>!~start

We're still doing an investigation to nail this down on our end a bit more, and I'll try and look into trying to make a reproducible test case. But! Any tips to help us investigate would be appreciated! Thanks! 😄

cc @dcodeIO @MaxGraey since y'all work on the runtime the most 😄

[peterhuene]

More context:

rt/itcms/__visit is reading out of bounds at 0xffffffed, which means a value of 1 was on the shadow stack given a TOTAL_OVERHEAD of 20 bytes (i.e. 1 - 20 = 0xffffffed).

Oddly enough, I found as-wasi/as-wasi/Console.log (0.4.5) to have the following code:

(func $~lib/as-wasi/as-wasi/Console.log (param $0 i32)
    global.get $~lib/memory/__stack_pointer
    i32.const 4
    i32.sub
    global.set $~lib/memory/__stack_pointer
    call $~stack_check
    global.get $~lib/memory/__stack_pointer
    i32.const 0
    i32.store
    global.get $~lib/memory/__stack_pointer
    i32.const 1
    i32.store
    i32.const 1
    local.get $0
    call $~lib/as-wasi/as-wasi/Descriptor#writeString
    global.get $~lib/memory/__stack_pointer
    i32.const 4
    i32.add
    global.set $~lib/memory/__stack_pointer
)

Note this code specifically that follows a write of 0 to the same address:

global.get $~lib/memory/__stack_pointer
i32.const 1
i32.store

Is that ever correct? I can't seem to find any other places where a non-zero constant is stored on the shadow stack like this.

Signs pointing to a asc bug?

[dcodeIO]

Makes me wonder if this is from as-wasi changetypeing raw fds to managed Descriptor objects. That would explain the 1, in that Stdout(): Descriptor { return new Descriptor(1); }; would be on the shadow stack. If this assumption is correct, changing Descriptor in as-was to be @unmanaged could solve this.

[peterhuene]

That does fix the issue. @torch2424 can you move this issue over to the as-wasi repo for them to properly decorate Descriptor such that it's not tracked by the GC?

[dcodeIO]

Fwiw, we also have a console implementation in the standard library now. Let me know if there is something else you'd need that as-wasi supports but our standard library doesn't yet :)

[peterhuene]

Thank you, can also confirm that using console from stdlib also solves the issue as it bypasses the problematic code in as-wasi.

[torch2424]

Totally! I can move this over to the as-wasi repo! Thank you so much for the help here @dcodeIO and @peterhuene ! 😄 Closing this one out 🎉