browser support?
Closed this issue ยท 22 comments
Should this work on browser?
I have tried to use nottinygc in our WASI project, but it fails.
With the standard tinygo GC the execution runs successfully.
Tested using @wasmer/wasi npm package and also https://github.com/bjorn3/browser_wasi_shim
We are using tinygo 0.28 (current dev branch).
Any hint is welcome! Thanks.
Hi @p4u - to be honest browser support isn't a priority but with a wasi shim present, I don't see a reason it generally wouldn't work as the usage of wasi is relatively limited.
Can you confirm you are compiling with -scheduler=none as in the README? That stack trace looks somewhat familiar as what I see when forgetting the flag - currently unfortunately the scheduler isn't supported as I haven't been able to figure out how to debug the crash when not disabling it.
If you are using that flag, would you be able to share anymore details such as the whole tinygo invocation, or even the Go source code if it's public?
I did spend some time removing all go routines in the code cause I was compiling with scheduler=none (else tinygo crashes in compilation time).
The code is public yes, I'll polish it a bit and provide the repository link.
Thanks for your work.
I have been able to reproduce the same behavior with wasmtime.
I'm using the following command to compile on tinygo 0.28.1
tinygo build -target=wasi -opt=1 -gc=custom -tags="custommalloc nottinygc_finalizer" -scheduler=none -o artifacts/g16_prover.wasi wasi/main.go
Now the error has a trace that points to nottinygc@v0.3.0/gc.go:54:20
GC Warning: Out of memory - trying to allocate requested amount (4112 bytes)...
Error: failed to run main module `artifacts/g16_prover.wasi`
Caused by:
0: failed to invoke command default
1: error while executing at wasm backtrace:
0: 0x17369 - abort
at /home/p4u/repos/vocdoni/tinygo-fresh/lib/wasi-libc/libc-bottom-half/sources/abort.c:5:5
1: 0x1792d - sbrk
2: 0x1437 - <unknown>!GC_unix_get_mem
3: 0x1753 - <unknown>!GC_init_headers
4: 0xa899 - <unknown>!GC_init
5: 0xb0af - <unknown>!GC_generic_malloc_inner
6: 0xb3b6 - <unknown>!GC_generic_malloc
7: 0xb445 - <unknown>!GC_malloc
8: 0x1ce74 - runtime.alloc
at /home/p4u/go/pkg/mod/github.com/wasilibs/nottinygc@v0.3.0/gc.go:54:20
9: 0x2b328 - <unknown>!runtime.initAll
10: 0x2b163 - runtime.run
at /home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/scheduler_none.go:24:9 - _start
at /home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:21:5
2: wasm trap: wasm `unreachable` instruction executed
Thanks, it's good to hear we can repro outside browser as it'll be easier to debug. If it also repros with wazero, it has some nice debugging features that may help even more. Will be happy to give it a try if you're able to share a repo.
In the meantime, I have created a branch with bdwgc compiled with debug symbols, it might give some more info if you try building with it instead
https://github.com/wasilibs/nottinygc/tree/bdwgc-dwarf
go get github.com/wasilibs/nottinygc@04597e1e005659483aa197a30c68aeb8070d0e0f
That being said, the most interesting would be the line number within sbrk, unfortunate that for some reason we're getting the line number in wasi-libc's abort but not sbrk ๐ฟ
I have tried one guess though, if you can try it out that would be great
a17c521
go get github.com/wasilibs/nottinygc@a17c52165f0439687410f4344507c657487e16f2
Using 04597 and wazero
GC Warning: Out of memory - trying to allocate requested amount (4112 bytes)...
error instantiating wasm binary: module[] function[_start] failed: wasm error: unreachable
wasm stack trace:
.abort()
0x164e2: /home/p4u/repos/vocdoni/tinygo-fresh/lib/wasi-libc/libc-bottom-half/sources/abort.c:5:5 (inlined)
.sbrk(i32) i32
.GC_unix_get_mem(i32) i32
.GC_init_headers()
.GC_init()
.GC_generic_malloc_inner(i32,i32) i32
.GC_generic_malloc(i32,i32) i32
.GC_malloc(i32) i32
.runtime.alloc(i32) i32
0x1c016: /home/p4u/go/pkg/mod/github.com/wasilibs/nottinygc@v0.3.1-0.20230616010811-04597e1e0056/gc.go:54:20 (inlined)
.runtime.initAll()
._start()
0x2af72: /home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/scheduler_none.go:24:9 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/panic.go:52:7 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:21:5 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/interface.go:91:16 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:73:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:91:21 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1613:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1101:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1082:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1122:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1135:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:321:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:368:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:442:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1291:19 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1280:26 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1261:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1229:59 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:478:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:646:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:512:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:770:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:624:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:271:12 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/algorithm.go:615:8
Same output using a17c521 and wazero
GC Warning: Out of memory - trying to allocate requested amount (4112 bytes)...
error instantiating wasm binary: module[] function[_start] failed: wasm error: unreachable
wasm stack trace:
.abort()
0x164e2: /home/p4u/repos/vocdoni/tinygo-fresh/lib/wasi-libc/libc-bottom-half/sources/abort.c:5:5 (inlined)
.sbrk(i32) i32
.GC_unix_get_mem(i32) i32
.GC_init_headers()
.GC_init()
.GC_generic_malloc_inner(i32,i32) i32
.GC_generic_malloc(i32,i32) i32
.GC_malloc(i32) i32
.runtime.alloc(i32) i32
0x1c016: /home/p4u/go/pkg/mod/github.com/wasilibs/nottinygc@v0.3.1-0.20230616013945-a17c52165f04/gc.go:54:20 (inlined)
.runtime.initAll()
._start()
0x2af72: /home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/scheduler_none.go:24:9 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/panic.go:52:7 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:21:5 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/interface.go:91:16 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:73:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:91:21 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1613:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1101:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1082:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1122:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1135:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:321:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:368:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:442:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1291:19 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1280:26 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1261:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1229:59 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:478:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:646:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:512:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:770:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:624:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:271:12 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/algorithm.go:615:8
Let me know if there is anything else I can try. I wonder if the issue is related to mimalloc C library. Would it make sense to try an updated version of it?
I am preparing the repository, so you can try to reproduce. It is not easy because there are many repositories forked and modified, and even tinygo requires some small modifications (related with reflect package).
Thanks a lot for trying ๐
I did try updating mimalloc a little back and ran into this issue, let's see if anyone can help with it
But if your app is entirely TinyGo, you won't be exercising mimalloc at all so maybe not an issue. This library originated in code using TinyGo with re2 c++ library linked in to fix it's regexp support, in such scenarios mimalloc gets used.
Thanks for using wazero, it has a parameter to log function calls - it'll be a huge dump but I would use it to see what the parameter to the sbrk call is. If it seems like a tough task happy to try it after the repo becomes ready. Yeah I've been down the custom TinyGo road too ๐
Thanks for using wazero, it has a parameter to log function calls - it'll be a huge dump but I would use it to see what the parameter to the sbrk call is. If it seems like a tough task happy to try it after the repo becomes ready. Yeah I've been down the custom TinyGo road too sweat_smile
So, your work on nottinygc might be a game changer for us and all the people trying to use WASI+tinygo. Memory management is a big issue on browsers when using WASM. Thus, I'll do whatever is in my hands to help you debug this issue. What's the wazero parameter to log function calls?
But if your app is entirely TinyGo, you won't be exercising mimalloc at all so maybe not an issue. This library originated in code using TinyGo with re2 c++ library linked in to fix it's regexp support, in such scenarios mimalloc gets used.
Understood.
Apologies, while direct usage of wazero can log wasm function calls, it looks like the CLI only exposes host function logging. Let me see if this is something changable
https://github.com/wasilibs/wazero/blob/main/cmd/wazero/wazero.go#L185
On tinygo 0.28.1 I made some minor changes, here the diff.
You can find the full source on git clone https://github.com/vocdoni/tinygo.git -b wasi2.
commit 7bb8ff70e05e03a4844de475317b533a169650e1
Author: p4u <pau@dabax.net>
Date: Fri Jun 16 10:48:24 2023 +0200
testing wasi
Signed-off-by: p4u <pau@dabax.net>
diff --git a/lib/libclang_rt.builtins-wasm32.a b/lib/libclang_rt.builtins-wasm32.a
new file mode 100644
index 00000000..8a04c61f
Binary files /dev/null and b/lib/libclang_rt.builtins-wasm32.a differ
diff --git a/src/reflect/type.go b/src/reflect/type.go
index 22017900..12f3c2c1 100644
--- a/src/reflect/type.go
+++ b/src/reflect/type.go
@@ -949,7 +949,7 @@ func (t *rawType) AssignableTo(u Type) bool {
}
if u.Kind() == Interface {
- panic("reflect: unimplemented: AssignableTo with interface")
+ // panic("reflect: unimplemented: AssignableTo with interface")
}
return false
}
diff --git a/targets/wasi.json b/targets/wasi.json
index 4c43193f..8c1fcefd 100644
--- a/targets/wasi.json
+++ b/targets/wasi.json
@@ -16,7 +16,10 @@
],
"ldflags": [
"--stack-first",
- "--no-demangle"
+ "--no-demangle",
+ "{root}/lib/libclang_rt.builtins-wasm32.a",
+ "--initial-memory=2147483648",
+ "--max-memory=2147483648"
],
"extra-files": [
"src/runtime/asm_tinygowasm.S"
Then clone the code repository and checkout branch wasi2 (note I have added a vendor directory which also contains nottinygc)
git clone https://github.com/vocdoni/gnark-prover-tinygo.git -b wasi2
The main file that includes nottinygc is wasi/main.go.
With tinygo, binaryen, wazero, etc. Installed, you can run:
make prover-tinygo-g16-wasi-wazero
Once you have run this, artifacts are created so you can just try to compile the tinygo part:
tinygo build -target=wasi -opt=s -gc=custom -tags="custommalloc nottinygc_finalizer" -scheduler=none -o artifacts/g16_prover.wasi wasi/main.go
wazero run artifacts/g16_prover.wasiI hope you can reproduce it :) Let me know if you have an issue or I can help somehow.
Apologies, while direct usage of wazero can log wasm function calls, it looks like the CLI only exposes host function logging. Let me see if this is something changable
https://github.com/wasilibs/wazero/blob/main/cmd/wazero/wazero.go#L185
I could modify wazero source code locally and install a version with this flag always enabled.
Ah! I understood wrong the issue. So I'm using wazero cli for runing the WASI binary.
This is the output for wazero with hostlogging=all (maybe it helps)
wazero run --hostlogging=all artifacts/g16_prover.wasi ๎ฒ INT โ
--> ._start()
==> wasi_snapshot_preview1.environ_sizes_get(result.environc=54892,result.environv_len=54888)
<== errno=ESUCCESS
==> wasi_snapshot_preview1.fd_prestat_get(fd=3)
<== (prestat=,errno=EBADF)
==> wasi_snapshot_preview1.clock_time_get(id=monotonic,precision=1)
<== (timestamp=7416046245433,errno=ESUCCESS)
==> wasi_snapshot_preview1.clock_time_get(id=monotonic,precision=1)
<== (timestamp=7416046258808,errno=ESUCCESS)
==> wasi_snapshot_preview1.clock_time_get(id=monotonic,precision=1)
<== (timestamp=7416046260113,errno=ESUCCESS)
==> wasi_snapshot_preview1.clock_time_get(id=monotonic,precision=1)
<== (timestamp=7416046261331,errno=ESUCCESS)
==> wasi_snapshot_preview1.args_sizes_get(result.argc=65168,result.argv_len=65208)
<== errno=ESUCCESS
GC Warning: Out of memory - trying to allocate requested amount (4112 bytes)...
error instantiating wasm binary: module[] function[_start] failed: wasm error: unreachable
wasm stack trace:
.abort()
0x164e2: /home/p4u/repos/vocdoni/tinygo-fresh/lib/wasi-libc/libc-bottom-half/sources/abort.c:5:5 (inlined)
.sbrk(i32) i32
.GC_unix_get_mem(i32) i32
.GC_init_headers()
.GC_init()
.GC_generic_malloc_inner(i32,i32) i32
.GC_generic_malloc(i32,i32) i32
.GC_malloc(i32) i32
.runtime.alloc(i32) i32
0x1c016: /home/p4u/go/pkg/mod/github.com/wasilibs/nottinygc@v0.3.1-0.20230616013945-a17c52165f04/gc.go:54:20 (inlined)
.runtime.initAll()
._start()
0x2af72: /home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/scheduler_none.go:24:9 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/panic.go:52:7 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:21:5 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/interface.go:91:16 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:73:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/hashmap.go:91:21 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1613:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1101:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1082:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1122:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1135:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:321:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:368:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:442:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1291:19 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/env.go (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1280:26 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1261:22 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:1229:59 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:478:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:646:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:512:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:165:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:770:15 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/arch_tinygowasm.go:624:24 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/runtime_wasm_wasi.go:271:12 (inlined)
/home/p4u/repos/vocdoni/tinygo-fresh/src/runtime/algorithm.go:615:8
Thanks a lot for these repro directions! So I think a major issue is "--initial-memory=2147483648". A big difference between TinyGo's allocator and this one is that TinyGo assumes ownership of the entire heap, while here we request memory for usage, and things like mimalloc an also request from the heap, they each get pages that don't collide.
--initial-memory is the amount of memory reserved by the binary but is not about capacity - for a model where the allocator takes over the entire heap, it does effectively allow it to "preallocate", but otherwise it is more about how much memory is used by the data segment etc of the wasm binary. And when --initial-memory == --max-memory, for a system like nottinygc, that means there is no more memory left to request so it fails with out of memory. It is important for initial-memory to not be set when using nottinygc (I didn't think about this issue until debugging this so this looks like a good point to add to the docs, thanks).
For preallocating a fixed-size heap, a flag could be added to wazero or similar because in the end it is the runtime that sets up the heap, just like it would be the JVM doing it. It might also be possible for this library to provide a utility function to call on startup for it, and presumably even just something like _ = make([]byte, 2*1024*1024) would accomplish something similar.
So I tried removing the flag and now have this mysterious error I haven't seen before, though some logic does seem to be run
GC Warning: Failed to expand heap by 131072 KiB
GC Warning: Failed to expand heap by 131072 KiB
GC Warning: Failed to expand heap by 131072 KiB
GC Warning: Failed to expand heap by 64 KiB
GC Warning: Out of Memory! Heap size: 4012 MiB. Returning NULL!
Will keep on looking but wanted to give this update just in case it could spur any ideas too.
Looking at the traces closer, it does look like at startup, the heap is growing up to 2GB, which is the limit causing the Failed to expand heap. When I remove --max-memory, then it goes up to 4GB, which allows it to get past startup, but then still fails at that limit.
While I have seen cases where TinyGo's default allocator will have fragmentation causing out-of-heap where nottinygc fixes the problem, it doesn't mean things are ideal with nottinygc either. Just want to confirm, does this sound like a problem with the garbage collector or is it possible the code itself has a memory leak unrelated to GC?
BTW, bdwgc, the library this wraps, has many environment variables which work just fine with -env option to wazero run, for example GC_PRINT_STATS
Also, it looks like it already has a GC_INITIAL_HEAP_SIZE option which may be what you need for presizing, though first we need to get it to actually work before optimizing :)
Looking at the traces closer, it does look like at startup, the heap is growing up to 2GB, which is the limit causing the
Failed to expand heap. When I remove--max-memory, then it goes up to 4GB, which allows it to get past startup, but then still fails at that limit.While I have seen cases where TinyGo's default allocator will have fragmentation causing out-of-heap where nottinygc fixes the problem, it doesn't mean things are ideal with nottinygc either. Just want to confirm, does this sound like a problem with the garbage collector or is it possible the code itself has a memory leak unrelated to GC?
The code performs hundreds of thousands of arithmetic operations. For doing so, it stores in memory enormous tables of uint64[4]. There might be a leak, but I think it is just the normal behavior. It might be that 4 GiB is not enough.
I used initial/max memory tinyGo flags because in my understanding there is some issue with memory expansion in the browser. For some reason, if I don't set --min-memory to the maximum required amount for the heap memory, the program stops its execution when the min is reached.
However, if nottinygc can properly expand the memory on browser, I don't need to set the minimum. The code will be a little slower because it requires to expand, but should not be relevant.
Finally, it might be that the current code requires more than 4 GiB. In that case, it cannot be executed on browsers. I would need to perform some changes and optimizations. On previous tests, I achieved 1.8 GiB, so should be possible.
Now that I know the min-memory issue (IMHO it would be ideal if nottinygc crashes in compilation time if min-memory is set by tinyGo), I will try again with the previous code and see if it works.
I greatly appreciate your help.
So, I can confirm there is some issue when using nottinygc regarding the memory allocation. And this seems a deeper problem.
I have rolled back to the more optimized code I have. This code has been tested on several browsers and works.
Without using nottinygc, I'm able to execute the code and the runtime allocates up to 1.74 GiB (I have profiled it, but I can also see the number using the top command during the execution).
File: g16_prover.wasi
Type: alloc_space
Time: Jun 19, 2023 at 10:03am (CEST)
Duration: 21.70s, Total samples = 1.74GB
Same code using nottinygc (removing the --min and --max memory flags of tinyGo).
Memory allocation goes up to 18 GiB and crashes GC Warning: Out of memory - trying to allocate requested amount (16384 bytes).... Probably it reaches some operating system limit.
In my opinion, there is some issue with nottinygc related to how the memory is liberated. It might be that it's not "fast" enough and the heap grows without opportunity to decrease? Maybe you never tested before with a program that performs thousands of operations and allocates so much memory at once?
I'm not certain how to proceed from this point. What do you think?
I have updated the repositories so the version of the code that consumes 1.74 GiB can be tested. Just in case it is useful for something...
You would need to add these three commits on top of the current tinyGo dev branch: https://github.com/vocdoni/tinygo/tree/wasi3
Then you can checkout wasi_nottinygc branch of gnark-prover-tinygo and execute the test.
git clone https://github.com/vocdoni/gnark-prover-tinygo -b wasi_nottinygc
make prover-tinygo-g16-wasi-wasmtime
Thanks for updating the repo @p4u - I will investigate a little more later this week. I'm not confident I will find anything great, but let's see. Unfortunately bdwgc is also a conservative GC, so while it seems to perform much better than the TinyGo default GC, it may end up using extra memory anyways in this sort of application. I will try running the code with some bdwgc debug variables to see if it's possible to understand the behavior better.
Hi @p4u - just did a brief look again and noticed the crash stack trace showing regexp
<Go type>:156:32 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/regexp.go:872:15 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/regexp.go:191:14 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/syntax/parse.go:71:29
.regexp.MustCompile(i32,i32) i32
0x5b303: /opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/regexp.go:137:16 (inlined)
/Users/anuraag/git/tinygo/src/runtime/runtime.go:76:22 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/sort/sort.go:26:21 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/regexp.go:317:24 (inlined)
<Go type>:156:32 (inlined)
/opt/homebrew/Cellar/go/1.20.5/libexec/src/regexp/syntax/parse.go:71:29
.runtime.initAll()
._start()
0x49dd8: /Users/anuraag/git/tinygo/src/runtime/scheduler_none.go:24:9 (inlined)
/Users/anuraag/git/tinygo/src/runtime/runtime.go:76:22 (inlined)
/Users/anuraag/git/tinygo/src/runtime/extern.go:26:21 (inlined)
I couldn't git grep regexp in your repo but perhaps a dependency uses it. I have found regexp with tinygo to be mostly unusable even with nottinygc, it can even fail to compile some expressions outright. It might be tricky if it's being used by a dependency, but would you be able to give go-re2 a try if the app uses regex heavily?
https://github.com/wasilibs/go-re2
Of course, it may just be coincidence that regexp "broke the camel's back" but isn't actually the heavyweight part of the app, not sure if your app uses it heavily or not.
Hi @p4u - it's been a long time I don't know if you are still looking at this. After seeing another similar repro, I had some hints and have made a patch. Basically
- Large amounts of random data
- Using most of the 32-bit limit
cause the default conservative GC to fail because the probability of misidentifying pointers goes up tremendously. bdwgc has a note about it here
So I have started adding type information, starting with only arrays
There is more that can be done, but I believe arrays are probably the most significant issue so if you still are interested, perhaps you can try the latest version and see if it helps you? For context, I could confirm it fixes the issue with random embedded files (i.e., large byte arrays of random data in global variables) reported in corazawaf/coraza-proxy-wasm#234