Integrate standard wasm tests
icefoxen opened this issue · 8 comments
wabt has tests for the interpreter, here. They're run with the wabt interpreter and use an extended form of wat syntax.
In fact it appears the better source for this would be the reference interpreter tests, which appear to be in a similar form and might be easier to integrate.
Okay, so the extended wat syntax is annoying but necessary, so there are a few potential solutions to it:
- parity-wasm apparently uses the
wabtassembler with some magic flags that dump out a JSON spec file along with the wasm modules to test, then parses the spec and makes it do its thing. - Or we can write our own parser for the wat test syntax.
The first seems easier; the magic incantation seems to be wast2json infile.txt -o outfile.json and it will just do what we want. Parsing json is trivial with serde and we can steal it from parity-wasm anyway, so.
By "extended syntax" do you mean the bit in the (;; STDOUT ;;) block at the bottom of each file? I had a look at a few of them and I think that this one leads me to believe that this block is just being treated as text to compare against the stdout of some test harness. The test harness is, I think, just taking every exported nullary function and running them all.
I could probably rig up something similar that just reports test pass/fail based on diffing the expected/actual stdout texts, I'll have a try.
Oh, actually we can probably just nobble the existing test scripts to work with your code instead.
(They are indeed doing stdout diffing.)
That's more or less what I'm doing, or at least was the original plan. The standard test scripts use wabt's wat2wasm with a flag that tells it to assemble the actual modules and dump out a json specification of how to test each module. This specification is easy to load with serde (esp. if you steal parity-wasm's code for them) and then you just need the actual runner code.
This is in progress, see bin/nanowasm.rs. I considered making a separate crate for the specification loading but it doesn't yet seem worthwhile.
Ah cool, I think I misunderstood what you meant about the JSON. If there are any boring bits that need done I may have some time over the weekend to contribute something?
Sure, if you want. Some things that needs doing is just making it actually try to load and run more of the tests in spec/run-nanowasm-tests.py, or handle more tests cases nicely in bin/nanowasm.rs, or refactor the test handling out... maybe make the nanowasm binary its own sub-crate.
Oh and it turns out there's a wabt binding crate, though there's some work that needs doing before it supports wast2json.