wasmerio/wasmer

Fuzz tests are broken

Opened this issue · 1 comments

annenkov commented

Describe the bug

Running fuzz tests results in errors. It looks like the errors are not the result of actual implementation bugs, but rather issues related to tests themselves.

These errors do not seem to be specific for each target. However, they are provided according to which targets they were discovered in.

equivalence_universal, deterministic
Does not compile with several errors.

error[E0432]: unresolved import `wasmer::Val`
 --> fuzz/fuzz_targets/equivalence_universal.rs:7:79
  |
7 | use wasmer::{imports, CompilerConfig, EngineBuilder, Instance, Module, Store, Val};
  |                                                                               ^^^ no `Val` in the root

warning: unused imports: `CompilerConfig`, `EngineBuilder`, `Module`, `Store`, `imports`
 --> fuzz/fuzz_targets/equivalence_universal.rs:7:14
  |
7 | use wasmer::{imports, CompilerConfig, EngineBuilder, Instance, Module, Store, Val};
  |              ^^^^^^^  ^^^^^^^^^^^^^^  ^^^^^^^^^^^^^            ^^^^^^  ^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

error[E0061]: this method takes 1 argument but 0 arguments were supplied
   --> fuzz/fuzz_targets/equivalence_universal.rs:151:18
    |
151 |             if f.ty().params().is_empty() {
    |                  ^^-- an argument of type `&_` is missing
    |
note: method defined here
   --> /home/danil/Projects/wasmer/lib/api/src/externals/function.rs:220:12
    |
220 |     pub fn ty(&self, store: &impl AsStoreRef) -> FunctionType {
    |            ^^
help: provide the argument
    |
151 |             if f.ty(/* store */).params().is_empty() {
    |                    ~~~~~~~~~~~~~

error[E0061]: this method takes 2 arguments but 1 argument was supplied
   --> fuzz/fuzz_targets/equivalence_universal.rs:152:32
    |
152 |                 let result = f.call(&[]);
    |                                ^^^^ --- an argument of type `&mut _` is missing
    |
note: method defined here
   --> /home/danil/Projects/wasmer/lib/api/src/externals/function.rs:297:12
    |
297 |     pub fn call(
    |            ^^^^
help: provide the argument
    |
152 |                 let result = f.call(/* store */, &[]);
    |                                    ~~~~~~~~~~~~~~~~~~

universal_cranelift, metering, universal_llvm

Example output (the wasm code varies a bit, but the error is still the same)

thread '<unnamed>' panicked at fuzz/fuzz_targets/universal_cranelift.rs:45:51:
called `Result::unwrap()` on an `Err` value: Validate("constant expression required: global.get of locally defined global (at offset 0x1c)")
...

Failing input:

        fuzz/artifacts/universal_cranelift/crash-d23758903381c6d20ca6f965a4b9fd29fb67b65f

Output of `std::fmt::Debug`:

        (module
          (global (;0;) i32 i32.const -2086588512)
          (global (;1;) i32 global.get 0)
        )

This behaviour (validation error) seems to be valid since constant expressions are only allowed to refer to imported globals. However, the tests stop at this point, preventing any progress. A possible cause of it could be an updated wasm-smith dependency, since wasm-smith is supposed to generate valid code.

universal_singlepass

thread '<unnamed>' panicked at fuzz/fuzz_targets/universal_singlepass.rs:51:13:
Validation error: func type returns multiple values but the multi-value feature is not enabled (at offset 0xb)
...
Failing input:

        fuzz/artifacts/universal_singlepass/crash-dfbb59b3f49ee30734e54f339b0e59727cb0e9ee

Output of `std::fmt::Debug`:

        (module
          (type (;0;) (func (result f32 i32)))
        )

This should be caught by this condition, but the error I get is slightly different from the ones being checked.

Configuration:

wasmer 4.2.5 (0460050 2023-12-23)
binary: wasmer-cli
commit-hash: 04600507844bae978239531c8e6265d410aefe6d
commit-date: 2023-12-23
host: x86_64-unknown-linux-gnu
compiler: singlepass,cranelift,llvm
rustc 1.70.0 (90c541806 2023-05-31)
binary: rustc
commit-hash: 90c541806f23a127002de5b4038be731ba1458ca
commit-date: 2023-05-31
host: x86_64-unknown-linux-gnu
release: 1.70.0
LLVM version: 16.0.2

Steps to reproduce

  1. Clone the repository.
  2. Run the following command in the root of the repository (the concrete fuzz test name and features vary depending on fuzz target):
cargo +nightly fuzz run universal_cranelift --features="universal cranelift"
  1. See error.

Expected behavior

Tests should run for some extended period of time without failing on input that is considered valid.

Actual behavior

equivalence_universal does not compile, other tests fail early for possibly wrong reasons.

Additional context

Similar failures were reported by the OSS-Fuzz project: