vmware-labs/wasm-workers-server

Add support for Zig

Angelmmiguel opened this issue · 2 comments

Angelmmiguel commented

Zig is a simple and general-purpose programming language. Recently, it's gaining more traction thanks to its benefits. Zig supports Wasm + WASI compilation natively and they use it extensively.

Since it can be compiled to WebAssembly, so there's no requirement of a language runtime like Python, Ruby or JavaScript. The goal for this task is to create the wrapper code to interact with Wasm Workers Server. The same approach we took for the Rust and Go support.

Technical requirements

These are the steps to add Zig support:

  1. Read the JSON metadata that comes via STDIN. Here you have the example from the Rust kit:

    wasm-workers-server/kits/rust/src/io.rs

    Lines 15 to 23 in 5631fb4

    pub struct Input {
    url: String,
    method: String,
    headers: HashMap<String, String>,
    body: String,
    kv: HashMap<String, String>,
    #[serde(default)]
    params: HashMap<String, String>,
    }

  2. Define the Request, Response and Cache entities. If it's available, it's better to use primitives from the Golang standard library or common usage libraries.

  3. Initialize a Request object from the JSON metadata.

  4. Initialize the Cache store.

  5. Call the worker function with the Request instance.

  6. Detect the content in the response. If it's a valid UTF-8 string, we can return. If it uses a different encoding, you need to encode it in base64.

  7. Transform the response and the Cache status into the expected JSON output. Note that the base64 flag is only enabled when the content is encoded.

    wasm-workers-server/kits/rust/src/io.rs

    Lines 59 to 65 in 5631fb4

    pub struct Output {
    data: String,
    headers: HashMap<String, String>,
    status: u16,
    kv: HashMap<String, String>,
    base64: bool,
    }

  8. Print the JSON output via STDOUT

Additional notes

  • The library should be defined in the kits/zig folder.
  • The documentation is available in the site: https://workers.wasmlabs.dev/docs/languages/introduction.
  • Currently, we're not publishing the libraries to language registries. Instead, we document how to get it from the GitHub repository.
voigt commented

Hey @Angelmmiguel,

I started working on this issue (see PR). I followed the Golang implementation as closely as possible. Unfortunately, I was not able to use Zig standard library for the Response entity (see issue). Request seems to be working.

The implementation is still quite hacky as I wanted to get the "break through" first; tests are missing as well.

This is what executing wws and curl http://127.0.0.1:8080/basic looks like at the moment:

$ zig build -Dtarget="wasm32-wasi" && WASMTIME_BACKTRACE_DETAILS=1 wws ./zig-out/bin/
⚙️  Preparing the project from: ./zig-out/bin/
⚙️  Loading routes from: ./zig-out/bin/
⏳ Loading workers from 2 routes...
✅ Workers loaded in 332.203667ms.
    - http://127.0.0.1:8080/basic
      => ./zig-out/bin/basic.wasm
    - http://127.0.0.1:8080/main
      => ./zig-out/bin/main.wasm
🚀 Start serving requests at http://127.0.0.1:8080

raw input json: {"url":"/basic","method":"GET","headers":{"sec-ch-ua":"\"Chromium\";v=\"115\", \"Not/A)Brand\";v=\"99\"","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"none","host":"127.0.0.1:8080","sec-ch-ua-platform":"\"macOS\"","sec-fetch-user":"?1","sec-fetch-mode":"navigate","dnt":"1","accept-encoding":"gzip, deflate, br","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","sec-ch-ua-mobile":"?0","sec-fetch-dest":"document","upgrade-insecure-requests":"1","accept-language":"en-US,en;q=0.9","connection":"keep-alive"},"body":"","kv":{},"params":{}}
Hello from function
output json: {"data":"Zig World!","status":200,"base64":false,"headers":{"content-type":"text/plain","x-generated-by":"wasm-workers-server"},"kv":{"hello":"world"}}
Done.
error while executing at wasm backtrace:
    0: 0x30ba - _start
                    at /Users/c.voigt/.asdf/installs/zig/0.11.0/lib/std/start.zig:209:42
[2023-08-14T11:52:06Z INFO  actix_web::middleware::logger] 127.0.0.1 "GET /basic HTTP/1.1" 503 47 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" 0.013806

Input & Output entities look good to me (please correct me if I'm wrong!). Output JSON is written to stdin as well as stderr (therefore, it is displayed in the log).

However, the wasm runtime doesn't exit gracefully, and I'm puzzled why. I assume this is the reason, why the request fails. Maybe you (@Angelmmiguel) have an idea?

Edit: After doing some research I understood that there are two ways a WASI module is executed: command- and reactor-mode. Seems like WWS requires reactor! It is working now, as I changed the compile configuration - I will now proceed with making things nice & polished.

  • 🎉1
Angelmmiguel commented

That's amazing @voigt ! Thank you very much for your contributions here! I'm looking to demo Zig support 🤩