Extism JavaScript PDK

Note: This is very experimental. If you are interested in helping or following development, join the #js-pdk room in our discord channel.

Overview

This PDK uses QuickJS and wizer to run javascript as an Extism Plug-in.

This is essentially a fork of Javy by Shopify. We may wish to collaborate and upstream some things to them. For the time being I built this up from scratch using some of their crates, namely quickjs-wasm-rs.

How it works

This works a little differently than other PDKs. You cannot compile JS to Wasm because it doesn't have an appropriate type system to do this. Something like Assemblyscript is better suited for this. Instead, we have compiled QuickJS to Wasm. The extism-js command we have provided here is a little compiler / wrapper that does a series of things for you:

It loads an "engine" Wasm program containing the QuickJS runtime
It initializes a QuickJS context
It loads your js source code into memory
It parses the js source code for exports and generates 1-to-1 proxy export functions in Wasm
It freezes and emits the machine state as a new Wasm file at this post-initialized point in time

This new Wasm file can be used just like any other Extism plugin.

Install the compiler

We now have released binaries. Check the releases page for the latest.

Note: Windows is not currently a supported platform, only mac and linux

Install Script

curl -O https://raw.githubusercontent.com/extism/js-pdk/main/install.sh
sh install.sh

Then run command with no args to see the help:

extism-js
error: The following required arguments were not provided:
    <input>

USAGE:
    extism-js <input> -o <output>

For more information try --help

Note: If you are using mac, you may need to tell your security system this unsigned binary is fine. If you think this is dangerous, or can't get it to work, see the "compile from source" section below.

Try it on a script file. Name this `script.js:

Note: You must use CJS Module syntax when not using a bundler.

// script.js

const VOWELS = [
    'a', 'e', 'i', 'o', 'u',
]

function count_vowels() {
    let input = Host.inputString()
    let count = 0
    for (let i = 0; i < input.length; i++) {
        if (VOWELS.includes(input[i].toLowerCase())) {
            count += 1
        }
    }
    Host.outputString(JSON.stringify({count}))
    return 0
}

module.exports = {count_vowels}

extism-js script.js -o count_vowels.wasm
extism call count_vowels.wasm count_vowels --input="Hello World!" --wasi
# => {"count":3}

Using with a bundler

The compiler cli and core engine can now run bundled code. You will want to use a bundler if you want to want to or include modules from NPM, or write the plugin in Typescript, for example.

There are 2 primary constraints to using a bundler:

Your compiled output must be CJS format, not ESM
You must target es2020 or lower

Using with esbuild

The easiest way to set this up would be to use esbuild. The following is a quickstart guide to setting up a project:

# Make a new JS project
mkdir extism-plugin
cd extism-plugin
npm init -y
npm install esbuild --save-dev
mkdir src
mkdir dist

Add esbuild.js:

const esbuild = require('esbuild');

esbuild
    .build({
        entryPoints: ['src/index.js'],
        outdir: 'dist',
        bundle: true,
        sourcemap: true,
        minify: false, // might want to use true for production build
        format: 'cjs', // needs to be CJS for now
        target: ['es2020'] // don't go over es2020 because quickjs doesn't support it
    })

Add a build script to your package.json:

{
  "name": "extism-plugin",
  // ...
  "scripts": {
    // ...
    "build": "node esbuild.js && extism-js dist/index.js -o dist/plugin.wasm"
  },
  // ...
}

Let's import a module from NPM:

npm install --save fastest-levenshtein

Now make some code in src/index.js. You can use import to load node_modules:

Note: This module uses the ESM Module syntax. The bundler will transform all the code to CJS for us

import {distance, closest} from 'fastest-levenshtein'

// this function is private to the module
function privateFunc() { return 'world' }

// use any export syntax to export a function be callable by the extism host
export function get_closest() {
  let input = Host.inputString()
  let result = closest(input, ['slow', 'faster', 'fastest'])
  Host.outputString(result + ' ' + privateFunc())
  return 0
}

# Run the build script and the plugin will be compiled to dist/plugin.wasm
npm run build
# You can now call from the extism cli or a host SDK
extism call dist/plugin.wasm get_closest --input="fest" --wasi
faster World

Compiling the compiler from source

Prerequisites

Before compiling the compiler, you need to install prerequisites.

Install Rust using rustup
Install the WASI target platform via rustup target add --toolchain stable wasm32-wasi
Install the wasi sdk using the makefile command: make download-wasi-sdk
Install CMake (on macOS with homebrew, brew install cmake)

Compiling from source

Run make to compile the core crate (the engine) and the cli:

make

To test the built compiler (ensure you have Extism installed):

./target/release/extism-js bundle.js -o out.wasm
extism call out.wasm count_vowels --wasi --input='Hello World Test!'
# => "{\"count\":4}"

Why not use Javy?

Javy, and many other high level language Wasm tools, assume use of the command pattern. This is when the Wasm module only exports a main function and communicates with the host through stdin and stdout. With Extism, we have more of a shared library interface. The module exposes multiple entry points through exported functions. Furthermore, Javy has many Javy and Shopify specific things it's doing that we will not need. However, the core idea is the same, and we can possibly contribute by adding support to Javy for non-command-pattern modules. Then separating the Extism PDK specific stuff into another repo.

What needs to be done?

Implemented so far:

Host.inputBytes
Host.inputString
Host.outputBytes
Host.outputString
Var.get
Var.set
console.log
console.error
throw Error

The above are implemented but need some more validation and resilience built into them. debating whether I should implement the bulk of the code in js or rust. Working on implementing the other pdk methods.

I've got the exports to work, but it's a fragile and complicated solution. Will write it up soon, and maybe it can be replaced with something simpler.

Code-TA/js-pdk