microsoft/aici

support native controllers

mmoskal opened this issue · 0 comments

In some cases the controllers are trusted and performance is particularly important. In these cases we should be able to deploy a natively compiled controller. We would need a library that exposes AICI host interface and communicates with aicirt just like normal "wasm-runnner" does.

This is particularly important with speculative decoding