(0.10-beta) send+module: invoking lua functions from another thread

[alemidev]

i'm developing a nvim plugin which is mostly a thin wrapper around a native library. such library works in background and wants to invoke callbacks to do arbitrary work on-demand for new events.

a minimal example of my use case:

# Cargo.toml
[package]
name = "example-mlua"
version = "0.1.0"
edition = "2021"

[lib]
name = "mluaexample"
crate-type = ["cdylib"]

[dependencies]
mlua = { version = "0.10.0-beta.1", features = ["luajit", "send", "module"] }

// lib.rs
use mlua::prelude::*;

#[mlua::lua_module]
fn libmluaexample(lua: &Lua) -> LuaResult<LuaTable> {
  let exports = lua.create_table()?;

  exports.set("callback", lua.create_function(|_, (cb,):(LuaFunction,)| {
    std::thread::spawn(move || {
      let mut count = 0;
      loop {
        std::thread::sleep(std::time::Duration::from_millis(200));
        if cb.call::<(String,),()>((format!("calling callback: {count} errors"),)).is_err() {
          count += 1;
        }
      }
    });
    Ok(())
  })?)?;

  Ok(exports)
}

in addition to this, nvim exposes a lua interface to libuv's async handle, which is safe to invoke across threads and will wake the main loop thread to run the wrapped function

-- your neovim init.lua

local native = require('libmluaexample')
local async = vim.loop.new_async(function() print("something") end)
native.callback(function() async:send() end)

with this setup the callback given to the foreign thread should only interact with the async handle, which should be threadsafe. however after requiring the native module neovim will consistently crash after not many invocations.

---

since this is the new beta built this may be a temporary or already known bug, so feel free to close.

also i'm not sure this is supported use at all, i see with the send feature enabled lua values hold a mutexed weak ref, but it's not clear to me how this mutex is handled when mlua is a module

if my usage is unsupported, is it possible to clear the callback environment or run it in another context to call it safely? is it possible to lock the lua state and allow callbacks to execute safely from other threads?

anyhow, thanks for the wonderful library and your time, looking forward to mlua 0.10!

[khvzak]

Unfortunately send feature flag is not compatible with module mode.
In module mode mlua does not control access to the Lua VM and cannot guarantee that lock is always acquired before accessing lua api.

One of possible solution can be using channels for communicating between threads and invoking callback when a value is present in the channel (checking periodically).

[alemidev]

Thanks for your reply, sorry for taking a while to get back at you!

I think it could be possible to handle the state mutex "the other way around": acquiring it when leaving module code and releasing it as soon as we enter module code (basically as first step of every mlua function). This would still mean that the main Lua thread needs to yield from time to time but it can either be done manually or periodically with a set_hook() invocation. I may try to do this sooner or later!

---

In the meantime I'm doing as you suggest: keeping a global static channel where i send my LuaFunctions and exposing a poll_callback() function to Lua, which returns an Option<LuaFunction> to invoke. I'm struggling to pass function arguments, as IntoLuaMulti is Sized and can't be dynamically dispatched, but I think a solution is possible.

However, since beta.2 released, this workaround doesn't compile anymore: LuaFunctions need to be Send+Sync to be passed on the channel, but send and module are incompatible. I think my use case is safe: the callbacks are only executed by Lua main thread, and so will never run concurrently or after Lua ended, but beta.2 won't allow (due to Cargo resolution order beta.2 is picked over beta.1, also breaking projects relying on beta.1 and send+module).

You can check our project for context: build with --features=lua54, relevant mlua file is here

Would you be open to either drop this restriction or allow something like unsafe-send feature which is compatible with module?