Support for luvit?

[jed]

I'm wondering whether you have considered adding luvit support, now that a stable branch exists. Any thoughts?

[nrk]

This is something I'd definitely consider for redis-lua 3.0 since it's pretty much impossible to adapt the current library to add some kind of support for luvit (the next major version of redis-lua is going to be a complete rewrite of the library anyway). I will give it some thoughts in the next few days.

[agladysh]

Maybe it is luvit should provide support redis-lua?

That separate ecosystem thing is disturbing.

[nrk]

@agladysh just to clarify, instead of making redis-lua dependant on luvit I was actually thinking about having a separate module (something like require('redis.luvit')) that could reuse code from the standard client where / if possible. It's just a thought anyway, right now I don't really have any idea if it's worth the effort but what's sure is that the main target of redis-lua will remain the plain Lua interpreter.

[agladysh]

@nrk Let me rephrase: What makes luvit so special that it warrants a dedicated module in a generic library like redis-lua?

[jed]

thanks, @nrk. I was thinking something like that, or at least splitting the protocol code off from the client code into something platform agnostic, and then writing a luvit-specific client (which i could definitely help with).

[ignacio]

I patched redis-lua so I could provide a LuaSocket implementation that was backed by LuaNode and it worked fine, except for PUB/SUB. Maybe both LuaNode and Luvit should provide something that looks like luasockets api and a mechanism to simulate synchronism.

[twojcik]

+1, I'd love to see support for luvit!

[bortels]

To answer @agladysh - luvit is an attempt to replicate the node.js architecture in lua. The key point of that architecture is that blocking calls are forbidden; rather, you make a request, and register a callback with that request, and when the data is ready you get your callback. By doing so, you can run a single thread of execution (or one per CPU) - no context switching whatsoever, which means no context switching overhead. With modern epoll/kqueue network handling, this makes a server that can be remarkably fast and concurrent - speedy responses to tens of thousands of connections is common - without that much extra complexity (beyond what you'd need to do the same thing with threads, for example).

Drawback is - you're breaking compatibility with blocking libraries, on purpose. Or, to put it another way, if something blocks, it blocks EVERYTHING. So - we look for a non-blocking API for the library, something that supports callbacks.

The reason we look for async support in the library, rather than in luvit, is that knowing what exactly to do to allow async calls will differ from library to library - the people who would know best are the people who, in this case, are working directly on redis-lua. In this case, nrk pointed out that it may be very difficult to retrofit this sort of support into the current version, and he'd consider it for 3.0 - which, incidentally, is cool, thank you for considering it @nrk.

[agladysh]

Support for asynchronous calls is good and I have nothing against that. But if it is all what is required, it should not be called "support for luvit", and it should be done in a generic manner, so everyone, not just luvit users, would benefit.

[bortels]

Can you name a group who would need/want async calls other than luvit users? I am unaware of any other projects that want non-block async libraries, other than node.js, and they already have nice async redis modules. I think you're splitting hairs - everyone who wants this (right now) is luvit users - the two groups are the same. Besides - I doubt very much it wouldn't be done "in a generic manner" anyway - I'm not even sure what a non-generic manner would be...

[agladysh]

@bortels Well, my company, for instance is interested in switching to async libraries, all that is stopping us is lack of libraries that support that. And we're not using luvit and do not plan to.

[bortels]

And - to answer my own question, luanode is a non-uv-based "node clone" using lua, and would want the same async support - except there's already a redis library for it using their core libraries.

[twojcik]

Hi,
I'm working in luvit-redis lib, it's available here https://github.com/twojcik/luvit-redis and it wraps hiredis c lib - it's async part. It's still work in progress and lot of things need to be done (see TODO.md), but it works (except multibulk replies ).

[nrk]

Thanks guys for all the feedback! After reading your comments and giving it a thought I think it's probably better to leave implementations for async-driven redis client libraries for specific runtimes such as LuaNode and Luvit to dedicated projects. LuaNode already has a module for redis, as pointed by @bortels and @ignacio himself, and @twojcik has started working on a module for Luvit.

Obviously this doesn't change the fact that a way to perform async calls on plain Lua / LuaJit is still planned for the next major version of redis-lua and, by the way, it would be cool to get some help from someone used to develop async stuff in Lua as soon as I start thinking about it.

@ignacio: I'd be interested to see your changes to redis-lua for LuaNode even if it didn't fully work in the end (just out of curiosity), are they public?

[ignacio]

@nrk Sure, I just abused the 'scheme' field. I haven't submitted my changes anywhere since I couldn't get pub/sub to work, but the only change I needed to do to redis.lua was this:

local function create_connection(parameters)
    local perform_connection, socket

    if parameters.scheme == 'unix' then
        perform_connection, socket = connect_unix, require('socket.unix')
        assert(socket, 'your build of LuaSocket does not support UNIX domain sockets')
    elseif parameters.scheme == 'luanode' then
        perform_connection, socket = connect_tcp, require('cosocket').tcp
        assert(socket, 'your build of LuaNode does not support LuaSocket emulation')
    else
        if parameters.scheme then
            local scheme = parameters.scheme
            assert(scheme == 'redis' or scheme == 'tcp', 'invalid scheme: '..scheme)
        end
        perform_connection, socket = connect_tcp, require('socket').tcp
    end

    return perform_connection(socket(), parameters)
end

And a connection is created like this:

local params = {
    host = '127.0.0.1',
    port = 6379,
    scheme = "luanode"
}

local redis = Redis.connect(params)
redis:ping()

The changes I made to LuaNode are in the coroutines branch. Here is the relevant code and this is a test to see if a simple client/server app written with LuaSocket could be easily adapted.

[robert-chiniquy]

@nrk Hi! I would love to use redis-lua's async interface when 3.0 comes around, and would be willing to help with it. Is there a timeline or a list of issues or design direction for this? Seems like it will come with a number of other major changes.

[nrk]

Hi @robert-chiniquy, unfortunately the development of redis-lua has been lackluster in the past 6 months mostly because the library has proven to be quite stable at this point. This doesn't mean that I'm abandoning it's development but it's also true that I didn't put much thoughts on the evolution of the library for 3.0 (at least besides what's made it into the master branch) due to my lack of time, which means that I don't have a clear plan on how we could support an optional async interface yet. I still welcome ideas, experiments or proof of concepts based on the master branch if anyone is willing to help, I should be able to get back working on redis-lua in a few weeks.

[nrk]

I'm closing this issue, at least in its current form. Adding an asynchronous interface to redis-lua is still an interesting idea, but I'd prefer not to stick to a specific runtime such as Luvit.