Luvit 3.0
SinisterRectus opened this issue · 31 comments
The idea of releasing a new major version of luvit with breaking changes to fix outstanding issues or to address some of its shortcomings comes up occasionally. We recently chatted again about this in Discord. I think it would be good to have a place here for people to share any thoughts on the subject.
I will share my ideas in an independent post below this one. Please do not hesitate to share your own. Thanks.
Luvit was reborn as version 2.0 in March 2015. I joined the community a year later in early 2016 when I leveraged luvit as a niche solution to developing Discord bots in Lua, out of which I grew a small sub-community of Discord users. Since then, I've been an occasional contributor to the project and luvit has become my go-to framework for almost anything related to Lua. I consider myself an amateur programmer, so I do not claim to have any sort of well-informed professional opinion on what luvit is or should be. This is just my opinion based on my experience using luvit.
Luvit as a Node.js clone
When luvit 2.0 was released in March 2015, node was on version 0.12, io was on version 1.5, and JavaScript was ES5. By the end of the year, node and io had re-merged to become node 4.0 and ES6 was finalized. Since then, node has undergone changes sufficient to bring it to version 15.8, while callbacks have given way to promises and async/await-style coding with annual updates to JS/ES. In the same time, luvit advanced to version 2.17 with minimal API changes and today it is still based on the old callback-style of coding. I think that if luvit is intended to be a node clone, then some effort should be made to bring it up-to-date with respect to the current node APIs and modern JavaScript (or TypeScript).
Alternatively, the connection to node can be loosened and we can make luvit more of its own framework. Maybe there are some other frameworks or runtimes that we can look towards for inspiration? Deno is an obvious alternative, although I have not had the chance to look closely at it. The only other dedicated async I/O framework that I am remotely acquainted with is vibe.d. There is Python that has async I/O in its standard libraries, plus languages such as Julia and Janet that use libuv, but I'm not sure to what extent these may serve as inspiration for a Lua framework.
Regardless of whether we continue the node-clone approach or redesign luvit into its own framework, I think that coroutines should be emphasized over callbacks. As shown by Tim's "coro" libraries, coroutines can be used to make a robust async/await-style API.
Luvi and luv
I think that the modularization of luvit into luvi and luv was a fantastic idea and this should stay the way it is. I don't think that there are any critical issues that need to be addressed here, but there are some points worth mentioning.
Luvi loads deps/require and deps/pretty-print if they exist. I never understood why these modules are not just included in luvi. Can we move them?
The idea of switching to xmake or zig to build luvi or luv has been thrown around recently, which may involve making changes to the way we use certain libraries. I can't comment much on this, but I think @truemedian can. He's already made some progress using GitHub Actions and zig.
Additionally, @creationix has suggested moving luv from a callback API to a polling API. See polluv as an experiment. We can't easily break luv's current code because its versioning is bound to libuv's and because it is also a LuaRocks package, so any major changes to luv will have to be carefully considered.
Lua source files
We are maintaining Lua source files across three repositories. Some of them have multiple copies (for example, http-codec is in luvit and lit) and some have duplicated code (for example, luvipath in luvi appears to be a duplicate of pathjoin, which is in both luvit and lit). I think that we should make an effort to deduplify these source files and put them in one central source where possible. There is also the matter of moving luvit deps visited in #952.
Custom module system
One of the major defining features of luvit is its custom module system with relative requires, deps and libs resolution, lit packaging, and luvi bundling. It is a departure from standard Lua's module system and has several standing issues. I think that some of these issues are addressed by switching to luvit-loader, but that switch was put on hold due to the necessity for breaking changes (#932). Bumping luvit to 3.0 would give us an opportunity to revisit these issues.
Documentation
Not much to say about this one. Luvit's documentation is weak, but at this point, I think our time would be better spent updating luvit's 6 year old code instead of documenting it. We could then document the new code as its developed or immediately after.
LuaJIT vs PUC Lua
LuaJIT is great. It's fast, it has a nice FFI, it's popular, and it is based on Lua 5.1. Despite being 15 years old, Lua 5.1 is still a common and capable language today. At the same time, Lua 5.1 is now 3 major versions removed from the current Lua 5.4, and there are no plans to update LuaJIT's feature set to include anything beyond 5.2. I think that it would be worthwhile to offer a "PUC Lua" version of luvit and luvi.
This is a great write-up @SinisterRectus! I'll make my own post with some proposals later, but first some quick feedback:
connection to node can be loosened and we can make luvit more of its own framework
I agree. I instantly regretted making luvit a node clone as soon as the 2.0 rewrite was done. The virgo project that sponsored the luvit 2.0 rewrite used node style APIs extensively and it allowed us to be lazy about documentation by simply pointing people to node's website. But none of that is important or relevant anymore.
If someone really wanted to have an updated node clone, it could be done in userspace easy enough. But I feel it's not a good use of our limited resources for the core team to maintain such an API. We should instead focus on use cases where luvit is a good technology and make those better while embracing Lua's strengths. The coro-* and weblit libraries, for example were a great experiment and could be made into a very robust framework with some polish and good documentation.
I never understood why these modules are not just included in luvi
This was because they were non-trivial libraries and keeping them in userspace with magic hooks in the luvi binary made iterating on them a lot easier. Also it allows people to make really lean luvi based binaries that don't use them at all since they were optional. We can consider having a small set of libraries like this that are baked into luvi and official core APIs.
switching to xmake or zig to build luvi or luv
I also think this is a great idea, but haven't had the time to dig into the technical details about what is needed.
moving luv from a callback API to a polling API
If we did this, it wouldn't be a new version of luv, but an entirely new library. The entire purpose of this is to allow luajit's FFI to have better access to libuv. There would be virtually zero shared code between this approach and the luv bindings. For now this should probably stay a userland experiment. The binary extension mechanism allows this to be done completely externally if desired.
I think that we should make an effort to deduplify these source files and put them in one central source where possible
Agreed, but this is a tricky bootstrapping problem. For example, luvi needs the path joining algorithm for it's primitive vfs APIs. This would be another reason to consider simply moving these core libraries into luvi instead of making them optional with magic hooks. The luvit 2.0 rewrite focused on the most minimal and flexible core possible, but I think we've reached a point in the years since where we can pick a few libraries to include with core that make sense there.
some of these issues are addressed by switching to luvit-loader
+1000
I think our time would be better spent updating luvit's 6 year old code instead of documenting it
Sounds good to me. As long as we do write docs afterwards.
LuaJIT vs PUC Lua
These have diverged to the point that it's really hard to support both equally. And some things don't even make sense in both. For example the polluv experiment doesn't make any sense for PUC Lua since it doesn't have the fast JIT based FFI built in. I'm fine with spending some time and seeing exactly how much effort/compromise is required to support both. There is a similar dynamic with supporting luarocks and lit at the same time and stock lua/luajit vs luvi based builds. We kind of went crazy making luvit 2.0 be super flexible and can probably keep a lot of those qualities in the 3.0 version as long as we're aware of the tradeoffs and think they are worth it.
One thing I would like to explore in a breaking change is find out if it's feasible to remove the openssl dependency, but still have access to common hashing algorithms and TLS (1.3 at least). The hashing algorithms could probably be done in lua and be fast enough for most things with luajit. But TLS is a bit more work. This desire is mostly out of pain from building openssl and trying to get cross compiling working better.
The issue with dropping openssl for better cross compilation capabilities is that none of the TLS libraries I know of support cross compilation with the ease we're looking for.
If we were willing to drop server TLS capabilities, we could use a pure-zig approach: alexnask/iguanaTLS. However it is only TLS 1.2 and only client side. It is very easy to cross compile though.
Hmm, only client-side and 1.2 will work for a lot of use cases. Very few web service APIs are TLS 1.3 only and very few don't support 1.2. For servers, I tend to use nginx as my TLS terminator and proxy to the luvit server speaking plain HTTP.
I think it's safe to say that we can't drop openssl completely since I'm sure there are some people who need it. But maybe we don't provide prebuilt binaries with openssl on the platforms where it's difficult and then encourage most people to use the new and improved tiny version that has a working TLS client.
Having more bindings makes luvi (since in reality this is what has to be tackled first) 3.0 a bigger project, but I suppose that's unavoidable.
Here's my 2 cents for whatever it's worth. My inclination would be to lean much harder into the existing Lua ecosystem:
- Drop Luvit/Luvi-specific require behavior to as close to zero as possible (related: #1053, #932). Basically, the idea would be for Luvit modules to work totally fine when require'd from PUC Lua.
- This would entail removing support for relative requires, circular requires, etc. Support for modules in
depsandlibscould be kept but would be runtime-specific (i.e. support fordepsandlibscould be added by the runtime or not, but the modules shouldn't need to care about that). There's a bit of discussion about this sort of thing here: #932 (comment)
- This would entail removing support for relative requires, circular requires, etc. Support for modules in
- Modules/packages could be published to both Lit and Luarocks, where Lit would support the
libs/depsway of structuring your workspace and Luarocks would support its way (i.e. coro-http-luv and Lit'scoro-httpwouldn't need to be separate--identical versions could be published to Lit and Luarocks). - Luvi would be something like luastatic, but with more focus on cross-platform building and with built-in support for things like adding openssl, etc to the binary. This is not too far off of what it is now, but the idea would be to make it work with any Lua version and make everything optional (so you could just use it as a luastatic replacement if you want, i.e. if I wanted to move this thing I wrote from luastatic to luvi, it would support that use case; here's its luastatic build script for reference)
I have a feeling this might not work as well in practice as it does in my imagination, but I thought I'd at least throw it out there.
Our custom require seems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.
A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with the require issues. But if we can use Luarocks out of the box, would Lit be useful?
I agree with your vision of luvi. This reminds me of ninite where you check only what you want. It would be more work on our end to provide various iterations of luvi, but I think it would be nice if it works.
I think the largest issue with require right now is that it has some pretty weird edge cases that obviously weren't thought through. For example: file.lua fails over the luvit require and then ends up in lua's require, instead of requiring ./file.lua. Being able to do requires to relative paths is pretty helpful (along with being able to require deps/ and libs/.
Yeah I agree that relative-requires, plus libs/deps/bundle-requires, are super useful. One thing I thought of is having a totally new system that complements require rather than modify or replace it. For example, import("./path").
I like the idea of not reusing require for the new logic. I'm +1 on renaming the relative imports to import.
Our custom require seems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.
@SinisterRectus that aligns with my past experience in LuaRocks as well: in LuaRocks 1.x, I shipped a patched require that would do magic automatic versioning (if you did require("x") which depends on Y < 1.2, then the subsequent require("y") would pick up the right version of Y even if you had multiple versions installed). People were very reluctant on loading a module that patched a standard library function.
For LuaRocks 2.x, we switched to providing an optional package loader that you can load with require("luarocks.loader") which then enables the magic behavior. Some projects used it, mostly to auto-load the LuaRocks-configured paths at front, not to do version resultion, others were free to ignore it. Still, the magic behavior would occasionally trip people up in the event of conflicting dependencies.
Ultimately, very recently in 3.4, we introduced dependency version lockfiles (a la yarn and others), which allow users to extract their full dependency tree into a plain list with fixed versions and be able to see what exactly is going on and avoid surprises. In my experience, Lua people more often want to avoid having multiple versions of things installed rather than having something manage them. I'm not sure if that's the expectations are the same for people coming from the Node world (I'm assuming the expectation there is more like node_modules is a black box and I only care about the things I'm depending on directly).
So yeah, +1 to keeping more advanced behavior separate from require, and, if possible, allowing people to use individual modules in a simple way without requiring that advanced behavior.
A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with the require issues. But if we can use Luarocks out of the box, would Lit be useful?
The question would be how to keep the advanced behavior "hidden" from Lua end-users, so they could require the top-level modules as usual, and load them via LuaRocks.
I'm not very familiar with the libs/deps specifics in the Luvit world (I was invited into this thread by @creationix to share the LuaRocks experiences), but if there's some advanced logic that's not supported by LuaRocks, such as a node_modules-style nested tree of dependencies, one option would be to use Lit as the build backend of the LuaRocks packages, where it would produce the tree for a given top-level module and LuaRocks would package the whole thing as "vendored" modules (i.e. the rock for x storing x.lua and x/libs/y.lua or something like that), getting simplicity at the cost of replication. That approach would be essentially "wrapping Lit package as LuaRocks packages", if the goal is to make them more widely accessible.
Since LuaRocks 3.0 there's a luarocks init command which sets up a local lua_modules tree where people into which people can load their dependencies (we also added user/module notation (though for luarocks.org, not Github), etc.). From a quick look at https://github.com/luvit/lit/blob/master/package.lua and https://github.com/luvit/lit/tree/master/deps, I think the existing functionality in LuaRocks could provide a somewhat similar end result (though with a more verbose directory structure, at least in the current version), but usually I don't see people committing their lua_modules with resolved dependencies into the repo like that (what happens with C-based deps which need to be compiled?).
Still, I ended up not 100% happy with the ergonomics of using LuaRocks 3.x in this model, precisely because I didn't want to touch require, so users have to use a clunky wrapper script over the interpreter or configure their LUA_*PATH* environment variables correctly — this looks like the kind of things a smart import would solve.
In general, Lit seems to me like a much leaner and cleaner package manager. LuaRocks carries a lot of baggage from being very old, assuming almost nothing about its environment, and trying to be all things for all people. If you have a solution that works, I think it's unlikely that you'd get a nicer experience by switching over to LuaRocks today, honestly, apart from gaining access to the luarocks.org collection of packages.
It seems that the changes in LuaRocks 3.x have brought it a bit closer to Lit in functionality, though not quite there yet — still, it would be interesting to know what else is missing and if there are ways we could bring the projects closer together for mutual benefit. Perhaps by having Lit learn how to read dependencies from the luarocks.org repository? (that would be the opposite approach, "wrap LuaRocks packages as Lit packages", essentially.)
Perhaps by having Lit learn how to read dependencies from the luarocks.org repository
I think this could work well. The biggest difference between using luarocks and lit to install packages is where it puts them. If lit could pull from the luarocks repository and install them in deps folders (which is identical logic to node_modules btw), they should work without needing package authors or luarocks.org to change anything.
Also at the same time, I'm still feeling like separating the "smart" require from the built-in require and renaming luvit require logic to a new function like import will clear up a lot of misunderstandings.
Actually I misspoke. If we rename require to import when using the new search paths, then existing code that uses require will not be able to find modules installed to deps/*. So I guess the two approaches are mutually exclusive.
Disclaimer: I'm not an expert on the technology Luvit is based on, and I almost didn't want to participate in this dicussion as it feels like the "big boys table" and I'm certainly not qualified enough to give much input here. However, perhaps you will find it enlightening to hear a random user's perspective :)
Some more background: I've worked with Lua for the past five years or so, and for the past two I've been using both Luvit and NodeJS in parallel. As a result, I can describe my experience, especially that of someone trying to learn, and that might be of interest when you think about the future of this framework (as it's vital to get new people on board).
Let's talk about first impressions, because they matter, a lot. Whenever someone is looking for a Lua-based framework that might make their lives easier, they will probably pay attention to at least some of the following things:
- Which problems does this framework solve?
- How active is the GitHub repository? How are issues handled?
- What is the project's documentation like?
- What features and functionality will I get access to via existing APIs?
Ultimately, users want to know: Can the framework solve their problems, and Is it likely to be a liability or an asset?
Now, let's go back in time about two years, when I first heard about Luvit. What was my initial impression then, and has it changed now?
- The framework appeared to solve at least the "Lua has no batteries" problem I was looking to solve. So far, so good! Though it's hard to tell, since the website doesn't really say much in this regard.
- Well, there appear to be many different repositories and I don't even know what they're for. They all seem pretty dead, however. Not a good sign!
- There are issues from literally years ago still open, and quite a few never even received a reply? Uh-oh...(most people probably leave here)
- "Alright, but I really want to use Lua and these Node-like APIs sound like they will allow me to do what I can easily do in JavaScript (at the cost of having to use that abomination of a language). Let's take a look at the docs..."
- One of the very first pages I looked at was the https://Luvit.io/docs.html link. What does one find here? A link to NodeJS? (I didn't know what Node was back then, or how it relates to Luvit) That doesn't help at all.
- What is this project about? There wasn't even an introductory paragraph explaining to someone who is new what Luvit is and why they should use it, in terms that they can understand.
*Instead, I was bombarded with links to various things that meant absolutely nothing to me ("luv, luvi, luvit, lit; I don't know what to do with it" :D). - Almost mockingly, it's followed by 'Reading source code is always fun!". I mean, sure, but if I have to understand the code before using it, I might as well write it myself.
- At that point I thought, "Are they kidding me?" and clicked on the docs for the "http" API, only to find... nothing. It was essentially empty, and still is to this day.
- I don't think words can express how incredibly bad this first impression was, and it saddens me to see nothing has changed in that regard.
TL;DR: The only reason I'm following the project still is because I'm way too stubborn for my own good. Most people will not stick around after an experience like the one I just described! And could anyone blame them when there are so many other technologies and programming languages around that are easy to use?
The obvious conclusion from this is that how Luvit is currently represented simply doesn't inspire confidence. It looks like using it will be a liability, much like the rest of the Lua ecosystem, which frequently suffers from similar issues... and that's a shame because I think providing a Node-like experience in Lua (which doesn't have to mean a direct clone) is in actuality a really cool idea.
Speaking of Node, I later went and did what I sought to do with Luvit, using NodeJS. It took me mere minutes until I had a basic prototype of the app, a WebSockets-based server, running. When I first tried to do this in Luvit it took an entire day and a lot of help from someone in the Discord just to achieve the same basic thing, but there were still a lot of question marks / stuff that didn't really work the way I wanted it to.
I guess what I'm saying is, Luvit does a very poor job of explaining what it is, how to use it, and users are likely going to have a bad experience overall. All of these things are crucial to get right for any Open Source-project if it is to ever be successful (assuming success equals people using the software you spent so much time creating), and all the amazing technology behind it will go underappreciated if UX is completely ignored.
The good news is, there are people out there who like Lua enough to potentially use Luvit over Node, if the state of the ecosystem in general hasn't yet scared them off, and they might be willing to help. I know because I'm one of them. However, their life should be made easy and not needlessly difficult, as is currently the case. It takes a concerted effort but a little goes a long way already.
Finally, my thoughts about some of the more technical points you made:
- Luvit doesn't need to be a direct clone of Node in order to be useful. If I'm able to do anything Node can do with its APIs and things are made very simple, i.e. they "just work", then it really doesn't matter how exactly it functions. If it does offer very similar APIs that's certainly not bad, as it can help people understand more easily, but it's not strictly necessary given it has the same utility and ease of use
- LuaJIT being in a weird maintenance limbo implies, in my opinion, that purely relying on it moves the project closer to "liability" than "asset" in the grand scheme of things. Undeniably there's value in using it, but supporting the latest official Lua versions shouldn't be neglected either. For an ideal user experience, one should be able to run their native Lua apps using the latest PUC-Lua in Luvit without issues and similarly be able to use 5.1/LuaJIT-features as well as Luvit-APIs when needed. This may be harder than I imagine, but it'd be pretty damn useful if you could just use it for anything and it simply works out of the box
- For package management/lit, using the new LuaRocks approach and supporting lockfiles similar to NPM would likely be the most intuitive. If there was more interoperability and cooperation between the projects, that could help in creating something many people can get behind. IME Lit works much better than LuaRocks (on Windows), but it has an even smaller ecosystem and doesn't always do things in the way that one would expect... so perhaps we could get the best of both worlds by making them interoperable and agreeing on a set standard?
I hope no one takes offence at what I wrote. I'm ardently campaigning for a better user experience because it plays directly into one of the main strengths that Lua as a programming language has, and also because most developers (in any language) don't seem to value it nearly highly enough. I understand why and I don't blame them, but I think it's unfortunate because the purpose of any software is to be used, by users who often don't want to know all the details. It follows that using it should be as simple as possible, to the point of being foolproof even.
If you would like to help Luvit become a true equivalent to NodeJS in spirit, then you should IMHO focus much more on making things "just work", making it work intuitively, and making sure that the technology can be used by almost anyone on any platform with as few stumbling blocks as possible. Software that people don't understand they won't use, and software that isn't used is literally a waste of time to create and maintain. Node is extremely easy to use; that's one big reason why people use it in the first place. The large amount of libraries are just a side effect of it filling a real need in my eyes and if Luvit was more user-friendly it might have more actively-maintained packages, too. The same could be said about LuaRocks, too, but that's besides the point.
Given the power of LuaJIT and the accessibility of Lua, I think with Lua one could achieve what Node/JS managed to do, but better, if there was any project that prioritized accordingly. Luvit seems to be the one best suited to it from what I can see, so the question is... is this something you actually want? If so, I'm happy to help.
Of course I'm not really sure what the project's vision and goals are (or were), so perhaps I'm envisioning something very different from what you set out to do. Defining that and then featuring it prominently somewhere might be a good first step.
The biggest problem for newcomers to Luvit right now is the significant lack of documentation that has multiple root causes, but all of them are not equal. Namely:
- Tim doesn't have very much time to put into Luvit anymore, and since he was the one with the most effort behind the project, it kind of stagnated it.
- Luvit's builtin libraries mimic NodeJS's, this introduces a few problems:
- implementing JS libraries in Lua verbatim does not create idiomatic Lua code.
- Being non-idiomatic Lua code (callbacks, etc) nobody wants to touch them, much less document them
- There hasn't been anyone step up with the time or knowledge to write helpful, complete documentation.
Luvit doesn't need to be a direct clone of Node in order to be useful.
This hits the issue perfectly. Luvit's builtin libraries should be easy and simple to use in Lua
LuaJIT being in a weird maintenance limbo
The LuaJIT 2.1.0 beta isn't in a maintenance limbo, Mike commits consistently see here.
For package management/lit, using the new LuaRocks approach and supporting lockfiles similar to NPM would likely be the most intuitive.
Lockfiles for lit would definitely be useful, even if just as an excuse to rewrite lit (which has a lot of quirks).
If there was more interoperability and cooperation between the projects [lit and luarocks]
This would be nice, but as hisham stated above, theres still a gap between the two.
I almost didn't want to participate in this dicussion as it feels like the "big boys table" and I'm certainly not qualified enough to give much input here.
Every experience counts here! And you make some very valid points about the approachability of Luvit. I agree that this can and should be be improved to be more on-par with Node. I think one of the the biggest hurdles here is just a man-power issue. The community of Luvit and Lua contributors and users is easily dwarfed by the size of Node's and JavaScript's. We would have to set realistic expectations about how high we can climb the hill on which JS sits, but that should be no excuse to exclude important features like documentation.
There have been a handful of attempts to make batteries-included versions of Lua. I defintely see Luvit as one of those, even if it is not marketed as one. For all of its shortcomings, I haven't found a well-equipped Lua runtime that is easier to get running cross-platform than Luvit is, although that may be for lack of trying.
Thank you for your input, @Duckwhale.
I can try to open a bunch of issues and even PRs for some of the problems I see, but to be honest it didn't seem like anyone is actively managing things here on GitHub, so I haven't bothered yet.
There are stale issues and PRs in every single of the organization's repositories and clearly no triaging takes place, which I'm sure will send all the wrong signals to potential contributors or even normal users (if the lack of documentation didn't scare them off).
I'm not well-versed in the technical aspects, but if you can assist with that I could give it a try and see if the docs can't be salvaged. I also suggest that issues be handled more proactively; if you agree that's important I could make some suggestions as to how that could look (in a separate issue).
My main worry here is that any effort will be sucked into the void and nothing will change. I fully understand the reasons you listed for the general inertia of the project, and naturally I don't expect anyone to magically conjure up time they don't have :)
I do, however, know from experience that even sacrificing a few minutes every day to answer issues/PRs and perform simple janitorial tasks goes a long way in keeping the worst at bay. It's of utmost importance to capitalize on community goodwill, and to that end things like the GitHub repository and documentation must give the right impression or there won't be anything to capitalize on.
If nothing else, they should make people feel like the software is worth using, and that the project is managed in such a way that contributing seems like a worthwhile investment. This, unfortunately, isn't currently the case. Luckily, there's no reason this couldn't change with some minimal effort in the right places.
Perhaps one should start there and see what the result will be before thinking about larger rewrites of the codebase (as implied by "Luvit 3.0")? That is to say, it might be more beneficial to think of it in terms of re-focusing instead of re-writing.
Very good points @Duckwhale.
You're right that luvit is kinda scary in it's current form to anyone who knows what red flags to look for. This wasn't always the case. There was once a time many years ago where working on luvit was my full time job. You'll quickly notice when this time was by looking at the dates on the blog posts and overall commit velocity to the project. But even then despite me working full time to promote it (including me speaking at international node.js and lua conferences about luvit), despite me staying on top of the issues and doing all the things you mentioned, it never took off. Eventually my employer got impatient and pulled the plug on funding the open source stuff.
The project sat nearly completely dead for years until a new community emerged independent of me. This is the new community with an active discord channel and largely centered around the discord bot framework. I eventually noticed the new community and have been slowly off and on trying to give it time. I'm still the sole owner / sysadmin of the website (including the production package repository) which explains why it is occasionally down for hours or days at a time.
I used to be super active in open source authorship and now have somewhere around 500 personal github repositories. What I didn't learn about open source till it was too late is the cost of a project becoming popular. My most popular project was a tiny bash function known as nvm. The support workload became so great that I gave it away completely and it now sits in it's own organization that I don't follow or watch with a new person in charge https://github.com/nvm-sh/nvm. My second highest support load was either JS-Git or Luvit. Both had died (near zero interest) after just a year or two and I was busy living life and working at various companies that didn't allow me much open source time.
Once at Luaconf in Brazil during a Q&A session, Roberto Ierusalimschy was asked what could be done to help Lua get more popular. Was it better docs or certain key libraries, or more conferences or what? His answer surprised me. He said he was not interested in growing the popularity of the project. I now kinda understand. If luvit gets popular, it will mean more pressure and stress for me. It will mean I'll need to pay more hosting fees, I'll need to be more diligent to keep the server running all the time. I'll need to field angry messages from people upset about the lack of good documentation (not everyone will be as polite as you have been). I (or someone else) will need to do all this extra work with nothing to show for it. Maintainer burnout is a real problem today and I don't know the solution.
That said, I think your suggestions are great and support them. I would love newcomers to this project to have a good experience despite the risk that it might cause the project to get popular. Documentation in particular of the code and the project itself (how to cut releases, how to triage github issues, etc) will do wonders to crowdsource the maintenance this project desperately needs and currently lacks.
I've been thinking over a few technical challenges associated with a Luvit 3.0 and the opinions that have been stated above.
Current Modularization
Obviously, the modularization of luvit, lit and luvi has caused a few separate problems (luv is not part of this consideration, and should remain separate). A small portion of luvit libraries has been duplicated in multiple different places, and while not a huge problem, it makes maintenance a nightmare. Additionally, most issues with luvit tend to span more than one repository in my experience.
Ideally, things wouldn't be this way. My proposal to fix this specific problem is merge luvit, luvi and lit into one single monorepo where all luvit development is centralized, and libraries can be shared throughout to prevent duplication.
This would solve the below issue.
I think that we should make an effort to deduplify these source files and put them in one central source where possible
Agreed, but this is a tricky bootstrapping problem. For example, luvi needs the path joining algorithm for it's primitive vfs APIs.
Cross Compilation
Luvi is currently build hell (partially thanks to CMake, partially because of all the libraries it includes, and all of the options it supports) and can only output executables for it's native platform.
First, replacing the build system without also remaking Luvi from scratch in order to deal with all of the external libraries is a non-trivial job. Zig has been thrown around as an option, but I will not overtly recommend it because of my involvement. Zig would support cross compiling luvi to essentially every platform and it's build system is more comprehensive than CMake.
Second, allowing Luvi to cross compile should be rather easy if luajit is the only Lua backend that we support. Luajit supports outputting bytecode for all of it's architechtures from any native platform, the only issue would be with including every version of Luvi in a single binary so that it could be actually cross compiled. However, as long as luvi could be cross compiled, we would be able to use it to bootstrap the lit and luvit builds (as it does currently).
However, rewriting Luvi is a much larger endeavour, and would require a lot of effort (probably from multiple community members).
The Luvit "Official" Libraries
Luvit's libraries are birthed from the idea behind a Node.js code, but this lead to all of Luvit's built-in libraries becoming object and callback oriented. Object orientation is a special case, because a few of Luvit's structures did benefit from it. However, callbacks are not idiomatic Lua, and I believe that they should be removed (unlikely) or at least hidden (the creationix/coro-* libraries are a good example of how this looks).
As far as where I stand, a rewite of all of the libraries is not a horrible option, even if it is a lot of work.
The Require Situation
Here's my 2 cents for whatever it's worth. My inclination would be to lean much harder into the existing Lua ecosystem:
Drop Luvit/Luvi-specific require behavior to as close to zero as possible (related: #1053, #932). Basically, the idea would be for Luvit modules to work totally fine when require'd from PUC Lua.
This would entail removing support for relative requires, circular requires, etc. Support for modules in
depsandlibscould be kept but would be runtime-specific (i.e. support fordepsandlibscould be added by the runtime or not, but the modules shouldn't need to care about that). There's a bit of discussion about this sort of thing here: #932 (comment)Modules/packages could be published to both Lit and Luarocks, where Lit would support the
libs/depsway of structuring your workspace and Luarocks would support its way (i.e. coro-http-luv and Lit'scoro-httpwouldn't need to be separate--identical versions could be published to Lit and Luarocks).Luvi would be something like luastatic, but with more focus on cross-platform building and with built-in support for things like adding openssl, etc to the binary. This is not too far off of what it is now, but the idea would be to make it work with any Lua version and make everything optional (so you could just use it as a luastatic replacement if you want, i.e. if I wanted to move this thing I wrote from luastatic to luvi, it would support that use case; here's its luastatic build script for reference)
I have a feeling this might not work as well in practice as it does in my imagination, but I thought I'd at least throw it out there.
Our custom
requireseems to be that which trips people up the most when they move into luvit-land, so I agree; that should be a high-priority fix.
A more streamlined solution for using Luarocks modules would be cool. I think this kind of goes hand-in-hand with therequireissues. But if we can use Luarocks out of the box, would Lit be useful?
Yeah I agree that relative-requires, plus libs/deps/bundle-requires, are super useful. One thing I thought of is having a totally new system that complements
requirerather than modify or replace it. For example,import("./path").
Documentation
Luvit's documentation sucks. And on top of the lack of documenation, a large portion of Luvit's library source code is uncommented. My proposal to fix this is either to either retrofit, or simply include in a library rewrite, documentation inline with the code as comments, this would use a tool (like ldoc or similar) to simply run over all of the code and generate the respective documentation for every library.
Keeping documentation right alongside the code would allow anyone modifying the code, and the reviewers signing off on pull requests to verify that whatever changed in the code is reflected in the documentation accordingly.
When you say mono-repo, luvit, luvi, and lit would still be separate applications, yes?
Correct, it would just put all of the code in single place, so that things that need to be shared between all of them, can be.
Edit: for some clarification that I didn't provide above, moving luvit, luvi and lit into one place would also serve to link their versions together, so luvit 3.1 uses luvi 3.1 and lit 3.1.
I really would love to drop openssl in luvi. Also several of the libraries that were added over the years aren't really used much.
And I think we can for http servers since those are often sitting behind some kind of reverse proxy that handles TLS termination. But the thorn in my side is always https client support. I don't think we can drop that without loosing a large part of the community. For example, I'm pretty sure discordia speaks to discord's API servers over https.
If there was a way to handle TLS clients without including all of openssl, I'm all for it, especially if it's portable. Part of me wonders how hard TLS 1.3 only would be in pure lua or if there is a self-contained library for only TLS 1.3. Most API servers support this by now I think so lots of use cases would stay viable.
I know TLS 1.3 is easier to implement, but that also means either finding another library or also implementing all of the cryptographic functions that it uses behind the scenes. Alas, any crypto library in C land is usually a pain to implement, harder to cross compile, and might not support all of the targets we want to.
I'm definitely shilling Zig here, but it both has the cryptographic primitives in it's standard library (which means support for a lot more targets than we can hope to [because of libuv]) and already has a working TLS 1.2 client library which is relatively easy to deal with. If TLS 1.3 is a target, then effort would have to be made, but at least TLS 1.3 is easier to deal with.
Part of me wonders how hard TLS 1.3 only would be in pure lua or if there is a self-contained library for only TLS 1.3.
I've thought about this too. It's been done in JS. For example, https://github.com/digitalbazaar/forge is a whole crypto suite. I don't know of any in Lua. I think no one is that ambitious.
I'm definitely shilling Zig here...
No complaints here, especially if we end up using it as our cmake replacement.
I wonder if supporting only TLS 1.2 would be good enough for now. That's more widely deployed than 1.3 anyway I think.
We might want to consider rustls and see if we can bind it to luv
That looks like a good option too, I especially like that option if it also gives us modern crypto primitives similar to what libsodium exposes.
I worry about creating our own homegrown libraries and the maintenance that would bring.
Agreed. It's one thing to implement sha256 in pure lua because there is almost zero maintenance for that, but all the algos needed for TLS brings the surface area big enough that there are bound to be problems that need addressing.
Here's my opinion, although I'm also just a random user, and I'm not sure how much this helps (most of this comment is just reiterating other people's posts). My background: I originally started using luvit because I wanted to write a discord bot in Lua, but after a while I started using it as "Lua with many nice batteries included".
Following sections are roughly in order of most important/relevant to me first.
Documentation
My proposal to fix this is either to either retrofit, or simply include in a library rewrite, documentation inline with the code as comments, this would use a tool (like ldoc or similar) to simply run over all of the code and generate the respective documentation for every library.
I very strongly agree with this. Additionally, I think it would be beneficial for this documentation to have some kind of machine-readable type information. This would allow users to automatically generate definitions for IDE completion (e.g. EmmyLua annotations) and/or for typed languages that transpile to Lua (e.g. TypeScriptToLua).
When writing or debugging code, I find myself quite often searching for the code for libraries I'm using, but I rarely know whether they might be in lit or luvit, or even one of creationix's GitHub repositories. A lit/luvi/luvit mono-repo would definitely help here.
The luvit libraries
Right now, the first step to any project for me, as a luvit user, is installing the coro-* libraries, and I do not use the base libraries unless I'm writing a standalone script. I believe making the coroutine wrappers the default would be good, although I don't know if that's possible.
luv/dependencies
I sometimes use luvi or lit make to package code I write, even when they have no dependency on any of the luvit ecosystem or luv, simply because of the convenience. Although I acknowledge this is most likely an unsupported use case, it would be nice to be able to build luvi with smaller dependencies or only a subset of its dependencies. Cross-compilation would also make this easier.
Honestly, I never use any Lua runtime other than LuaJIT, even when not using luvi(t), so keeping it as the only runtime seems very reasonable to me.
I don't have much experience with the rest of the subject, although I can say that HTTPS client capability was a requirement for a good portion of my projects using luvit, but any HTTP server I wrote (mostly using weblit) was either only for local use or behind nginx.
LuaJIT is great. It's fast, it has a nice FFI, it's popular, and it is based on Lua 5.1. Despite being 15 years old, Lua 5.1 is still a common and capable language today. At the same time, Lua 5.1 is now 3 major versions removed from the current Lua 5.4, and there are no plans to update LuaJIT's feature set to include anything beyond 5.2. I think that it would be worthwhile to offer a "PUC Lua" version of luvit and luvi.
I agree that a version of Luvi and Luvit that supports PUC Lua would be nice for anyone who might happen to need it, but it means that we would have to get rid of the assumption that code is running on LuaJIT in luvit's libraries. From a quick search: luvit/buffer, luvit/dns and luvit/require, and luvit's main.lua entrypoint all use ffi; while luvit/los, luvit/pathjoin and luvit's init.lua all use jit.
However, I do also think that luvit's libraries should undergo a refactor or rewrite to cleanse the code of the callback-oriented mess that they currently are. Doing so would also allow us to get rid of any luajit assumptions that are currently in place and either remove them entirely, or use them as optional optimizations if available.
change list
- loadstring. replace with load
- setfenv, replace with load
- table.getn, replace with #
- table.foreach, replace with pairs
- unpack, replace with table.unpack
- jit, remove it, replace jit.os with ffi.os
- ffi, a ffi module facebook or jmckaskill
this change is not right, remove it.
