Define a Product Vision for Luvit (and its ecosystem) to guide contributors and help them steer the project

[Duckwhale]

Source: #1135

It isn't currently very clear (to me) what exactly the goals for Luvit going forward should be. In order to properly guide development and documentation efforts in the future, it is crucial that the project's vision be written out. To that end, I've reviewed the 3.0 discussion and attempted to extract some key points that might be relevant.

Going by the comments that I've read, this is my current understanding:

• Luvit should not be reduced to a mere NodeJS clone
• Luvit should provide a "batteries included" runtime similar (in user experience and functionality) to NodeJS
• Luvit should be easily accessible, especially to new users (which it currently isn't)
• Luvit should provide a standardized package ecosystem (based on lit), similar to npm (and luarocks?)
• Luvit's development should be community-driven (see below)

Feel free to correct me if I'm wrong, or add to this if I missed something important. There's also things I'm unclear about; please clarify:

• Should Luvit support only LuaJIT, or both LuaJIT and PUC-Lua?
• If the latter: Should it support all versions of PUC-Lua or just the latest?
• Should (and could) it support both at the same time or would there have to be two entirely different "projects"/variants?
• What are the supported platforms that development should focus on?
• Are they different for the PUC and LuaJIT versions?
• What (kinds of) auxiliary libraries should be "core" to Luvit and which excluded?
• Which ones should be optionally provided via lit packages?
• Are there "sanctioned" lit packages that should be considered "officially maintained" or is it purely a "community effort"?
• Are the individual parts of the luvit ecosystem equally important or should they be prioritized differently? (i.e., luvit, lit, luvi, luv)
• To what extent should the community be involved in decision-making?
• Related: What would the project leadership look like ideally, and realistically?

If @creationix and other stakeholders could assist with defining the exact vision, I'll be happy to do the writeup. This could then be reviewed by you guys, and later published on the documentation website.

I'm aware that some of these questions may end up not being immediately relevant to define the project vision, but either way writing them down will still be beneficial and helps everyone interested in contributing to get a better picture.

It will also be helpful if you could define what is out of scope, and ideally add a "why" to every major decision that makes up the project vision.

[SinisterRectus]

After the 2.0 rewrite, I think that Luvit has largely been community-driven, with up to 5 people making most of the decisions and contributions. It's mostly been maintenance with occasional features added, though. No one is explicitly deciding what Luvit is or should be. We can try to define a product vision here, but frankly, I don't think anyone is up to the task right now.

When I opened #1135 in February, my intention was to see if there is a common vision for Luvit's future. In general, it looks like people agree that a refresh is a good idea, but we can't muster the manpower to put it in motion. I fully intended to write a prototype for 3.0 and hopefully answer some of the same questions that you have along the way. I think that if you have some tangible and working code to point to, it becomes easier to triage any ambitious goals for the project. Unfortunately, I have been dealing with other issues in my life that have prevented me from writing any Luvit code and have slowed down work on my own project Discordia. Understandably, other people in the community may have similar obligations that keep them from investing the massive amount of time required to rewrite Luvit.

I think your first set of bullet points are accurate and I think that you are unclear about the remaining bullet points for the same reason that you understand the previous ones: mostly everything about the future is in #1135. The decision to support certain versions of Lua or certain platforms or architectures is not something that we are discussing fervently in some private channel. Any design decisions are typically revealed ad hoc here on GitHub issues or pull requests or sometimes as a community in public Discord channels and previously IRC. I'd be happy to answer your questions as soon as we can, but I don't think any of us can right now since these discussions just haven't happened except in #1135.

[creationix]

Unfortunately, I have been dealing with other issues in my life that have prevented me from writing any Luvit code

I understand this completely. This is exactly the situation I've been in for the last long while.

That said, I don't think any of us are against documenting things and making it easier for the project to run more autonomously.

But I think the problem you'll run into is there still isn't a clear single use case for luvit. The reason we (attempt to) support so many different ways to use it (luajit vs the many versions of puc lua, lit vs luarocks, luv vs luvi, etc) is because different people have different needs and preferences.

At the same time, it would help a lot to narrow the scope and only support a single path assuming a single group is doing the maintenance.

Also I'm somewhat in conflict about the technical foundations of luvit. There has been a strong shift lately in web development (the world I live in) to strong type systems and dev-time compilers. Some of these I abhor, but some of these bring real improvements. Luvit, and especially luajit + libuv is an interesting technical foundation that makes for pretty good performance with pretty good ease of programming.

What this means in practice is much of our interest and ecosystem is people who use it for fun projects and less for professional projects. There is nothing wrong with this and I love and encourage people programming for fun, but it does pose a challenge finding experienced leadership. There isn't too much overlap between developers who would pick luvit for a project while also are comfortable digging into the native C code in luvi and fixing issues in there.

If @creationix and other stakeholders could assist with defining the exact vision...

I wish I did have a vision at the moment, but I don't. Initially I had the vision of making a better node.js by using luajit and coroutines. At the time JavaScript and node.js in particular was insanely popular, but people kept having serious issues with callbacks in JavaScript. The tooling around node and JS in general wasn't very good yet and luajit was a lot faster than V8.

But now the landscape has shifted a lot and luajit has stagnated. JavaScript has async/await, v8 based runtimes such as https://github.com/just-js can be extremely fast and make luvit look like a turtle. The tooling around JS in general is really robust (vscode, typescript, etc) and other languages like Go are also very robust for much of the same use cases. PUC Lua has diverged significantly from luajit and luajit has done almost nothing.

I honestly don't know what a good vision for luvit would be. I'm sure there is one and I suspect is has something to do with gaming, but I don't know what it is.

[Duckwhale]

We can try to define a product vision here, but frankly, I don't think anyone is up to the task right now

There is no try, only do or do not :P And I prefer doing over not doing, so here I am.

It's fine if some questions can't currently be answered, I'm mostly trying to get a better understanding of the ecosystem. It's difficult to decide how its documentation can best be improved if some of the fundamental questions are open, which is my main focus (for now).

but we can't muster the manpower to put it in motion

As I've mentioned before, it doesn't have to be sweeping changes or a full rewrite; I do think small steps lead to the same goal and they can be tackled one at a time. But before that can happen, some goal does need to be clearly defined.

This is just one step on the way and while I'm actively using Luvit, I would expect many more to follow (since I like improving the software I use). I hope you don't mind and I'm not getting on your nerves with all the questions, because asking lots of questions is the best way to learn.

I think your first set of bullet points are accurate and I think that you are unclear about the remaining bullet points for the same reason that you understand the previous ones: mostly everything about the future is in #1135.

I'm actively trying to separate the issues discussed there because it's too large a problem to think about otherwise, and no actionable steps can be taken. There will likely be issues related to each of the problems I've understood well enough to ask questions about.

Sure, some might be end up unresolved for the time being since the answer isn't clear, but I think quite a few could already be tackled with this approach :)

---

But I think the problem you'll run into is there still isn't a clear single use case for luvit.

I've noticed this, too, and maybe we can instead focus on some more general "goals" to prioritize officially. My suggestions would be:

• Providing a fast Lua runtime with a powerful standard library that is easy to use and well-documented (just like Node's)
• LuaJIT as the primary supported platform (unless PUC Lua is easily supported alongside it?)
• Building on the simplicity of basic Lua to make creating more complex applications equally accessible
• Creating extensive documentation and powerful tools to help people do this
• Everything else according to the community's needs (same as ever, really)

The point isn't to disavow all the other use cases, but rather to make sure that the ones that are "officially" intended to be used work as best as they can, and are well documented, with the rest prioritized according to its usefulness to the community. It's not exclusive, but not everything can be improved at once.

Disclaimer: This is approximately how I want to use Luvit, so it's obviously taking into account what I would want to spend my time working on first of all, but it's also what I think will be the most useful to people in general.

The reason we (attempt to) support so many different ways to use it (luajit vs the many versions of puc lua, lit vs luarocks, luv vs luvi, etc) is because different people have different needs and preferences.

I don't think it's feasible to support everything equally; but rather decide based on what's most useful or what people are willing to dedicate their time towards supporting. For me, this means LuaJIT/Lua 5.1 (for now), high-level APIs, lit over luarocks, documentation, tools, and things like serverside/web-related apps, so that's what I can justify spending time on since I've a stake in those domains.

I bet it's similar for other (potential) contributors. If you (for example) said "no, we only want to support PUC Lua and IoT use cases" the obviously there couldn't be any common ground, though it would still be a clear vision.

As I understand that isn't the case, so I will continue to work towards improving luvit as a general-purpose platform that covers the above use cases and we'll see what others think about the result. This isn't to the exclusion of other domains, but we could start with defining that as a key part of the project's vision since we seem to agree it's a worthy goal?

At the same time, it would help a lot to narrow the scope and only support a single path assuming a single group is doing the maintenance.

IMHO there can be multiple "groups" maintaing the features they find most useful. I'd guess that the ones I described above would be amongst the more popular ones, however. Officially aiming to make luvit better at them doesn't exclude also making it better at other things, but it would provide some much-needed direction and probably help new users get a feel of what luvit can do for them.

There isn't too much overlap between developers who would pick luvit for a project while also are comfortable digging into the native C code in luvi and fixing issues in there

While this is an area I'm not very proficient at (yet), I think the answer here is simple: Teach people how to do these things, at least in the context of luvit, and with time contributors will grow into "experienced leadership". This would be much easier if the Lua/LuaJIT documentation wasn't very minimalist, but I see no reason why luvit couldn't do better than them in this regard.

For me, one of the reasons for picking luvit over the alternatives is actively wanting to learn more about the "lower-level" underpinnings of both Lua and LuaJIT. It won't happen overnight, but there are bound to be others in a similar position.

In fact, I'd assume that people who use Lua (and luvit) extensively will soon become curious and want to learn more about how it works "under the hood" - and that's a great example where having good documentation will do most of the work for you. In other words, this is a short-term problem with sufficient documentation efforts directed at fixing it, and something I definitely plan on contributing to.

I honestly don't know what a good vision for luvit would be. I'm sure there is one and I suspect is has something to do with gaming, but I don't know what it is.

Lua's strength is accessibility and scripting, especially in games. Luvit can easily trump JS and Go simply by using a much better-designed language, and providing similar "batteries" and documentation for them - at least for the relevant (niche) audience. I wouldn't worry about speed too much unless there's a serious problem, because it's not the most important thing for many people in this context (and LuaJIT shouldn't be terribly slow either way).