Consolidation of multiple similar libraries

Question

Consolidation of multiple similar libraries

keithamus opened this issue 9 years ago · 43 comments

Hi @mysticatea and @k88hudson! Thanks for your hard work with npm-run-all!

I maintain a similar library - the functionality is slightly different but it shares a lot of common functionality with npm-run-all. My library is called parallelshell - which I maintain with @paulpflug. I am also aware of a few other libraries that attempt to solve the problem of properly parallelising npm (or shell) tasks:

concurrently which is written by @kimmobrunfeldt and has had contributions from @luv2code.
shell-executor which is written by @royriojas.
npm-parallel which is written by @spion
pkgrun which is written by @reggi

We're all trying to reach the same goal - run npm tasks quickly and easily in parallel (and sometimes serially), and have them log & exit in ways you'd expect. For the most part, each of us (I presume) has convergently developed these solutions. While it's great that we're all solving this problem, its bad that we're all solving it...

The point

I think we should combine forces, to make one canonical parallelisation tool. Personally, I see no point in continuing development of both parallelshell and npm-run-all - we may as well just have one. Having said that, I'm still using parallelshell because it solves particular bugs that I (and @paulpflug) have solved (with great pain) which npm-run-all is yet to solve.

The way I see it - I'm happy (as long as @paulpflug is) to port the fixes we have developed in parallelshell, over to npm-run-all rather than expend effort developing the features I want from npm-run-all inside of parallelshell. I dont want to speak for all other developers (@paulpflug, @kimmobrunfeldt, @luv2code, @royriojas, @spion, and @reggi) - but I for one would certainly like for all of these projects to consolidate under one label. If and when these projects converge, I'd like everyone of the original authors/contributors to be a contributor of the new project - to ensure that it remains well maintained. How does this sound to you?

FYI, I chose to file this issue here - rather than any of the other projects - because I think this project has the best name. By and large they are pretty much all interchangeable (or certainly could be with a bit of development).

Summary

npm-run-all and parallelshell are very similar (as are other projects)
We should develop one
(IMO) All of the maintainers of the other projects should all be core contributors of this project, and deprecate their respective projects.
Everyone is happy and users have one well developed tool to chose from.

paulpflug commented 9 years ago

👍

Answer 1 · 2015-10-16T14:57:04.000Z

I'm okay with unifying into this project.

Answer 2 · 2015-10-16T16:41:53.000Z

Yeah, I just figured out #6 :(

I was thinking of writing an alternate implementation that uses the data from package.json but doesn't rely on npm run at all - especially given that npm run seems to be so slow (or is it just slow for me?)

In any case, I agree it would be best if the effort was concentrated, but personally I'm not sure what the direction should be...

Answer 3 · 2015-10-16T16:53:08.000Z

Wow, really interesting.
I am grateful for your proposal.
I'd like to cooperate to make helpful tool for developers!

@spion I feel npm run is slow too. So I have been thinking that directly-spawn content of npm tasks. But it's very hard....

Answer 4 · 2015-10-16T17:57:25.000Z

👍 happy to merge, help or document :), I guess #6 is an issue that most of us have faced at some point, probably the most important thing a library that runs several things in parallel should have.

Answer 5 · 2015-10-16T18:26:24.000Z

I would definitely say spawning from the package.json, rather than npm run is a good idea. parallelshell has been working on solving the problem of properly killing subshells and the processes running in them, and we have darkguy2008/parallelshell#30 which is going pretty well thanks to @paulpflug. I think we could take this and turn it into a PR for npm-run-all.

Not depending on npm run as an executable also opens us up to the ability to have npm-run-all run arbitrary scripts, not just npm ones - perhaps with a -e flag (e.g. npm-run-all build:* --sequential -e 'echo "done!"').

Also I think it'd be a good idea to see how npm run itself works. Here's the run-script command which mostly builds up which scripts to run (e.g. pre and post prefixes) - L159-167 build up the commands to run, and L166 is what each command runs (i.e. it calls the lifecycle command with the package json, command, working directory and true). lifecycle.js is basically all of the execution magic, including making all of the env vars, building the shell string and actually running the command. When it all comes down to it, all npm run is essentially doing export ${lots_of_env_vars}; sh -c ${npm_script_string}.

I would be interested in @othiym23 and @isaacs opinions about possibly modularising some of this - so we can all re-use the same piece of code for execution logic. They may well also have some insight for us as they wrote most of npm's lifecycle.js 😄 (although most of that code is 5 years old and so I'd understand if they've forgotten).

Answer 6 · 2015-10-17T08:13:46.000Z

I too think that merging these projects makes sense. One comment about the naming though. I chose a general name because concurrently is not tied to npm run scripts, you can also use the tool for any shell command.

Answer 7 · 2015-10-17T08:28:43.000Z

How about we make 2 packages. the first focusing on sequential and parallel shell execution (name maybe 'parallelshell') and the second focusing on launching npm tasks (grep from project.json or similar)) while depending on the first for execution (name maybe 'npm-run')

I think that would be super transparent for the users and limits the necessary flag count.

Answer 8 · 2015-10-17T19:56:05.000Z

I agree, that sounds like a good idea.

Answer 9 · 2015-10-18T02:10:02.000Z

Thanks @keithamus for putting this together. I love this idea. I too think that this should have a more general name and not be npm specific. Just to play devils advicate, I'd love this all to be within one project with special flags for npm specific functionality (e.x. --npm-run or --pkg-glob) , just a thought. I embrace a clean / clear UNIX based design, but keeping in mind that we're all node.js developers and that whatever this thing ends up being is going to be very tied to that ecosystem is important. However two commands as @paulpflug describes also makes sense. 👍. With all that said I'm looking forward to seeing this through and would love to help in any capacity.

Answer 10 · 2015-10-20T04:43:57.000Z

Just found these projects that touch on this stuff.

Answer 11 · 2015-10-21T15:29:11.000Z

Hey everyone, awesome to see this coming together 👍 npm scripts for the win!

Personally I see two separate but related goals here;

To run tasks in parallel in a way that works on all platforms
To provide a better syntax for organizing and running npm scripts or groups of npm scripts

Obviously Windows compatibility is a goal for both 1 and 2, so some code/ideas could be shared, but I would prefer to see them remain separately focused on their respective goals.

If anyone is interested, I know some people on npm/who work on the cli and I would be happy to reach out if they have suggestions, or if any of you want to provide some feedback.

Thanks so much @keithamus for opening this issue and for linking to everyone's projects, you are all awesome 💃 I'm giving another talk on npm scripts at View Source conf in November, and I can tell you more and more people are getting excited about doing front end automation this way thanks to your work and blog posts 😄

Answer 12 · 2015-10-21T15:30:05.000Z

In fact, I will definitely bring this up with the folks at npm. Thanks also @mysticatea for writing this :)

Answer 13 · 2015-10-21T15:59:49.000Z

Hey @k88hudson! Thanks for the kind words. I definitely think feedback from the peeps at npm would help here.

Aside; @k88hudson I watched your (absolutely excellent) npm talk at Nordic.js which led me to npm-run-all and prompted this issue. Small world 🌏!

Answer 14 · 2015-10-21T16:36:15.000Z

I thought about it a while, here is what I want:

one project - three cli's
the commands should be short and make no flags necessary in default mode
run-seq / run-para for running whatever in parallel or sequential
(mainly for use in package.json, e.g.: {"scripts":"{"first":"run-seq 'echo 1' 'echo 2'"})
run-npm as a drop-in replacement for npm run with wildcards as a main feature. On console: run-npm watch:* would call all scripts starting with watch: in parallel. Calling run-npm watch should act as a shorthand for npm-run watch:* (yes I really want to save that line "watch":"Call all other watches" in package.json)
Sadly npm-run is already taken.
advanced scripts featuring placeholders (example for package.json: {"adv-scripts":"first":"ENV=%1 echo %2"}, would be called this way npm-run first VAR1 'hello world')
This will make it possible to create meta package.json, which holds scripts for several other projects: {"adv-scripts":"git:push":"cd %1 && git push"}
coffee-script - or anything else without curly brackets .. I hate them, mainly because of the german keyboard layout ;)

In my eyes a unsolved problem of all project-processing methods is to handle several projects. The worst case: Say I researched recently and found a new postprocessing tool far better than the other tool I used in 5 of my projects and now I want to migrate...
But even simple tasks like "bump the version of 4 projects and push them to github" is difficult.

Maybe we could collect what all others want and start putting a feature list together - so the development can start 😄

Answer 15 · 2015-10-21T19:50:42.000Z

one project - three cli's

the commands should be short and make no flags necessary in default mode

run-seq / run-para for running whatever in parallel or sequential (mainly for use in package.json, e.g.: {"scripts":"{"first":"run-seq 'echo 1' 'echo 2'"})

On the surface this idea seems good; because having a mix of flags is a bit of a pain, and a bit hard to disambiguate. However when you consider the power of npm-run-all chaining multiple commands, you can see that npm-run-all currently does a better job:

$ npm-run-all --sequential a b c --parallel d e f --sequential g h i
# To do this with three commands:
$ run-series a b c && run-parallel d e f && run-series g h i
# Or
$ run-series 'run-sequential a b c' 'run-parallel d e f' 'run-sequential g h i'

I have some tasks in a project right now which are mix of sequential and parallel (build all the things, then watch the output with several tools and serve html all at once). npm-run-all facilitates this in - what I think - the easiest way possible.

Having said that I think it could end up being pretty hairy having command line, npm, sequential and parallel all wrapped in one lib. How about the following instead:

One project, two clis
The two commands are run-all and npm-run-all

run-all takes a set of strings, which are shell commands, and --series and --parallel commands which behave much like npm-run-all does now.

$ run-all 'echo hello' 'echo world'
hello world
$ run-all 'sleep 1 && echo hello' 'echo world'
hello world
$ run-all --parallel 'sleep 1; echo hello' 'echo world'
world hello
$ run-all 'echo 1' --parallel 'sleep 2; echo 2' 'sleep 1; echo 3' 'echo 4'
1 4 3 2

npm-run-all works pretty much as it always did; it matches the behaviour of run-all except it calls aliases for npm-scripts:
```
$ npm-run-all test:* build:* --parallel watch:*
```

run-npm as a drop-in replacement for npm run with wildcards as a main feature. On console: run-npm watch:* would call all scripts starting with watch: in parallel. Calling run-npm watch should act as a shorthand for npm-run watch:* (yes I really want to save that line "watch":"Call all other watches" in package.json) Sadly npm-run is already taken.

I like the idea of npm-run watch defaulting to an alias for npm-run watch:* but we should make sure it is overridable. I imagine the logic to be a bit like this:

if packageJson.watch
  run packageJson.watch
elseif !packageJson.watch && packageJson.watch:*
  run --series packageJson.watch:*

advanced scripts featuring placeholders (example for package.json: {"adv-scripts":"first":"ENV=%1 echo %2"}, would be called this way npm-run first VAR1 'hello world') This will make it possible to create meta package.json, which holds scripts for several other projects: {"adv-scripts":"git:push":"cd %1 && git push"}

I'm going to disagree with this on a few points:

I think package.json#config is sufficient for metadata (although the environment variable is a little on the verbose side for my liking: $npm_package_config_<thing>).
If a package is so complex that it requires many config vars, it should probably be a script.
I think we need to be careful about exactly what we're implementing here. To me - the core feature set is "run commands in parallel in a cross compatible way" and "use that as a basis to quickly orchestrate a set of npm scripts". IMO anything else is scope-creep.

coffee-script - or anything else without curly brackets .. I hate them, mainly because of the german keyboard layout ;)

Ew no! Babel or bust!

In my eyes a unsolved problem of all project-processing methods is to handle several projects. The worst case: Say I researched recently and found a new postprocessing tool far better than the other tool I used in 5 of my projects and now I want to migrate...
But even simple tasks like "bump the version of 4 projects and push them to github" is difficult.

I agree, this is a problem (well, for my work its managing hundreds of components and keeping everything up to date and standardised across the board). I think this is an orthogonal problem though - mostly one of provisioning and less about orchestrating tasks.

Answer 16 · 2015-10-22T11:15:48.000Z

I like the idea of run-all and npm-run-all.
The collection of small programs is definitely good, but package.json#scripts field has small area, so I'd like to keep compact notation of this tool.

coffee-script - or anything else without curly brackets .. I hate them, mainly because of the german keyboard layout

I was surprised by the location of brace's keys!
But this tool has many asynchronous loops, so I'd like to use async/await (an ES stage 3 proposal).

BTY, I think we should be looking at npm/npm#9970 .
I'm thinking maybe we should change the separator of Glob-like patterns of npm-run-all.

Though I will add collaborators of this repository, may I add all people who is joining this conversation?

And there are requests about making changes.

Doesn't commit changes to the master branch directly.
Creates a PR then other members review it.

Answer 17 · 2015-10-22T12:11:30.000Z

I like @paulpflug's idea of having separate CLI commands for parallel and sequential run.

In @keithamus' example:

$ npm-run-all --sequential a b c --parallel d e f --sequential g h i
# To do this with three commands:
$ run-series a b c && run-parallel d e f && run-series g h i
# Or
$ run-series 'run-sequential a b c' 'run-parallel d e f' 'run-sequential g h i'

I like 2nd(this is not an option because lack of windows support) and 3rd more because they explicitly state which steps are run sequentially and which are run parallel. The 1st command has implicit hidden information: all those substeps are run sequentially.

Answer 18 · 2015-11-27T10:19:27.000Z

I think we have enough to get a prototype going. I'm going to make some time this weekend to get a first draft done and we can all see where to go from there.

Answer 19 · 2016-01-27T19:53:45.000Z

This sounds really good. Has there been any development on this? I'm currently using npm-run-all because it seems like it is going to be the destination project for all these similar efforts. The "one cross-platform, parallel script runner to rule them all". Any way I can help?

Answer 20 · 2016-01-28T04:42:30.000Z

@keithamus Good on you for opening this Issue and burying your lib in exchange for working together for an even better one. GitHub needs more developers like you. 👍

Also, thank you for your article on switching to npm. I just made the switch because of it and it's beautiful over here.

Now if you'll excuse me, I have to go build yet another grid system to pollute the sea of grids with... :(

@mysticatea Oh and good job on this. Works like a charm!

Answer 21 · 2016-01-28T10:56:42.000Z

Thanks for the kind words @corysimmons.

I know I said I'd get a draft done a couple of months ago - things got a bit in the way, but rest assured it is on my todo list (item number 3 😉) so I'll be pushing some code soon.

Answer 22 · 2016-02-12T11:14:41.000Z

hey,

Because I needed more functionality, I created a tool yesterday. Now it is finished with unit tests but without much documentation yet.
Haven't run the tests on windows, though.

So here is the spawn wrapper better-spawn with all the things I learned from parallelshell.

And here the script-runner

Usage

usage: run [<options> [cmd..]..]

options:
-h, --help         output usage information
-v, --verbose      verbose logging (not implemented yet)

    --silent       suppress output of children
-t, --test         no runing only show process structure
-s, --sequential   following cmds will be run in sequenz
-p, --parallel     following cmds will be run in parallel
-i, --ignore       the following cmd will be ignored for --first, --wait and errors
-f, --first        only in parallel block: close all sibling processes after first exits (succes/error)
-w, --wait         only in parallel block: will not close sibling processes on error
-m, --master       only in parallel block: close all sibling processes when the following cmd exits. exitCode will only depend on master
-f, --first        close all sibling processes after first exits (succes/error)

run also looks in node_modules/.bin for cmds
run-para is a shorthand for run --parallel
run-seq is a longhand for run
run-npm will match cmd with npm script and replace them, usage of globs is allowed

run-npm currently uses minimatch, but I'm not fully statisfied

I thought about templating of scripts:
Say we have a script "test":"mocha :'1'"

npm run test # would run mocha
npm-run test # would run mocha
npm-run test(--watch) # would run mocha --watch

It would be not to hard to implement, but I can't think of a use case currently

Answer 23 · 2016-03-20T15:11:05.000Z

Sorry to butt in. I came across parallelshell from an adventure in building with VS Code. It didn't work, so I went to its site, which led me here. Anyway...

Quoth the @kimmobrunfeldt:

In @keithamus' example:
$ npm-run-all --sequential a b c --parallel d e f --sequential g h i
# To do this with three commands:
$ run-series a b c && run-parallel d e f && run-series g h i
# Or
$ run-series 'run-sequential a b c' 'run-parallel d e f' 'run-sequential g h i'
I like 2nd(this is not an option because lack of windows support) and 3rd more because they explicitly state which steps are run sequentially and which are run parallel. The 1st command has implicit hidden information: all those substeps are run sequentially.

I'm confused as to your Windows comment. Using && works the same for me in Windows as it does in Unix/Linux--it runs the commands in series if the previous command succeeded. Maybe it changed with recent versions of Windows... What doesn't work the same is &. Also, what is the difference between run-series and run-sequential in the above example? Couldn't you just do run-sequential 'run-sequential a b c' 'run-parallel d e f' 'run-sequential g h i'? Honestly, with &&, I'm not sure run-series is even necessary, that's what && does (technically, that's what ; on Unix/Linux and & on Windows does; && also checks exit codes). I believe that the third command is irrelevant because of this.

I agree that the first command is bad in that it includes an implicit behavior, another reason I prefer the second example. I would suggest that, if you boil this down to one command with aliases, that the one command considers --sequential and --parallel mutually exclusive. Having used commands with ordered parameters in the past, it can easily become a PITA, requires additional knowledge, is more error-prone and isn't very intuitive, IMHO.

The second command, however, does brings up a question: what to do with exit codes (if anything)? None of these examples use || in case one (or more? or all?) of the parallel commands fails. This is where I think the decomposition and flexibility in the second example really shows its value. For example, imagine something like this (which isn't great, I just tried to make it more realistic than a b c):

run-series clean initEmail
    && run-parallel --must-succeed build 'sendEmail --subj="building \"${projName}\"" foo@bar.com'
    && run-series --any-must-succeed 'minify -d app/js -r "^.*\\.js"' imgify
        && run-parallel emailDev emailMgrs
        || run-series pkgErr emailDev

Although, since run-series is (so far, as I understand it) redundant, this could really be:

npm run clean && npm run initEmail
    && npm run-all --all-must-succeed build 'sendEmail --subj="building \"${projName}\"" foo@bar.com'
    && npm run minify -- -d app/js -r "^.*\\.js" && npm run imgify
        && npm run-all --any-must-succeed emailDev emailMgrs
        || npm run pkgErr && npm run emailDev

I could see needing run-series in order to bridge the gap between ; and & to allow users to run commands cross-platform in series regardless of exit code.

Lastly, I am very glad you all agreed to combine your efforts. There is way too much fragmentation and duplication of purpose and effort these days.

Answer 24 · 2016-03-20T21:47:17.000Z

I think we can agree this is a difficult topic.. we only have 1 line, no syntax highlighting and ' as an indicator for nesting.

With npm run we can get hold on the 1 line and nesting problem. But with the syntax we have a clear tradeoff between verbosity and clarity. I think the best way is to offer both possibilities..

I'm strongly for ordered parameters as &&, || and all other also have a significant order and everything else would be confusing.

So for you example, I think a desireable syntax would be:

run-npm clean initEmail buildAndEmail minify --any emailDev emailMgrs --on-error pkgErr emailDev

with all nested commands hidden behind own npm scripts.

Answer 25 · 2016-05-10T03:19:40.000Z

Hi all, may I know what's the status of the consolidation of all projects?
Is there any possibility to create a Github organization and put all related repos in there?

Answer 26 · 2016-07-07T03:48:39.000Z

Which project is the one where you are all merging into ?

Answer 27 · 2016-07-07T07:35:10.000Z

it's npm-run-all.

Answer 28 · 2016-09-18T19:09:04.000Z

I'm confused as to your Windows comment. Using && works the same for me in Windows as it does in Unix/Linux--it runs the commands in series if the previous command succeeded. Maybe it changed with recent versions of Windows...

Ok thanks for clarifying, I wasn't sure as I haven't used a windows shell for a while.

Answer 29 · 2016-12-22T21:40:40.000Z

Any update on this?

Answer 30 · 2017-05-18T21:48:41.000Z

Slightly different and slightly related, a project I recently made:
makfy

Answer 31 · 2017-10-23T14:00:59.000Z

@keithamus Has npm-run-all already become the one canonical parallelisation tool that you proposed ?

Answer 32 · 2017-10-24T08:31:54.000Z

Yes I would say so. Sadly I didn't get any time to work on it 😊 but I'd say it's now the tool I wanted parallelshell to be.

Answer 33 · 2017-10-24T09:27:59.000Z

That's awesome to hear. Great collaboration.
The sad thing is though that I'm going to use npm-run-all now.

Sadly I didn't get any time to work on it 😊

Well I don't understand that you guys have/find the time to build all these tools and maintain and support it. I must be a lot slower, or just less productive.
Currently rereading your article "How to use npm as a build tool", to rewrite my obnoxious large Gulp and Grunt scripts. Thanks for that too.

Answer 34 · 2017-11-13T06:45:54.000Z

From @pygy https://github.com/pygy/gosub

What does it have over npm-run-all? It passes parameters to subtasks. That's all. It doesn't take any options and can't run tasks in parallel unlike npm-run-all. ※

Answer 35 · 2017-11-30T22:33:04.000Z

@kimmobrunfeldt commented on Oct 17, 2015:

[…] because concurrently is not tied to npm run scripts, you can also use the tool for any shell command.

Is this goal achieved in this consolidation? As the name of this package suggests, I think, this is still pretty much tied to npm scripts. Am I wrong?

Thanks. 🙏

Answer 36 · 2018-01-14T12:49:50.000Z

Is the consolidation of all these projects into one megarepo still taking place? Cheers!

Answer 37 · 2018-01-14T18:41:51.000Z

Hi all I had originally missed this thread. I wrote gosub to scratch an itch (running subscripts in a npm/yarn-agnostic fashion), I didn't know that npm-run-all was doing it as well.

I'll probably keep using it for my libs given how simple it is (no options, does exatcly what I want), but I can add a banner at the top of the README pointing to whatever the meta-project ends up being.

Answer 38 · 2019-03-02T17:56:31.000Z

Set exec to a script:

"scripts": {
	"exec": "exec",
	"dev": "del dist; run-p 'exec -- tsc -w' 'exec -- nodemon'",
},

👍

Answer 39 · 2019-03-26T01:49:22.000Z

How to use this, to start multiple projects? For example, something like

Root
 |-- server/package.json
 |-- client/package.json
 |-- ui/package.json

Need to start the server, client and ui projects all - does this project solve that problem? how to achieve it?

Answer 40 · 2019-03-26T03:32:22.000Z

@KrishnaPG, it seems that your question is unrelated to this issue. Non-the-less, I'll try to help you. Try the run-p command. If you have any more questions, please open a new issue and tag me in it.

Answer 41 · 2020-05-05T20:27:52.000Z

I recently stumbled on something similar to the tools listed in the OP, but the one I found automatically took each command and multiplexed it into a configurable layout (like tmux terminal multiplexer).

The layout was specified in package.json of a project. It was really neat, because it would show each command's output separately in their own sub-panels.

But I can not find this package. I should've saved it!

EDIT: Okay, I found this: https://www.npmjs.com/package/multiplerun. That's similar to what I stumbled on before, but that one is specifically for use with iTerm in macOS. The one I had stumbled on was pure Node.js (I think) and would therefore run in any OS, in any terminal.

EDIT: There's also https://www.npmjs.com/package/stmux, but it isn't the one I found before.

Answer 42 · 2020-05-06T15:02:41.000Z

@trusktr searching for tmux on npm I found this: https://www.npmjs.com/package/multiview