jimweirich/rake

small brain teaser

MarcWeber opened this issue · 8 comments

MarcWeber commented 
rake -m  -j 4 -t abc

How long is this going to run?

multitask "a" do
  sh "echo a > a; sleep 2"
end

multitask "b" => "a" do
  sh "echo b > b; sleep 1"
end

multitask "c" => "b" do
  sh "echo c > c; sleep 1"
end

multitask "z" do
  sh 'sleep 1'
  rasie "bad"
end

multitask "abc" => ["c", "z"] do
end

Seriously: Why does'nt it run z and a in parallel, then stopping after 2 secs because "z" fails?

I mean devs want both: "--keep-going-in-case-of-failures" and "--stop-on-first-failure" like behaviour, fail early is more important. Am I missunderstanding anything this time?

jimweirich commented

I haven't reviewed the code, but I suspect that the exception is only transmitted to the main task when it waits for the result of task :z. Since task :z is run in a separate thread, exceptions thrown in that thread are not seen in the main thread.

MarcWeber commented

rake's documentation says "rake's version of makefiles" - as dev - I think its not enough. Worse: I somehow feel that it would have taken me less time to rewrite something "rake" like from scratch than trying to understand why -j 1 silently behaves like -j 2, why I should bother the user to lookup whether -m is ok or not etc. You know the rake community better than I do. Is it worth fixing? If so in which way? I'm not happy with drake either (without -j it behaves similar to rake, and with -j 4 it takes too long till it starts a build for whatever reason) Is there yet another alternative which is more dev and user friendly? Is there yet another library I may want to look at?

Thanks for your support. Its appreciated. I just don't think that the design of rake fits my needs very well.

jimweirich commented

Thanks for the feedback, but I'm not sure what you are asking to be fixed. The -j1/-j2 issue? Removing -m? The fail early issue? User friendliness issues?

If you wish to make a concrete proposal, I would be happy to evaluate it.

FYI: the -j1/j2 issue is already fixed in HEAD.

MarcWeber commented

There should be 2 ways to run rake:

  • fail as early as possible whenever a task failed (keep running tasks running)
  • fail as late as possible - try to run as many tasks as possible successfully (thus continue till no tasks are left which could be run because prerequisites failed)

Imagine task a and b taking 2min to run. But you have to rerun them (for whatever reason, eg because you use clean because something rake doesn't track has changed). Then why run b if z failed? If you're in a console the difference may not be that "big", because you can abort by ctrl-c. If you have to use editors loading the result into an error list aborting manually is not that nice. Fail early is what I want then.

So I want rake to abort. And I want a way to specify "-m" in the Rakefile or have a nice way to make a task a "multitask". The user should just run "rake target" without bothering thinking about -m or parallel jobs.

I'm aware than "-m" cannot (and should not) be removed for compatibility reasons.

jimweirich commented

Go ahead an make a feature request for the fail early part (i.e. open a specific issue for just that feature).

Then review issue #205 and see if that meets your desire for an easy way of making any task or file a "multitask". Feel free to add comments there.

Feel free to close this discussion thread once you've done that.

MarcWeber commented

Fixing "rake" and testing would have taken too much time.

Instead I've implemented a minimal alternative make like system for Ruby which does exactly what I need:
https://github.com/MarcWeber/rumake

There is yet another new feature: It builds the dependency chain which will take longest first.
Eg if there are 100 tasks not depending on anything and 20 task depending on each other
one thread will start with the 20 chain first.

Rake users can give it a try to find out how much value this provides to them for their projects.
Then it can be discussed whether those features are worth integrating into rake.

You may want to keep this issue open for a while or add this as "future ideas" to
your README to gather more feedback about whether users (beside me) want such.

jimweirich commented

Thanks for the feedback. I'll keep this request in mind, but I'm going to go ahead a close it for now.

MarcWeber commented

small update for those who are following this thread: I implemented "eta" feature and redoing a task if the command line changes. eta and previous commands are stored in a cache file