with-shutdown! macro not working

[abhigogna]

Tried running the following code

(cp/with-shutdown! [pool (cp/threadpool 4)]
  (do
    (cp/future pool (+ 1 2))
    (cp/pmap pool inc (range 10))
    (cp/pvalues pool (str "si" "mul") (str "ta" "neous"))
    (cp/pfor pool [i (range 10)]
             (* i (- i 2)))))

got this exception

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@25b45d20
   rejected from
   java.util.concurrent.ScheduledThreadPoolExecutor@7332b3f7[Terminated,
   pool size = 0, active threads = 0, queued tasks = 0, completed
   tasks = 13]

For what its worth I am on Java 8 and clojure 1.8.

[leon-barrett]

Unfortunately, that's how with-shutdown! works, and it's in the README and docstring: https://github.com/TheClimateCorporation/claypoole/blob/master/src/clj/com/climate/claypoole.clj#L201

The issue is basically that there's no way to safely shut down the pool in this case because some of the tasks may not even have been sent to the pool by the time your with-shutdown! block has ended--if you chained pmaps, then some inputs might not yet be ready. Instead, I can only recommend that you either dereference the results from the threadpool before killing the threadpool.

How could I help make that documentation clearer or easier to find? Also, what were you hoping would happen here?

[abhigogna]

Leon,

First of all, thanks for this wonderful lib.

Since this code snippet was part of your blog, referenced in readme, I thought this was suppose to work. As I was working through the tutorial, I realized what had happened. I had assumed that somehow with-shutdown! will make sure all the futures or pmaps have been realized before shutting down the threadpool.

My only suggestion is put the with-shutdown! code at the bottom of the tutorial. Also, if we can't really rely on this then why not just remove it altogether, until we make it more reliable. A non working code can be more harmful then non-existing.

[leon-barrett]

Ah, I see, that makes sense. I'll try to make the readme better; I guess that snippet is more confusing than helpful.

I think with-shutdown! is useful even if it can cause exceptions when not used carefully. I find it useful to know that my resources are definitely cleaned up correctly, which is otherwise a weakness of thread pools on the JVM.