work distribution in differents thread based on hash
hea-p1sec opened this issue · 4 comments
I would like to allocate some specific work to some specific thread based on some criteria.
So for this need, I plan to extend C-Thread-Pool to support multiple queues. Perhaps, one queue by worker thread is the right approach. And the new job will be distributed to the right queue based on hash computation in my application.
Could you provide some guidance about the design? I would like to contribute and I want to do it in a clean way with a clean design
Hiya and thanks first of all for considering extending the existing implementation.
Regarding your needs, you can currently create multiple thread-pools. So how would your suggestion differ from say, creating 2 separate thread pools with each having a single thread? Then you would effectively have two queues, with one thread each which sounds similar to what you mentioned.
The only problem with the current implementation (and from what I call recall) is that although you can pause on a thread-pool basis, resuming resumes all thread pools, so that could potentially be an obstacle for the user in your scenario. Fixing this issue in a POSIX compliant way would be one way of solving this and fitting the current implementation to your case scenario. There was a pull requests with a fix before which however did not get merged since it wasn't POSIX compliant.
Hi,
Thanks for the quick reply, indeed your proposal sounds good, it should fulfill my needs.
Regarding pause and resume feature, I am not sure I got your point, I plan to not use this future in my application. So the current implementation is fine. Please correct me, if I am missing something here.
The thing is that although you have 2 queues, there are scenarios (yours might not be it) where you have to synchronize the two thread pools. For example you might be wishing to wait for one thread pool to finish so that you can feed the output of that into the next threadpool. With the current implementation and due to the wait/resume issues I mentioned, this scenario would not work.
Ah OK, I understand your point now and indeed my scenario doesn't require this behavior. Every thread pool won't be interdependent, this is my main goal and I will achieve this by computing a specific hash (for job distribution on the relevant thread pool) based on some specific properties which will make the jobs independent.