Improve Parallelization
Closed this issue · 1 comments
Currently the app is getting stuck yuce.erlang-otp as its final, despite the fact that the file is NOT big at all. I'm not entirely sure why its getting stuck. This is a problematic thing.
Also I'm not a massive fan of this threading thing. First thing is I'm not even sure if its the most effective parallelization that I could be doing, and secondly is that I want it to flow in based on whichever one finishes first, like a Queue of sorts. The if not task.done() thing is not the most ideal solution here.
Update! So I discovered this was largely due to my network card being overloaded with too many network requests across multiple threads, as this issue was not present on my beefier and more powerful pc (in fact the script worked 2x as fast!). Based on this I decided to give control of the threads to the end-user, where it uses a very sane default of nproc threads, but can be adjusted according to whatever the user feels like. This not only absolves me of all responsibility for the script being too slow or too network-throttley, but it also gives the user control to decide based on their own hardware.