Question - Ray parallelization
Closed this issue · 3 comments
serbinsh commented
Hello
I am experimenting with running hytools image corrections on our cluster and noticed that while it states running on 4 CPUs, most of the time the processes is rather sequential or low CPU usage? There are short periods where the number of CPUs selected (4) are used concurrently, but a lot of other times they are not?
Is this normal? It looks like the parallelization is running images on a CPU correct? Such that it would not help to have more CPUs availible than images?
serbinsh commented
Also FYI - here is a repo we are developing for running hytools jobs on our cluster. Still very much a WIP
oldmanye007 commented
Basically, it is a coarse-grained parallelism. Yes, if available
CPUs>number of images, only one CPU is paired to one image.
…On Wed, Mar 9, 2022 at 1:40 PM Shawn P. Serbin ***@***.***> wrote:
Hello
I am experimenting with running hytools image corrections on our cluster
and noticed that while it states running on 4 CPUs, most of the time the
processes is rather sequential or low CPU usage? There are short periods
where the number of CPUs selected (4) are used concurrently, but a lot of
other times they are not?
[image: Screen Shot 2022-03-09 at 2 35 32 PM]
<https://user-images.githubusercontent.com/3067369/157519388-aa035466-fe83-4ad1-9033-cffda0a954a5.png>
[image: Screen Shot 2022-03-09 at 2 37 37 PM]
<https://user-images.githubusercontent.com/3067369/157519448-110caaed-b13b-4734-8573-ae4e0909d035.png>
Is this normal? It looks like the parallelization is running images on a
CPU correct? Such that it would not help to have more CPUs availible than
images?
—
Reply to this email directly, view it on GitHub
<#9>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABEO2NVWJSUUJ4VHJRKYVO3U7D5BVANCNFSM5QKQXDQA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
serbinsh commented
OK thanks @oldmanye007