[BUG] pytorch distributed training task is unschedulable when using volcano gang scheduling

Question

[BUG] pytorch distributed training task is unschedulable when using volcano gang scheduling

Closed this issue 3 years ago · 5 comments

What happened:
I can not submit pytroch training task successfully when using volcano as scheduler. After discuss with @shinytang6 and @SimonCqk , they find that kubedl DAG is in conflict with volcano gang scheduling.
For me detail info, please see volcano-sh/volcano#1959

What you expected to happen:
make pytroch training job which mentioned above schedulable when using volcano gang scheduler.

kerthcet commented 3 years ago

/cc

Answer 1 · 2022-01-20T02:07:02.000Z

@CaRRotOne @kerthcet Hi there, as we discussed in volcano-sh/volcano#1959, I'm working on this to fix the issue

Answer 2 · 2022-01-20T02:09:45.000Z

Please let me known if there's any progress. Thanks a lot.

Answer 3 · 2022-01-24T10:15:25.000Z

@kerthcet @CaRRotOne hi, I have posted a new pull request to track & fix this issue.

Answer 4 · 2022-01-25T05:09:09.000Z

@CaRRotOne @kerthcet I'll build a latest image tagged with daily and push to docker hub, if you want to try as soon as possible, please pull the latest commits in master branch :)