About using multiple GPUs to do lisa fine-tuning

Question

About using multiple GPUs to do lisa fine-tuning

orderer0001 opened this issue 8 months ago · 4 comments

If there are multiple GPUs, using the lisa method is also a direct script./scripts/run_finetune_with_lisa.sh? Do I need to set multi-GPU parameters?

Answer 1 · 2024-04-19T07:03:35.000Z

Thanks for your interest in LMFlow! Currently we are working on the full multi-GPU support for LISA. Model parallelism is not integrated yet. If you run the script directly, it is data parallelism and may require more memory than its single GPU version.

Please stay tuned for our latest update, thanks for your understanding 🙏

Answer 2 · 2024-04-19T07:32:42.000Z

Can other training methods be configured with multiple GPUs? Do I need to set parameters manually?

Answer 3 · 2024-04-19T07:41:12.000Z

When will Lisa training’s support for multiple GPUs be updated?

Answer 4 · 2024-04-20T14:49:01.000Z

Can other training methods be configured with multiple GPUs? Do I need to set parameters manually?

Yes. You may use ./scripts/run_finetune.sh, that script support model parallelism by utilizing deepspeed zero3.