does it support mutiple nodes mutiple GPU fine tuning？

[qingyuan18]

RT, didn't find multiple nodes multiple GPU ‘s fine tuning command guide (for chatglm)

does it use accelerate / deepspeed to do parallel training？
I can see the node_rank parameters in the script:
python3 -m torch.distributed.launch --nproc_per_node 4
--nnodes=1 --node_rank=0 --master_addr=xxx --master_port=yyy
uniform_finetune.py --model_type chatglm --model_name_or_path THUDM/chatglm-6b
--data alpaca-belle-cot --lora_target_modules query_key_value
--lora_r 32 --lora_alpha 32 --lora_dropout 0.1 --per_gpu_train_batch_size 2
--learning_rate 2e-5 --epochs 1

[dkqkxx]

your can use torchrun
torchrun --nnodes 1 --nproc_per_node $ngpu uniform_finetune.py $args --data $data