What does local rank mean in code

[Humanpaladin]

I don't quite understand what "local_rank" means in the code, could you explain it please? Thank you.

[lixin4ever]

Thanks for your attention.

The argument "local_rank" is used to support DistributedDataParallel (DDP).

Concretely, the "local_rank" stores some local & unique information for the process, e.g., the id of the current gpu, and it can be regarded as the local identifier of each process.

Our project is built on top of DataParallel (DP) rather than DDP, thus, you can simply leave this argument alone or adopt the default setting of this argument (i.e., set "local_rank" as -1).

[Humanpaladin]

Thank you very much for your reply.
Maybe I should learn something about the DDP later.