general_train.py: error: unrecognized arguments: --local-rank=0

[kbatsuren]

Hi @chengxuz,

When I run this example command on Google Colab, I run into the following error as unrecognized arguments: --local-rank=0.

!python -m torch.distributed.launch --nproc_per_node=1 --master_port=29123 baseline-pretraining/scripts/general_train.py --setting "BabyLM/exp_strict.py:opt125m_s1"

Output:

/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
usage: general_train.py
       [-h]
       [--setting SETTING]
       [--local_rank LOCAL_RANK]
       [--train_upto_epoch TRAIN_UPTO_EPOCH]
       [--exp_id EXP_ID]
       [--opt_model_size OPT_MODEL_SIZE]
general_train.py: error: unrecognized arguments: --local-rank=0
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 3049) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 196, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
baseline-pretraining/scripts/general_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-05-30_17:35:49
  host      : a29ac89ae867
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 3049)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

But I am still unsure about if it is related to Colab environments.

[chengxuz]

This looks like a combination of both python version difference (I have never tried 3.10, mine is 3.9) and torch version differentce? Can you try to replace the beginning python -m torch.distributed.launch with torchrun? If that does not work, maybe try to just naively replace the local_rank parser argument by local-rank?

[kbatsuren]

Thanks so much! torchrun is working without any issues. :)