robot-descriptions/robot_descriptions.py

Race condition when importing the same description in parallel

Closed this issue · 0 comments

Bug description

Actually, even if it the description is already in cache, we will run into an error similar to #114. Adapted the parallel command from that issue to run on my machine:

parallel -i python -c 'from robot_descriptions import fr3_mj_description; print({}, fr3_mj_description.MJCF_PATH)' -- 1 2 3 4
Error: Another git process seems to be running in this repository, ...
3 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
Traceback (most recent call last):
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 90, in clone_to_directory
    clone.git.checkout(commit)
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 800, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1386, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1183, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout 3d593a327ff8d5efd10f4926ccc477820e9252c1
  stderr: 'fatal: Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/fr3_mj_description.py", line 14, in <module>
    REPOSITORY_PATH: str = _clone_to_cache(
                           ^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 139, in clone_to_cache
    clone = clone_to_directory(
            ^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 98, in clone_to_directory
    clone.git.checkout(commit)
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 800, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1386, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1183, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout 3d593a327ff8d5efd10f4926ccc477820e9252c1
  stderr: 'fatal: Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.'
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Found commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 successfully!
4 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Found commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 successfully!
1 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml

At first read I had a hunch this error is related to the quick-and-dirty way we pass arguments via environment variables, which won't work well when we run in parallel. But in this instance the exception is rather raised from:

try:
clone.git.checkout(commit)
except GitCommandError:
print(
f"Commit {commit} not found, "
"let's fetch origin and try again..."
)

And the (currently uncaught) exception message reads:

Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.