rs_int_example failing

[f1ren]

Describe the bug
rs_int_example.py failing on An error occurred during each attempt of this action.

To Reproduce
Steps to reproduce the behavior:

1. ./run_docker.sh
2. python -m omnigibson.examples.action_primitives.rs_int_example

Expected behavior
Program ends after printing Finished executing place.

Screenshots

                  Plan has 2 steps
                  Executing arm movement plan step 1/2
                  Executing arm movement plan step 2/2
Traceback (most recent call last):
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 70, in <module>
    main()
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 60, in main
    execute_controller(controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple), env)
  File "/root/omnigibson/omnigibson/examples/action_primitives/rs_int_example.py", line 18, in execute_controller
    for action in ctrl_gen:
  File "/root/omnigibson/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 545, in apply_ref
    raise ActionPrimitiveErrorGroup(errors)
omnigibson.action_primitives.action_primitive_set_base.ActionPrimitiveErrorGroup: An error occurred during each attempt of this action.

Attempt 0: EXECUTION_ERROR: Could not navigate to the target position. Additional info: {'target pose': (tensor([ 0.0310, -0.6776,  0.0100]), tensor([ 0.0000,  0.0000, -0.8991,  0.4377]))}

Attempt 1: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Attempt 2: POST_CONDITION_ERROR: Grasp completed, but no object detected in hand after executing grasp. Additional info: {'target object': 'apple'}

Desktop (please complete the following information):

• OS: Ubuntu 24.04.1 LTS
• Isaac Sim Version (see docker)
• OmniGibson Version 1.1.1 (docker)

Additional context
The behavior is non-deterministic and different attempts are printed depending on state at failure.

[hang-yin]

Hi @f1ren , thanks for reaching out!

Since the action primitives use sampling-based motion planning, it is expected that the robot behavior can be non-deterministic. This primitives feature is also somewhat experimental and needs a lot more tuning. We are actively working on making this more robust and experimenting with alternative motion planners. But for now, you can try the following to make the primitives more fault-tolerant:

In omnigibson/examples/action_primitives/rs_int_example.py, you can look for controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, apple) and controller.apply_ref(StarterSemanticActionPrimitiveSet.PLACE_ON_TOP, cabinet). This apply_ref function has an attempts argument, which defaults to 3. You can try to make this higher either 1) directly in the example you are running or 2) make the default higher in omnigibson/action_primitives/stater_semantic_action_primitives.py::StarterSemanticActionPrimitives::apply_ref.

We do have some really exciting progress, stay tuned for our next few releases!