facebookresearch/habitat-lab

How to keep sensor observations from gui controlled agent?

Closed this issue · 1 comments

ambervg commented

❓ Questions and Help

At the end of the rearrange demo I want to make a video with observations from the visual sensors. The following visual sensors seem to available:

  • 'agent_0_head_rgb'
  • 'agent_0_head_depth'
  • 'agent_0_articulated_agent_arm_rgb'
  • 'agent_0_articulated_agent_arm_depth'

The sensors from gui controlled agent (agent_1) are not in there. If I understand correctly, that is because these are removed here:

habitat-lab/habitat-hitl/habitat_hitl/_internal/config_helper.py

Line 77 in af326bd

gui_controlled_agent_config.sim_sensors.clear()

I would like to keep these visual sensors from agent_1 so that I can save their observations into a video after the simulation. To do this I tried doing the same as here. So, I did the following:

  1. Adding the following to hitl_rearrange.yaml:
habitat_hitl:
    remove_gui_sensors: False,
    remove_visual_sensors: False,
  1. Make the removal of the gui/visual sensors conditional by changing this in habitat-hitl/habitat-hitl/_internal/config_helper.py:
# Line 79
if config.habitat_hitl.remove_gui_sensors:
      gui_controlled_agent_config.sim_sensors.clear()

# Line 103
if config.habitat_hitl.remove_visual_sensors:
        sim_sensor_names = [
            "head_depth",
            "head_rgb",
            "articulated_agent_arm_depth",
        ]
        for sensor_name in sim_sensor_names + lab_sensor_names:
            sensor_name = (
                sensor_name
                if len(sim_config.agents) == 1
                else (f"{gui_agent_key}_{sensor_name}")
            )
            if sensor_name in gym_obs_keys:
                gym_obs_keys.remove(sensor_name)

However, this is giving me the following error:

Traceback (most recent call last):
  File "/home/amber/my-thesis/habitat_experimenting/rearrange_step2.py", line 610, in main
    hitl_main(config, create_app_state)
  File "/home/amber/habitat-lab/habitat-hitl/habitat_hitl/core/hitl_main.py", line 63, in hitl_main
    hitl_headed_main(hitl_config, app_config, create_app_state_lambda)
  File "/home/amber/habitat-lab/habitat-hitl/habitat_hitl/core/hitl_main.py", line 129, in hitl_headed_main
    driver = HitlDriver(
  File "/home/amber/habitat-lab/habitat-hitl/habitat_hitl/_internal/hitl_driver.py", line 67, in wrapper
    return callable_(*args, **kwds)
  File "/home/amber/habitat-lab/habitat-hitl/habitat_hitl/_internal/hitl_driver.py", line 233, in __init__
    self._reset_environment()
  File "/home/amber/habitat-lab/habitat-hitl/habitat_hitl/_internal/hitl_driver.py", line 376, in _reset_environment
    self._obs, self._metrics = self.gym_habitat_env.reset(return_info=True)
  File "/home/amber/miniconda3/envs/habitat/lib/python3.9/site-packages/gym/core.py", line 283, in reset
    return self.env.reset(**kwargs)
  File "/home/amber/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 323, in reset
    return self._transform_obs(obs), info
  File "/home/amber/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 289, in _transform_obs
    [(k, obs[k]) for k in self._gym_obs_keys]
  File "/home/amber/habitat-lab/habitat-lab/habitat/gym/gym_wrapper.py", line 289, in <listcomp>
    [(k, obs[k]) for k in self._gym_obs_keys]
KeyError: 'agent_1_has_finished_oracle_nav'

Any tips on how to solve this?

ℹ️ System specs

OS: Ubuntu 22.04.1
Architecture: x86_64
habitat-sim version: 0.3.0
habitat-lab version: 0.3.0
habitat-hitl version: 0.3.0
habitat-baselines version: 0.3.0