[testing] task and config are not inferred from checkpoint path?

[chauvinSimon]

my question: how can I test an agent trained with a non-default cfg/task/XXX.yaml config?

---

I have trained two agents on the FrankaCubeStack task:

• Agent1 with
  • the default config: cfg/task/FrankaCubeStack.yaml
  • the argument experiment=FrankaCubeStackDefault
• Agent2 with
  • controlType: joint_tor as modification in cfg/task/FrankaCubeStack.yaml
  • the argument experiment=FrankaCubeStackJT

After training, runs/ contains two folders:

• FrankaCubeStackDefault
• FrankaCubeStackJT

Each contains a different config.yaml:

• runs/FrankaCubeStackDefault/config.yaml has controlType: osc
• runs/FrankaCubeStackJT/config.yaml has controlType: joint_tor

---

Now I would like to evaluate (test) them

I would expect such a command to work:

• python train.py test=True checkpoint=runs/FrankaCubeStackJointTor/nn/FrankaCubeStackJointTor.pth num_envs=64
• I would expect the path config.yaml to be derived from the checkpoint, and, in it, the task.name: FrankaCubeStack to be used
• but no: the used task is Ant, which is the default in cfg/config.yaml

When adding the task=FrankaCubeStack argument:

• python train.py test=True task=FrankaCubeStack checkpoint=runs/FrankaCubeStackJointTor/nn/FrankaCubeStackJointTor.pth num_envs=64
• the task config cfg/task/FrankaCubeStack.yaml is used (with controlType: osc), not runs/FrankaCubeStackJointTor/config.yaml which was saved and used for training! 🤯

I probably have missed something there, and would be grateful if you could help me 🙏