araffin/sbx

[Question] framestack and train_freq for sbx

Opened this issue · 2 comments

Jackflyingzzz commented

Hi there!

Thank you for developing SBX! I'm currently working with SB3 for real-time robot control and was wondering if SBX supports the framestack using DummyVecEnv wrapper? Additionally, can SBX algorithms like crossQ perform update/gradient steps per episode instead of per RL step by adjusting the train_freq parameter (e.g., from (5, "step") to (2, "episode"))? This feature is crucial for my real-time RL loop.

Thanks again for your help!

Best regards,
Jacky

araffin commented

Hello,

real-time robot control and was wondering if SBX supports the framestack using DummyVecEnv wrapper? A

VecFrameStack, yes

Additionally, can SBX algorithms like crossQ perform update/gradient steps per episode instead of per RL step by adjusting the train_freq parameter (e.g., from (5, "step") to (2,

It should work but it might want to re-compile the training loop if the number of step changes.
Also, this jax version should be fast enough to use with robot (I did use it at 30Hz with 10 gradient steps per step with the DroQ configuration of SAC).

Jackflyingzzz commented

Hello,

real-time robot control and was wondering if SBX supports the framestack using DummyVecEnv wrapper? A

VecFrameStack, yes

Additionally, can SBX algorithms like crossQ perform update/gradient steps per episode instead of per RL step by adjusting the train_freq parameter (e.g., from (5, "step") to (2,

It should work but it might want to re-compile the training loop if the number of step changes. Also, this jax version should be fast enough to use with robot (I did use it at 30Hz with 10 gradient steps per step with the DroQ configuration of SAC).

wow glad to hear that, thanks for sharing!