Add accelerate to poetry dependencies

[liutianlin0121]

Hey Costa,

It seems that the 'accelerate' package isn't part of the poetry dependencies, although we're currently using it. I can submit a PR to update the pyproject.toml and poetry.lock files to include accelerate.

In addition, I encountered errors in calling `reward_model.module.reward_gain` and `reward_model.module.reward_bias` in the reward learning script, like [here](https://github.com/vwxyzjn/lm-human-preference-details/blob/main/lm_human_preference_details/train_reward_accelerate.py#L384-L385), and [here](https://github.com/vwxyzjn/lm-human-preference-details/blob/main/lm_human_preference_details/train_reward_accelerate.py#L411-L412). The error message is `AttributeError: 'AutoModelForCausalLMWithRewardHead' object has no attribute 'module' ` for me. However, directly using `reward_model.reward_gain` and `reward_model.reward_bias` work just fine. Is there a reason why `module` is important? Should we change `reward_model.module.reward_gain` to `reward_model.reward_gain`?

(I see! reward_model.module is needed when using multiple GPUs with accelerate.)

Tianlin

[vwxyzjn]

Hi @liutianlin0121, thanks for this issue. I can't believe that I didn't add accelerate to poetry. Could you make a PR? You can probably run poetry add accelerate@latest.

(I see! reward_model.module is needed when using multiple GPUs with accelerate.)

Yeah, so this is an unfortunate limitation of torch: for some reason under the distributed mode, the module is no longer directly accessible. Torch also kind of forces you to use exclusively forward function. In comparison, jax doesn't have these two issues.

[liutianlin0121]

I see, thank you!