GPU requirements

Question

GPU requirements

ejarkm opened this issue a year ago · 3 comments

Hi, first of all thanks for your work. I will definitely give it a try.

I was wondering if you could share some information about the training time and which GPUs you needed for train the model and if you have some recommendations about the size of the datasets you used.

Thanks a lot once again and merry Christmas!

Answer 1 · 2022-12-25T01:57:02.000Z

This model hasn't been trained by many, since PaLM is exclusive to Google and this repo is open-source alternative.

The model size is configurable through parameters, though smaller models train faster, bigger models perform better and for PaLM it has advantage on model scaling, meaning improvements do not get stuck on bigger model size.

Answer 2 · 2022-12-25T01:59:07.000Z

For pretraining datasets you can check Eleuther.ai. For RLHF fine-tuning datasets, you can check hh-rlhf from Anthropic and following-instructions-human-feedback from OpenAI.

Answer 3 · 2022-12-25T09:27:04.000Z

Thanks a lot for your reply, this is really useful information :)