Code for training llama pro?

Question

Code for training llama pro?

Opened this issue 10 months ago · 8 comments

yhyu13 commented 10 months ago

Hi,

If I get it correctly, you have used code from https://github.com/allenai/open-instruct as base.

Would you release the full code of reproducing llama2 pro 8B?

Thanks!

Answer 1 · 2024-01-07T16:10:34.000Z

Yes, of course. I will organize the code recently. Thanks for your interest.

Answer 2 · 2024-01-09T13:23:04.000Z

Hey @hills-code ,could you also add code for converting a model by adding identity blocks for training ?
I am excited to use similar techniques for other open-source models like Qwen!!
Thanks!:)

Answer 3 · 2024-01-09T16:01:25.000Z

Hey @hills-code ,could you also add code for converting a model by adding identity blocks for training ? I am excited to use similar techniques for other open-source models like Qwen!! Thanks!:)

I have added the block expansion script under the folder scripts. You can check it for reference. Hope it will be helpful!

Answer 4 · 2024-01-09T16:37:50.000Z

i will check that out,Thanks!

Answer 5 · 2024-01-10T08:05:34.000Z

@hills-code i was able to get it to work! Btw you only train the added layers and not even the lm head?
and also do you think directly going for SFT by skipping pretraining will work?

Answer 6 · 2024-01-12T03:03:59.000Z

@raghavgarg97 No, I don't think you can skip the "post-training" for extra blocks on some new corpus (e.g. bigcode, as mentioned in the paper) before applying tuning bc the purpose of LLaMA Pro is sort of "add new ability w/o forgetting by introducing more layers". The new ability gained is training new layers on new corpus.

Answer 7 · 2024-01-13T11:29:38.000Z

does anybody know how to continue pre-training LLaMA Pro? @yhyu13 @raghavgarg97 @yxgeee @hills-code

Answer 8 · 2024-01-16T12:49:47.000Z

nah，code has not been released yet