generated results are terrible with bloom
raihan0824 opened this issue · 4 comments
I use this repo to finetune bloomz-7b1-mt with alpaca data (50k conversation) and the results are terrible. It takes 8 hours to train with the same arguments as in how you finetune the llama. What could be the reason of this?
Firstly, alpaca data is not intended for conversation but rather for instructional purposes. You should consider leveraging other data specifically designed for conversation. It appears that Bloomz-7b1-mt is a model fine-tuned on other tasks, so it's hard to align using instruction data.
no, I did convert the data to the same format as in this repo. And when I use bloomz-7b1-mton alpaca repo, it works just fine, however, I want to make it conversational so I use this repo. I think the problem is the training hyperparameter because I find this repo's hyperparameter is different from this repo. What do you think?
and for some reason, the adapter_model.bin file size is so small (<1MB) even though I trained for 8 hours. it's so weird