NouamaneTazi/bloomz.cpp

document sampling parameters and/or minimal "viable" codegen model ?

laurentperez opened this issue · 1 comments

laurentperez commented

hello. your work is great 👍

I wrapped your binary under my bot/API project https://github.com/laurentperez/ava#what-models-or-apis-does-it-support-

I'm mostly interested in code (python) generation from Bloom as a developer assist. I'm not using it for creative writing. However I'm playing with it for translations to evaluate how the 7b1 model might respond to more complex python prompts.

I infered using the bloomz-1b1 bloomz-3b and bloomz-7b1 models. So far, 7b1 model gives the best results but it's being "too creative".

see example below, the "Me encuentro muy bien/me alegro" were too creative, they did more than a translation :

curl -v -XPOST -H 'Content-Type: application/json' -d '{"msg":"translate \"Hi, how are you?\" in Spanish:"}' http://localhost:8080/hf/bloom

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000  
translate "Hi, how are you?" in Spanish: Me encuentro muy bien. ¿Cómo estas tú? Yo estoy?: me alegro</s> [end of text]

thanks !

NouamaneTazi commented