turtlesoupy/this-word-does-not-exist

Questions regarding a "GPT-3" update

DonaldTsang opened this issue · 7 comments

DonaldTsang commented

Would it be useful to update this project with a better language model, or is that redundant?

xynydev commented

I don't think this would need a better one, you can't retrain GPT-3 either.
But if there would be a need or want for another model I'd suggest https://github.com/EleutherAI/gpt-neo/

DonaldTsang commented

Fairs, but can this system accept GPT-NEO tho

xynydev commented

Fairs, but can this system accept GPT-NEO tho

What do you mean by that?

Also apparently eleuther has a 6B model too https://github.com/kingoflolz/mesh-transformer-jax/.

BrandonKMLee commented

GPT-NEO and other FOSS model exists, but how does one "mill" them into a definition generator?

  • 👀1
turtlesoupy commented

You'll almost certainly need to fine tune them on a dataset... my dataset was a normalized form of the Apple dictionaries shipped on mac -- the code is here: https://github.com/turtlesoupy/this-word-does-not-exist/blob/master/title_maker_pro/dictionary_definition.py

Basically I fed a definition at a time with sequences like <word_def>word<definition>an interesting word<example>the word is right<end>

turtlesoupy commented

Going to close for now!

BrandonKMLee commented

@turtlesoupy thanks for the conversation! The potential is there to revamp this with multiple dictionaries.