feat: max-tokens for completion
kryptoklob opened this issue · 11 comments
should add a setting to constrain the max # of tokens in completions
should be pretty reasonable to PR if you wanna give it a try!
just need to add a field to the Settings type: https://github.com/transmissions11/flux/blob/af772d947237d53d0b720a75bbd2c49f83013db0/src/utils/types.ts#L18
add a LabeledSlider like so in the SettingsModal: https://github.com/transmissions11/flux/blob/af772d947237d53d0b720a75bbd2c49f83013db0/src/components/modals/SettingsModal.tsx#L90-L98
set a default value for it in the DEFAULT_SETTINGS object: https://github.com/transmissions11/flux/blob/af772d947237d53d0b720a75bbd2c49f83013db0/src/utils/constants.ts#L7
and pass it to the OpenAI call in App.tsx: https://github.com/transmissions11/flux/blob/af772d947237d53d0b720a75bbd2c49f83013db0/src/components/App.tsx#L323-L330
only thing idk how to handle is how to represent the "infinity" option on a slider 🤔
also hmmm except openai-streams doesn't seem to support setting max_tokens for the chat models for some reason: https://github.com/gptlabs/openai-streams/blob/4eb53a2509846f5f9e7bcb6180dbb98aa83fdcbf/src/lib/pinned.ts#L25
guessing this is just an oversight @ctjlewis, or is there a good reason i'm missing?
I think it would be nice to separate settings to settings that are relevant to generating GPT responses and settings that are relevant to completing new words.
With chat we can't work with max_tokens but when it comes to completing next words we can.
I think we can take also inspiration from TypingMind in this (and many more).
I think it would be nice to separate settings to settings that are relevant to generating GPT responses and settings that are relevant to completing new words.
100%. if you want to work on this that'd be super appreciated, though i'd recommend sharing a sketch of whatever you propose implementing before spending the time writing the code cuz i may be a little picky here
also hmmm except
openai-streamsdoesn't seem to support settingmax_tokensfor the chat models for some reason: https://github.com/gptlabs/openai-streams/blob/4eb53a2509846f5f9e7bcb6180dbb98aa83fdcbf/src/lib/pinned.ts#L25guessing this is just an oversight @ctjlewis, or is there a good reason i'm missing?
That was just copy-pasted from the type libs. They may have updated it since then. I don't exactly recall why that was easier - I think it was a way to keep the openai libs out as a dependency. We can update them by hand.
I think it would be nice to separate settings to settings that are relevant to generating GPT responses and settings that are relevant to completing new words.
100%. if you want to work on this that'd be super appreciated, though i'd recommend sharing a sketch of whatever you propose implementing before spending the time writing the code cuz i may be a little picky here
Yeah, will do some basic sketch in Figma.
How I would approach this would be like keep the modal but just add tabs (from Chakra). There will be 3 tabs - Global, GPT and Completion. In global there will be everything that is there now except model and temperature. In GPT and Completion there will be model and temperature plus in Completion there will be max tokens.
Two things I am not sure about:
- Is there a need to tell user what is the difference between GPT and Completion and if so how?
- Do we want to add more models for completion, now there is just text-davinci-003 but Imo not bad to add like 1-2 more (text-ada-001 or text-curie-001)?
Is there a need to tell user what is the difference between GPT and Completion and if so how?
Maybe lets change to "Chat" and "Completion" to make it more clear. Also can just put a little paragraph before the inputs once you select the tab.
Do we want to add more models for completion, now there is just text-davinci-003 but Imo not bad to add like 1-2 more (text-ada-001 or text-curie-001)?
yes lets add all of them if possible. including raw davinci, curie, etc
Maybe lets change to "Chat" and "Completion" to make it more clear. Also can just put a little paragraph before the inputs once you select the tab.
Yeah that makes much more sense.
Just want to let I will have time to work on this on weekend, if someone wants he can pick it up
@transmissions11 Pinned types are updated so max_tokens is supported. In the original API release, OpenAI did not include that I guess (1.1.0).
Just want to let I will have time to work on this on weekend, if someone wants he can pick it up
@AdamSchinzel are you still working on this would like to pick it up.