Make `skip_special_tokens` a generation parameter.
cyanic-selkie opened this issue · 2 comments
cyanic-selkie commented
Hi,
I have a few models that return structured output by utilizing special tokens as delimiters. As of now, vLLM always skips special tokens during decoding. Would it be possible to add skip_special_tokens
as a generation parameter?
TGI sort of supports this by giving you the option to return individual tokens with their IDs and a boolean indicating whether they are special or not.
bingfengyiren commented
I had the same problem as you.
WoosukKwon commented
@cyanic-selkie and @bingfengyiren We've just merged #1186, which adds skip_special_tokens
to SamplingParams
. This feature will be included in a new release which will be released very soon.