Make `skip_special_tokens` a generation parameter.

Question

Make `skip_special_tokens` a generation parameter.

cyanic-selkie opened this issue a year ago · 2 comments

Hi,

I have a few models that return structured output by utilizing special tokens as delimiters. As of now, vLLM always skips special tokens during decoding. Would it be possible to add skip_special_tokens as a generation parameter?

TGI sort of supports this by giving you the option to return individual tokens with their IDs and a boolean indicating whether they are special or not.

Answer 1 · 2023-09-04T12:22:52.000Z

I had the same problem as you.

Answer 2 · 2023-09-28T02:26:24.000Z

@cyanic-selkie and @bingfengyiren We've just merged #1186, which adds skip_special_tokens to SamplingParams. This feature will be included in a new release which will be released very soon.