janhq/cortex.llamacpp

feat: [support stream_option for OpenAI API compatible]

Closed this issue · 1 comments

nguyenhoangthuan99 commented

Problem

  • field: stream_options
  • object or null
  • Optional
    Defaults to null. Options for streaming response. Only set this when you set stream: true.
    {"include_usage": boolean
    }
    Optional
    If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value.

reference: https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options

related issue: https://github.com/janhq/internal/issues/160

gabrielle-ong commented

✅ QA: on cortex.llama-cpp v0.1.37-01.11.24
API request: v1/chat/completions

"stream_options": {
        "include_usage": false | true
},

include_usage = false:

Image

include_usage = true: additional chunk before [DONE] showing token usage statistics

Image

Image