incomplete display of outputs
Closed this issue · 7 comments
example 1:
input: Hi
output: ! How can I assist you today?
example 2:
input: One plus one equals to?
output: one equals two.
It is quite obvious that the first few characters were cut off. I tried to inspect the code. I believe the problem lies in ChatGPTAPI from line 102 to 116, but I am yet to figure out the exact problem.
Edit: I just realized it is not just the beginning, in fact within the body of the text there are also missing words/text. A quick sanity check would be asking it the definition of some standard object ('What is a blackhole') and you will be able to see there are missing words. Not sure what is the cause of this issue.
Same issues form here.
Tested with Google Palm API, no such issue was found. Maybe something is wrong with the swfit implementation of OpenAI API?
The problem is in AsyncThrowingStream with cancellation implementation.
Line 102 in sendMessageStream.
Seems it doesn't work properly:
return AsyncThrowingStream { [weak self] in
guard let self else { return nil }
for try await line in result.lines {
try Task.checkCancellation()
if line.hasPrefix("data: "),
let data = line.dropFirst(6).data(using: .utf8),
let response = try? self.jsonDecoder.decode(StreamCompletionResponse.self, from: data),
let text = response.choices.first?.delta.content {
responseText += text
return text
}
}
self.appendToHistoryList(userText: text, responseText: responseText)
return nil
}
The previous implementation (with no cancellation) has no issues.
The problem is in AsyncThrowingStream with cancellation implementation.
Line 102 in sendMessageStream. Seems it doesn't work properly:
return AsyncThrowingStream { [weak self] in guard let self else { return nil } for try await line in result.lines { try Task.checkCancellation() if line.hasPrefix("data: "), let data = line.dropFirst(6).data(using: .utf8), let response = try? self.jsonDecoder.decode(StreamCompletionResponse.self, from: data), let text = response.choices.first?.delta.content { responseText += text return text } } self.appendToHistoryList(userText: text, responseText: responseText) return nil }
The previous implementation (with no cancellation) has no issues.
Hi, do you mind to share the previous version of the code? Simply commenting try Task.checkCancellation() doesn't seem to work. Thanks!
Sure, here is the previous version:
func sendMessageStream(text: String) async throws -> AsyncThrowingStream<String, Error> {
var urlRequest = self.urlRequest
urlRequest.httpBody = try jsonBody(text: text)
let (result, response) = try await urlSession.bytes(for: urlRequest)
guard let httpResponse = response as? HTTPURLResponse else {
throw "Invalid response"
}
guard 200...299 ~= httpResponse.statusCode else {
var errorText = ""
for try await line in result.lines {
errorText += line
}
if let data = errorText.data(using: .utf8), let errorResponse = try? jsonDecoder.decode(ErrorRootResponse.self, from: data).error {
errorText = "\n\(errorResponse.message)"
}
throw "Bad Response: \(httpResponse.statusCode), \(errorText)"
}
return AsyncThrowingStream<String, Error> { continuation in
Task(priority: .userInitiated) { [weak self] in
guard let self else { return }
do {
var responseText = ""
for try await line in result.lines {
if line.hasPrefix("data: "),
let data = line.dropFirst(6).data(using: .utf8),
let response = try? self.jsonDecoder.decode(StreamCompletionResponse.self, from: data),
let text = response.choices.first?.delta.content {
responseText += text
continuation.yield(text)
}
}
self.appendToHistoryList(userText: text, responseText: responseText)
continuation.finish()
} catch {
continuation.finish(throwing: error)
}
}
}
}
Sure, here is the previous version:
func sendMessageStream(text: String) async throws -> AsyncThrowingStream<String, Error> { var urlRequest = self.urlRequest urlRequest.httpBody = try jsonBody(text: text) let (result, response) = try await urlSession.bytes(for: urlRequest) guard let httpResponse = response as? HTTPURLResponse else { throw "Invalid response" } guard 200...299 ~= httpResponse.statusCode else { var errorText = "" for try await line in result.lines { errorText += line } if let data = errorText.data(using: .utf8), let errorResponse = try? jsonDecoder.decode(ErrorRootResponse.self, from: data).error { errorText = "\n\(errorResponse.message)" } throw "Bad Response: \(httpResponse.statusCode), \(errorText)" } return AsyncThrowingStream<String, Error> { continuation in Task(priority: .userInitiated) { [weak self] in guard let self else { return } do { var responseText = "" for try await line in result.lines { if line.hasPrefix("data: "), let data = line.dropFirst(6).data(using: .utf8), let response = try? self.jsonDecoder.decode(StreamCompletionResponse.self, from: data), let text = response.choices.first?.delta.content { responseText += text continuation.yield(text) } } self.appendToHistoryList(userText: text, responseText: responseText) continuation.finish() } catch { continuation.finish(throwing: error) } } } }
Worked like a charm! Thank you so much! Wonder why they decided to change the code here.. Also I tried func sendMessage and from the testing of my use case, the two seem to have indistinguishable performance. I wonder for what use cases the difference in performance of these two will become apparent.
Thank you for your help, I'm going to try this later