Streaming response handling
Closed this issue · 10 comments
Hi!
A handful of endpoints support response streaming, but the hackney/finch stuff is so deeply nested that it's not fully clear how to actually use those.
Are there any examples or references what to do to properly handle streaming HTTP responses?
@philss Any chance you have some input on this? I unfortunately do not use Elixir in any production code so am a bit limited on being able to assist on this question.
I managed to get it working, it's using normal streaming functionality in clients like finch
@dvcrn ❤️ Any chance you can make a PR? ❤️
I'm not sure what the right format for a PR like this would be. There are a lot of aws endpoints that support streaming of data, and multiple clients are supported. For my stuff I only used finch and the streaming support that's baked into that, but it would be different for hackney 🤔
Maybe a note in the readme?
That would work :-) Anything to avoid the dreaded DenverCoder9 scenario and would help anyone else along :-)
If there's anything I can pick up from your README PR and integrate it natively within aws-elixir we can take it as a secondary step 👍
@dvcrn Can you give me a hint on how you did it?
I enabled the finch http client:
%AWS.Client{
access_key_id: key.access_key_id,
secret_access_key: key.secret_access_key,
region: "eu-central-1",
http_client: {AWS.HTTPClient.Finch, [finch_name: MyApp.Finch]}
}
But when I send a request with this (I am using the PollyAPI) I dont get a stream response like this:
{:ok,
%{
"Body" => <<73, 68, 51, 4, 0, 0, 0, 0, 0, 35, 84, 83, 83, 69, 0, 0, 0, 15, 0,
0, 3, 76, 97, 118, 102, 53, 56, 46, 55, 54, 46, 49, 48, 48, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 255, 243, ...>>
},
%{
body: <<73, 68, 51, 4, 0, 0, 0, 0, 0, 35, 84, 83, 83, 69, 0, 0, 0, 15, 0, 0,
3, 76, 97, 118, 102, 53, 56, 46, 55, 54, 46, 49, 48, 48, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 255, ...>>,
headers: [
{"x-amzn-requestid", "c8c11524-fb8d-40b6-aedb-7ff555ab2b6b"},
{"x-amzn-requestcharacters", "5"},
{"content-type", "audio/mpeg"},
{"transfer-encoding", "chunked"},
{"date", "Tue, 05 Mar 2024 11:31:39 GMT"}
],
status_code: 200
}}
That is not a streaming response (am I wrong?) also the transfer encoding is chunked.
Can you give me a hint on what I have to do differently?
@RudolfVonKrugstein Possibly this: https://github.com/philss/aws-s3-stream-download-poc helps you along the way until @dvcrn (hopefully) gets back to you with more details.
Adding to v1 to try and improve some documentation around this type of operation
@RudolfVonKrugstein @onno-vos-dev sorry about that, I didn't receive email notifications for some reason...
Here's my hacked together implementation: https://github.com/dvcrn/chatgpt-ui/blob/main/lib/chatgpt/anthropic.ex#L97-L218
Basically you have to
- Enable streaming support with a new finch client because this repo here doesn't support streaming yet (https://github.com/dvcrn/chatgpt-ui/blob/main/lib/chatgpt/streaming_finch.ex)
- Add a callback handler to :streamfx opt (that's just how I called it, has to match what you use in StreamingFinch)
- Inside the callback function you need to decode the aws streaming response which looks like this: https://docs.aws.amazon.com/transcribe/latest/dg/streaming-setting-up.html#streaming-event-stream
- Prelude tells you how many bytes are for the entire thing and for headers
- Next 4 bytes are CRC
- next n bytes (however many the prelude says there are) are header content, followed by actual payload
- payload size is
x = (total size - header size - 4x4), so we know that after headers, the nextxbytes are payload
Then just decode the json, get the "bytes" key and base64 decode that
decoded_payload =
payload
|> Jason.decode!()
|> Map.get("bytes")
|> Base.decode64!()
|> Jason.decode!()Hope that helps!
Oh has this been added?