Our server is receiving a large media file in the request body and we want to send it to a third party service. This service expects a multi-part form request.
We don't want to collect the file into server's memory. That's why we want to stream this multi-part data to the third party service.
For illustartion, we make the request to httpbin.org/post or some local server that waits for a seconds and returns a hard-coded response.
Observing the memory consumption on the Xcode debug navigator, we see very large spikes in memory although we create an AsyncStream for the third-party API request body.
ScrShot.2024-10-19.at.10.57.17.mp4
Spike is around 00:06
The server starts with 10MB. With incoming requests it spikes to 2-4GB. After the requests are finished the new baseline becomes 70MB.
Send thirty 100MB files concurrently.
For example
- download
100MB.binfile from https://ash-speed.hetzner.com to the~/Downloadsfolder - run the script
chmod +x ./scripts/curl_concurrent.sh
./scripts/curl_concurrent.sh