umccr/htsget-rs

Implement proper AsyncSeek+AsyncReader

brainstorm opened this issue · 0 comments

We should really fix this issue in order to not download excessive bytes for the get_content() method. Potentially have those changes more upstream so that all storage backends can benefit from the improvement:

async fn get_content<K: AsRef<str> + Send>(&self, key: K, options: GetOptions) -> Result<Bytes> {
// It would be nice to use a ready-made type with a ByteStream that implements AsyncRead + AsyncSeek
// in order to avoid reading the whole byte buffer into memory. A custom type could be made similar to
// https://users.rust-lang.org/t/what-to-pin-when-implementing-asyncread/63019/2 which could be based off
// StreamReader.

Otherwise the side effects are unnecessary egress charges for cloud and prohibitively slow requests/responses.