[QST] Decompress one input buffer into multiple output buffers

[iomartin]

I'm using the C batched API in a program I'm writing. I have some input that I split into multiple batches, compress, then copy the different compressed chunks back to a contiguous host buffer. This is working as expected.

The problem is when I later try to decompress this data. I no longer know when each of the compressed chunks starts and ends, as they are part of a contiguous buffer. Also, I don't know how many buffers I'll need to store the decompressed data.

For example: if I have to compress 128k and my chunk_size is 64k, this would give me batch_size = 2. Suppose the first chunk compresses to 8k and the second to 10k. Then I have an 18k contiguous compressed buffer. When I try to decompress it, I don't know how many chunks I'll need to store the output.

Unfortunately, it's not possible for my application to keep any state so that it knows how to split the compressed data. One possible solution is that I know that I will never have to compress anything larger than 1M, so I could always use just one batch. But I'm afraid that this could hurt performance.

[akshaysubr]

This is exactly what the high lever interfaces (HLIF) are for. It essentially does the input chunking, compacting compressed buffers and storing additional metadata in the compressed stream itself. The HLIF decompressor can then read this metadata and then know how to decompress that compacted stream to a contiguous decompressed buffer.

Here's a link to the high level quickstart doc:
https://github.com/NVIDIA/nvcomp/blob/main/doc/highlevel_cpp_quickstart.md

[iomartin]

Thanks, I switched my code to use the HLIF and it is working as expected :)