[QST] How to compress a folder that has mulitple files using nvcomp and save the output as a compressed file. Also how to decompress that compressed file and save it using nvcomp ?
selvaraj-sembulingam opened this issue · 4 comments
How to compress a folder that has mulitple files using nvcomp and save the output as a compressed file. Also how to decompress that compressed file and save it using nvcomp ?
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
The low-level benchmark programs, are primarily implemented in https://github.com/NVIDIA/nvcomp/blob/main/benchmarks/benchmark_template_chunked.cuh , with customizations for each compressor in various .cu files in the same directory, for example https://github.com/NVIDIA/nvcomp/blob/main/benchmarks/benchmark_cascaded_chunked.cu for the cascaded compressor. The shared code gives an example of how to load multiple files and compress them collectively. For decompression, the benchmark code gives an example of how to decompress data that's already in GPU memory and then transfer it back to main memory.
In between those two, you could send the compressed data back to main memory, then save the data out to a file on disk. However, it's important to note that the original file names and sizes aren't included in that compressed data, so you'll need to record them, too. Similar thing on the decompression side: you can load the data from disk into main memory, send it to GPU memory for decompression, and send it back to main memory for saving out to multiple files.
It may be worth being mindful of the overhead of the data transfers, though, because it can be significant compared to the time to compress in some cases. nvcomp usually provides the biggest benefit for applications where they're compressing data that's already in GPU memory, and then later decompressing the data in GPU memory for use on the GPU. If you can reduce data transfer overhead, that can make a big difference. There is a GPU Direct Storage example for systems that support it, though many systems don't.
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.