[QST] Why isn't there an API to decompress data as one single chunk ? (ZSTD)
shreer1998 opened this issue · 6 comments
I could see that the current API to decompress the data compressed by ZSTD is nvcompBatchedZstdDecompressAsync.
Are there any plans to provide an API to decompress data as one single chunk (entire data is one single batch) ?
If yes, when can we expect it and if it's no, plans what's reason behind it ?
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
@shreer1998 do you have a use case for the single chunk decompression on the GPU? For zstd or other formats. Please let us know. So far we’ve been focusing on supporting Apache Parquet/ORC type of files that usually have many chunks, so we’ve been prioritizing the batched interface.
@nsakharnykh Yes, I have a use case with single chunk ZSTD decompression in GPU. Basically in our use case the ZSTD compression happens on the CPU and we want perform decompression on the GPU. It will be of great help if you provide single ZSTD decompression on GPU. If incase you can do that, when can we possibly expect it to be available ?
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
Hi @shreer1998,
Can you help us understand why you can't split the data into many smaller chunks on the CPU? I.e. through multiple calls to the CPU compressor (possibly even with many threads?)
Note, there is no minimum batch size in the nvcompBatchedZstdDecompressAsync API, so you can call it with 1 chunk (up to 2 GB in size).
Unfortunately, the performance will be slow with such a single chunk.
Hello, @eschmidt-nvidia.
Thanks for the response. I didn't have the option to tweak the CPU zstd compression into batched compression as the CPU part was frozen for some reason. Given the fact that chunk sizes can be up to 2GB, it solves my issue for now. I thought it was restricted to 64KB.