visus compress-dataset seems to not work

[hoang-dt]

The compress-dataset command seems broken. When tried on a dataset, the data is either not compressed, or is different from data compressed using the "import export" commands. As a work around, I'm attaching a Python script that will do the compression using "import export" commands.

idx-compress.zip

[scrgiorgio]

Hi Duong, can you share exactly what is not working? I need the command line in order to debug. Usually it's better if you can replicate the issue with a toy example (like cat in dataset directory).

…

On Wed, Oct 2, 2019 at 11:50 PM Duong Hoang ***@***.***> wrote: The compress-dataset command seems broken. When tried on a dataset, the data is either not compressed, or is different from data compressed using the "import export" commands. As a work around, I'm attaching a Python script that will do the compression using "import export" commands. idx-compress.zip <https://github.com/sci-visus/OpenVisus/files/3683494/idx-compress.zip> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#86?email_source=notifications&email_token=AAJTZP32WCUI66NPRHXJAM3QMUJT7A5CNFSM4I4333N2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HPIA33A>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJTZP4DJ5JNZJOI7NWD6P3QMUJT7ANCNFSM4I4333NQ> .

[scrgiorgio]

I think I fixed it.

fc067c3

[hoang-dt]

Sorry Giorgio I didn't get a chance to produce a reproducible test yet. Will get around doing that today. I see that you changed

String compression = query->field.default_compression;

to

String compression = block_header.getCompression();

The problem with the second line is that, if the compression has parameters, during decompression I don't get the parameters passed to the Decoder, only the name of the compression itself (e.g. zip, zfp etc). I need to get zfp-16 not just zfp to be able to decode correctly. What is the problem with the first approach?

[scrgiorgio]

ok that was the offending line for compressDataset. We first write the data uncompressed: each block actually store (in the header) the information that the block is stored as not compressed. When later i'm trying to compress it (i.e. compressDataset), first i read the block, and I realize from the header it's still uncompressed; then I compress it using the new compression scheme. It could even be that different blocks are compressed with different compression algorithms (depending on their numbers, read coarse vs fine blocks), so we should really stick to use the header as a source of information if the block was compressed or not. We should not overrule this using IDX... Does that make sense?

…

On Tue, Oct 8, 2019 at 5:54 PM Duong Hoang ***@***.***> wrote: Sorry Giorgio I didn't get a chance to produce a reproducible test yet. Will get around doing that today. I see that you changed String compression = query->field.default_compression; to String compression = block_header.getCompression(); The problem with the second line is that, if the compression has parameters, during decompression I don't get the parameters passed to the Decoder, only the name of the compression itself (e.g. zip, zfp etc). I need to get zfp-16 not just zfp to be able to decode correctly. What is the problem with the first approach? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#86?email_source=notifications&email_token=AAJTZP23OAV3IAT3RLAJABLQNSULVA5CNFSM4I4333N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUVQFQ#issuecomment-539580438>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJTZPYAQWN6BNM4ZB5J2KTQNSULVANCNFSM4I4333NQ> .

[spetruzza]

Note: the python script appends the new "default_compression" to the ones existing, so if you have a dataset already compressed with zip the new compression will be appended. Visus probably reads the first entry of default_compression and decide to compress with that one.
In the attached script I am using a regular expression to remove any existing "default_compression" from the idx before doing the export.
Another issue in the script is that it requires the output to be in a different folder, but that should be another easy fix to be done, if necessary.

compress.py.zip

[hoang-dt]

Hi @scrgiorgio . So for now I think we can leave things the way they have been (compression info read from block header instead of from .idx). For zfp I can decompress without the parameter (number of bit planes encoded), so it's fine. In the future though, maybe we have to think more on how to store compression parameters. I made a change here to make this work with zfp, please consider merging it #88

@spetruzza With this PR the decompression for zfp should work fine with latest visus commits.