matsui528/rii

rii.py Ks assert not consistent with nanopq's Ks assert

ashleyabraham opened this issue · 1 comments

In rii.py line 35 there is an assert for Ks to check if it is less than 256 (which should be 2**32), but in nanopq pq.py line 37 similar assert checks to see if Ks is less than 2**32 (which is correct IMHO). So when passing in a nanopq to rii it throws AssertionError if the Ks is larger than 256. I am using a very large dataset 6.7 million rows. So my Ks is set 2**12 = 4096 and nanopq works as expected, but rii throws the AssertionError.

This behavior is expected.

I designed nanopq with a maximum flexibility. So you can try 256<Ks.

In rii, each vector must be encoded to a set of 8-bit unsigned chars, meaning that Ks=256. See flattened_codes: https://github.com/matsui528/rii/blob/main/src/rii.h#L81

I would suggest using higher M instead of higher Ks in your use case. For example, let's say your vector is 120-dim and you encode it by M=4 with Ks=2**12. resulted in 4 * 12 = 48 bit/code. Then you can encode it by M=6 with Ks=2**8, resulted in 6 * 8 = 48 bit/code.