rii.py Ks assert not consistent with nanopq's Ks assert
ashleyabraham opened this issue · 1 comments
In rii.py line 35 there is an assert for Ks to check if it is less than 256 (which should be 2**32), but in nanopq pq.py line 37 similar assert checks to see if Ks is less than 2**32 (which is correct IMHO). So when passing in a nanopq to rii it throws AssertionError if the Ks is larger than 256. I am using a very large dataset 6.7 million rows. So my Ks is set 2**12 = 4096 and nanopq works as expected, but rii throws the AssertionError.
This behavior is expected.
I designed nanopq
with a maximum flexibility. So you can try 256<Ks
.
In rii
, each vector must be encoded to a set of 8-bit unsigned chars, meaning that Ks=256
. See flattened_codes
: https://github.com/matsui528/rii/blob/main/src/rii.h#L81
I would suggest using higher M
instead of higher Ks
in your use case. For example, let's say your vector is 120-dim and you encode it by M=4
with Ks=2**12
. resulted in 4 * 12 = 48 bit/code
. Then you can encode it by M=6
with Ks=2**8
, resulted in 6 * 8 = 48 bit/code
.