quantizer
skaae opened this issue · 2 comments
The quantizer doesn't work with q_levels=512. I'm not sure why but, to me it seems that it should. Maybe the epsilon is too small? For q_levels=512 you get quantized values that are 512 not 511.
I find this variant easier to read and it works :
samples = (q_levels-1)*samples + 0.1
samples = samples.long()
Yeah, the epsilon is too small which causes 512 - EPSILON to be the same as 512 in 32 bit floats. Changing EPSILON to, say, 1e-2 should fix the problem.
Your variant skews the quantized data distribution at the edges - very few samples that should get the highest quantization level will get it (only 10% assuming uniform distribution). That's why the epsilon should be small.
This probably doesn't matter that much anyway, but I'd like to avoid such arbitrary choices whenever possible.