Sparse Vector

Question

Sparse Vector

Closed this issue 3 years ago · 4 comments

Just wondering,
if pqkmeans can be optimized for sparse vectors
, csr format, instead of dense ones

thank you

Answer 1 · 2021-12-27T01:43:42.000Z

No. You need to convert sparse vectors to dense ones before applying pqkmeans.

My general advice is to project the sparse data to a lower (denser) space, e.g., from D=10K to D=100. That way, PQ will work better.

Answer 2 · 2021-12-27T07:46:31.000Z

Thanks. Is there a way to project in a fast way on low dense ? (umap exists but, not so fast). Yes, kmeans require gaussian vectors…..

…

On Dec 27, 2021, at 10:43, Yusuke Matsui ***@***.***> wrote: No. You need to convert sparse vectors to dense ones before applying pqkmeans. My general advice is to project the sparse data to a lower (denser) space, e.g., from D=10K to D=100. That way, PQ will work better. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

Answer 3 · 2021-12-29T15:41:04.000Z

You can try multiplying by a random matrix. Multiplication of a matrix and a sparse vector would be generally fast.

Answer 4 · 2021-12-29T15:48:13.000Z

Thanks. Interesting way to give “gaussianity” to a vector…. and should preserve distance.

…

On Dec 30, 2021, at 0:41, Yusuke Matsui ***@***.***> wrote: You can try multiplying by a random matrix. Multiplication of a matrix and a sparse vector would be generally fast. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.