Query embed is very slow
joshniemela opened this issue · 0 comments
joshniemela commented
Query embedding is very slow and even gets beaten by python in many cases, this has been found to be due to the padding strategy being fixed which means short queries of a few words will be padded to 512 tokens.
query_embed also uses the embed function which has a lot of overhead due to parallelisation which is not use for a single query, I propose making the query_embed function its own thing.