seatgeek/thefuzz

Inaccurate search in a large pandas table

DimIsaev opened this issue · 1 comments

DimIsaev commented

How to optimize the search for a similar row in a large data table ~ 1million records

pre-vectorization of each line?

%%time
process.extractOne(text, df['title'])

image

maxbachmann commented

Probably the easiest way to improve the performance would be to use RapidFuzz instead of thefuzz, which uses a significantly faster implementation of the same algorithm. Since it is mostly API compatible, it only requires you to change the import from from thefuzz import process to from rapidfuzz import process.

  • 👍2