BM25的Vectorizer
基于sciket-learn的TfidfVectorizer，继承修改得到的Bm25Vectorizer 使用方法和sciket-learn的方法完全相同
from BM25Vectorizer import Bm25Vectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
if __name__ == '__main__':
# format_weibo(word=False)
# format_xiaohuangji_corpus(word=True)
bm_vec = Bm25Vectorizer()
tf_vec = TfidfVectorizer()
# 1. 原始数据
data = [
'hello world',
'oh hello there',
'Play it',
'Play it again Sam,24343,123',
]
# 2. 原始数据向量化
bm_vec.fit(data)
tf_vec.fit(data)
features_vec_bm = bm_vec.transform(data)
features_vec_tf = tf_vec.transform(data)
print("Bm25 result:",features_vec_bm.toarray())
print("*"*100)
print("Tfidf result:",features_vec_tf.toarray())
输出如下：
Bm25 result: [[0. 0. 0. 0.47878333 0. 0.
0. 0. 0. 0.8779331 ]
[0. 0. 0. 0.35073401 0. 0.66218791
0. 0. 0.66218791 0. ]
[0. 0. 0. 0. 0.70710678 0.
0.70710678 0. 0. 0. ]
[0.47038081 0.47038081 0.47038081 0. 0.23975776 0.
0.23975776 0.47038081 0. 0. ]]
*****************************************************
Tfidf result: [[0. 0. 0. 0.6191303 0. 0.
0. 0. 0. 0.78528828]
[0. 0. 0. 0.48693426 0. 0.61761437
0. 0. 0.61761437 0. ]
[0. 0. 0. 0. 0.70710678 0.
0.70710678 0. 0. 0. ]
[0.43671931 0.43671931 0.43671931 0. 0.34431452 0.
0.34431452 0.43671931 0. 0. ]]
Process finished with exit code 0