mosuka/cockatrice

If i want to change the weighting model to LM Model

Closed this issue · 4 comments

BruceLee66 commented

How could i do to use the LM model as weighting model?

mosuka commented

Is the LM Model mean "Linear Model"?
Cockatrice just uses Whoosh. Currently focuses on server implementation for using Whoosh on distributed environment.
Please refer following document: https://whoosh.readthedocs.io/en/latest/api/scoring.html
I hope this will help. :)

BruceLee66 commented

I mean the language model,and i have read this many times.But i did not find any solutions about it.My code is liking this

#自定义LM部分
def lm(tf,dl,cf,fl,u):
    #tf代表词在该文档中出现次数
    #dl代表文档长度
    #cf代表词在文档集合中出现次数
    #fl代表所有文档的长度之和
    #u代表参数
    return (tf/dl)*dl/(dl+u)+(1-dl/(dl+u))*(cf/fl)

class LM(WeightingModel):
    def scorer(self, searcher, fieldname, text, u=190,qf=1):
        if not searcher.schema[fieldname].scorable:
            return WeightScorer.for_(searcher, fieldname, text)
        return LMScorer(searcher, fieldname, text, u=u,qf=qf)

class LMScorer(WeightLengthScorer):
    def __init__(self, searcher, fieldname, text, u,qf=1):
        parent = searcher.get_parent()  # Returns self if no parent
        self.cf = parent.frequence_all(fieldname, text)
        self.fl = parent.field_length(fieldname)
        self.u=u
        self.setup(searcher, fieldname, text)

    def supports_block_quality(self):
        return True

    def _score(self, weight,length):
        return lm(weight,length,self.cf,self.fl,self.u)

    def max_quality(self):
        return self._maxquality

    def block_quality(self, matcher):
        return matcher.block_max_weight() * self.idf
mosuka commented

Thank you for explain.
I will change the API so that you can run own weighting model.
Please give me some time.