nd-ball/py-irt

Isn't the 3pl model wrong?

Closed this issue · 6 comments

muriloime commented

Shouldn't it be dist.Bernoulli(probs=lambdas[items]+(1-lambdas[items]) * p_star), uinstead of dist.Bernoulli(probs=(1-lambdas[items]) * p_star),

EntilZha commented

Hi, thanks for the bug report! Without digging super deep, I think this is probably a notional/naming issue. I believe that this and the 4pl models were developed similarly and mainly for the purpose of adding on an additional parameter to the 2PL representing whether an example was doable at all. I.E., if a particular item is non-answerable, then the parameter value should be zero or one. This is different from 3PL models in the literature, hence the unfortunate naming problem of having 3 parameters despite not being the traditional model (in our case, we expected random guess to be near zero).

So given that, if lambda were to represent the guessing parameter, you are correct, it should be changed. In our original implementation though, lambda was meant to be correlated with feasibility. Even given that, I looked at our 3PL/4PL code and for some weird reason we seem to have (1-lambda) in one place, but (lambda) in another, and they really should be consistent.

I'm guessing that the best long term solution is to come up with some naming or constructor that can make this a bit more explicit, i.e., give me a model which has feasibility, but not discriminability. In any case, we really should have a standard 3PL model where the third parameter is a guessing parameter. We'd welcome a PR for it of course too! Thanks and hopefully that answers your question.

muriloime commented

Thanks for the reply @EntilZha .

Yes, I had saw that too, it is inconsistent with predict function. I also have observed a couple of unexpected things, such as the choice of priors giving wrong results, and redundant parameters in the multidimensional ( I expected to have a*theta - d formulation instead of a*(theta -b) one). More than happy to discuss/help with these points

cheers
Murilo

muriloime commented

(if you can help with a naming convention I can definitely open a PR with this issue in particular)

jplalor commented

Thanks, Murilo!

To stay consistent with the traditional 3PL model, I think we should make the change that you mentioned initially (If you can open a PR for that, that would be great!).

For feasibility, maybe we should rename 4PL to something that more explicitly references feasibility.

@EntilZha I agree about the long-term fix, something that is more explicit is probably ideal.

  • 👍1
jplalor commented

3PL fix is done, so I'll close this issue. We can address the other items via new issues. Thanks @muriloime!