rtmdrr/testSignificanceNLP

Possible bug

trangham283 opened this issue · 3 comments

trangham283 commented

In lines 78 and 100, shouldn't the sign be flipped? i.e. increment r if delta > 2*orig_delta.
That's according to the Berg-Kirkpatrick et al. (2012) paper, unless I'm misunderstanding something.
Thanks!

allanj commented

BTW, I also found that Berg-Kirkpatrick et al. (2012) mentioned in the footnote,
we can also use

delta (x_i) < 0

besides

delta > 2*orig_delta
texttheater commented

So... this was fixed for Bootstrap, but not for Permutation?

texttheater commented

Someone correct me please, but I think the original version is correct assuming that data_A is the output of the baseline and data_B is the output of the algorithm to be tested. If you swap the two, the sign of the deltas is going to be flipped, so you have to flip the comparator.