0todd0000/spm1d

Order of dataset ttest2

Closed this issue · 12 comments

MauriGiorgio commented

Hi,
I am doing an analysis about the differences between two groups of objects (group A and group B) with the ttest2 (unpaired ttest) in Matlab but I have noticed that it gives me different results depending on the order in which I enter the inputs ( A and B or B and A).
Is it correct? if it was correct, in what order should I enter the inputs?

0todd0000 commented

Swapping the order of the input arguments like this:

spm1 = spm1d.stats.ttest2(yA, yB);
spm2 = spm1d.stats.ttest2(yB, yA);

should yield opposite results in the sense that the t statistics should have opposite signs. Thus the following statement should be true at all time points:

spm1.z == -spm2.z

Other than this no difference is expected. If you are indeed seeing different results please paste code and/or a figure into this issue thread.

MauriGiorgio commented
0todd0000 commented

The results will change for nonparametric permutation tests even if you don't change the input order. Permutation test results are based on randomly permuted observations so the results are not expected to be constant unless you both (a) keep all inputs constant and (b) control the random number generator (RNG) seed / state.

MauriGiorgio commented
0todd0000 commented

Try not changing anything, even the variable order, and just run the nonparametric inference code several times. You will find that the t* value is not constant. This means that non-constant t* is caused by the algorithm itself and not by the variable order.

MauriGiorgio commented
0todd0000 commented

Please try running the following code.

% create dataset:
yA = randn(10,1);
yB = randn(10,1);


% conduct non-parametric test several times:
rng(0)
alpha      = 0.05;
two_tailed = true;
iterations = 1000;
for i = 1:5
    ti = spm1d.stats.nonparam.ttest2(yA, yB).inference(alpha, 'two_tailed', two_tailed, 'iterations', iterations);
    disp( ti.p );
end


You should see that the result ti.p changes each time the test is run. The results should be:



    0.4940

    0.5160

    0.4840

    0.5140

    0.4580


If this does not help to solve your problem, please post code replicating the problem into this thread.

MauriGiorgio commented
0todd0000 commented

As I explained earlier, if I do the non-parametric 1d t-test2 I always get the same t* value.

This is only possible if you are either (a) controlling the random number generator state, or (b) running all iterations. In either case it is not a problem, it is a feature of this non-parametric analysis approach.



If I change the order of A and B the t* value changes.

Again, this may not necessarily be a problem, it may instead simply be a reflection of this non-parametric analysis approach.



I can only answer your questions more specifically if you attach specific code and specific results. Please copy-and-paste your code, and please add numbers to the following statements:

I always get the same t* value.

  • What is the t* value?

If I change the order of A and B the t* value changes.

  • What is the t* value for the AB case?
  • What is the t* value for the BA case?
  • Does the t* value change if you run the AB and/or BA cases multiple times?
MauriGiorgio commented
0todd0000 commented

If I use the order A-B (like in the code above) the t* value is always 2.253

This is because you are using rng to control the random number generator state. Try changing rng(0) to rng(1) or rng(2) or rng(n) where n is any positive integer. The t* value should change.

Identically, if you do the following, you should see that t* changes.

rng(0)
alpha      = 0.05;
two_tailed = true;
iterations = 1000;
snpm1      = spm1d.stats.nonparam.ttest2(A, B);
snpm2      = spm1d.stats.nonparam.ttest2(A, B);
snpm1i     = snpm1.inference(alpha, 'two_tailed', two_tailed, 'iterations', iterations);
snpm2i     = snpm2.inference(alpha, 'two_tailed', two_tailed, 'iterations', iterations);
disp( snpm1i )
disp( snpm2i )
MauriGiorgio commented