Post hoc testing
Closed this issue · 2 comments
Hello again, Todd. Happy New Year!
Searching through your forum, I think I have found the answer to my question, but just wanted to be sure. Sorry if this creates extra work/spam
If I find an interaction over a specific region when running a 3-way ANOVA (say AxC in the attached image at 20 - 25 % of the time series), can I ONLY report post hoc differences either within, or overlapping this region? A previous message in the forum states the following, so I'm assuming the region in my initial 3way test constitutes the region of interest (ROI) , soI ONLY report specific post hoc differences in/overlapping the ROI
> Here are two thoughts:
* If an ROI is defined, and a given cluster lies partially inside the ROI, and also extends beyond the ROI, that cluster is usually considered to be part of the ROI, by convention
* If an ROI is defined, and a given cluster lies completely outside of the ROI, it must be ignored because it is irrelevant to the null hypothesis_
And so would this mean that, whereas plotting 95 % CIs for my two bouts (B, above - within measures) shows a lack of overlap at both 35 - 50 % AND 55 - 80 % of the time series, I only report that SPM identified a difference at 60 - 75 %?
If I find an interaction over a specific region when running a 3-way ANOVA (say AxC in the attached image at 20 - 25 % of the time series), can I ONLY report post hoc differences either within, or overlapping this region?
For post hoc analysis You can analyze and report anything provided it does not contradict the main ANOVA results.
A previous message in the forum states the following, so I'm assuming the region in my initial 3way test constitutes the region of interest (ROI) , soI ONLY report specific post hoc differences in/overlapping the ROI
Yes, you can use an ROI in this manner. ROIs often pertain to a priori hypotheses, but they can indeed be used to constrain post hoc analyses as you suggest. However, one problem with this approach is that it artificially increases power (i.e., lowers the critical threshold); the smaller the ROI the greater the power, so you will be more and more likely to reach the critical threshold as you shrink the ROI. Regardless, the best general approach is (as above): ensure that your reported post hoc analyses do not contradict the main ANOVA results.
And so would this mean that, whereas plotting 95 % CIs for my two bouts (B, above - within measures) shows a lack of overlap at both 35 - 50 % AND 55 - 80 % of the time series, I only report that SPM identified a difference at 60 - 75 %?
Yes, this sounds fine. However, note that a lack of overlap in CIs does not imply significant difference unless the CIs are calculated in the appropriate design-dependent manner. Please refer to "Appendix F: Confidence interval design dependence" in the following paper:
https://www.researchgate.net/publication/273576725
In your case you'd need to calculate CIs with a three-way interaction model. Note that this type of CI calculation is not directly supported in spm1d.
Great stuff. I think that answers my question
- If the 3 way ANOVA shows a main-effect for bout ONLY at 60 - 75 % then I only report post-hoc results overlapping with this region
- Reporting post-hoc results at 35 - 50 %, for example, contradicts the results of the initial ANOVA
If that's correct, then consider the problem solved! Thank you very much