spm1d feature requests
Opened this issue · 15 comments
The following functionality is being developed for future versions of spm1d:
STATISTICS
- P values for non-significant results
- Direction selection ("positive" or "negative") for one-tailed tests
- Post hoc tests (for ANOVA)
- Post hoc tests (for multivariate analyses)
- ANOVA: unbalanced designs (currently only possible for one-way ANOVA)
- ANOVA: SS and MS terms
- Reliability: ICC
- Power analysis (including sample size calculation)
- Effect size, CV, SEM
- Equivalence tests
- Support for circular data (0 to 360 deg)
- False discovery rate procedures
- Multiple regression
- Multivariate analysis: two- and three-way MANOVA
- Multivariate analysis: CCA for multivariate independent variables
- ANOVA: improved non-sphericity corrections
- ANCOVA and MANCOVA
- ANOVA: support for string labels
- Bayesian inference
- Meta-analysis (heterogeneity, etc.)
- MATLAB interface to SPM12, SPM8
GENERAL
- Plotting: improved customization (e.g. line and cloud style specifications for mean / SD plotting)
COMPLETED
The following features (previously listed above) are available in spm1d 0.4 (released 01 October 2016):
- Non-parametric inference
- Confidence intervals
- Normality testing
- Python 3.X compatibility
Please reply to this post or create a new issue if you have any additional feature requests.
Hi Todd,
I was wondering if Bayesian Inference will be part of future versions?
Regards,
Bernard
Hi Bernard,
Yes, absolutely. I'll add it to the list above.
Todd
Hi Todd,
Not sure if you are still considering other functions, totally random but came to my mind when doing a review. If more and more studies are reporting waveforms and if data are made publicly available, then meta-analytical methods of waveforms should be something for the future? And test of things like heterogeneity, meta-regression, sub-group analysis etc would be something that I need is useful?
Many thanks,
Bernard
Hi Bernard, thanks very much for the suggestion! I'll add it to the list above. If enough public 1D data are available meta-analysis would indeed be possible. It might take a few studies to develop a general software tool, but it could definitely useful. Thanks!
Todd
Hi Todd,
Post hoc tests and factorial MANOVA are my main my expectations ; also, I was wondering if other implemented p-value corrections are planned, such as Holm–Bonferroni, Šidák or FDR ?
Hi, thank you for the feedback! Yes, a variety of corrections including the Holm–Bonferroni and Šidák corrections will be considered along with the listed "post hoc tests" as a high priority for development.
FDR procedures are already implemented, but require a bit of clean-up prior to public release; "FDR" wasn't previously included in the list above, but it's now there, thank you for the suggestion!
Factorial MANOVA will indeed also become available, but I'm not yet sure how comprehensive the post hoc procedures will be for MANOVA.
Hi Todd,
I was wondering if a N-way repeated measure ANOVA with muscles as one of the repeated factors was equivalent to a N-1 way MANOVA. If it is not equivalent, do you think it is a good alternative as factorial MANOVA are not available yet in your toolbox?
Thanks
Jason
Hi Jason,
They're similar but not equivalent. RM ANOVA makes more assumptions than MANOVA makes, and there are correspondingly fewer parameters estimated in RM ANOVA, implying that it's more powerful than MANOVA. There's a discussion on this topic here:
https://stats.stackexchange.com/questions/13197/differences-between-manova-and-repeated-measures-anova
and an internet search for "manova vs. repeated measures anova" should yield a number of other relevant discussions.
Two-way MANOVA is in-development for spm1d but will likely not be available until 2018 or later. If you're after N-way MANOVA functionality (N > 1), then spm1d can only really be used for approximate post hoc analysis.
Cheers,
Todd
Thanks!
Hi Todd,
I'd just like to +1 to the ANOVA unbalanced designs. I've posted in stack overflow about nested ANOVA, and having an unbalanced nested ANOVA would be great for the work I do where the observations are inherently unequal. John McDonald talks about the caveats for an unbalanced nested ANOVA here: http://www.biostathandbook.com/nestedanova.html
but also offers an "excel" spreadsheet for the implementation of a "Satterthwaite approximation" for changing the means squares at each level in an unbalanced situation.
cheers,
Kieran
Hi Kieran,
Thank you for voting up this feature. I've moved unbalanced designs up toward the top of the list to prioritize its development for future versions of spm1d.
The Satterthwaite approach is indeed useful, and is relatively straightforward to implement for 0D data analysis. For 1D data analysis the necessary correction is more complex, and I believe that the preferred approach in the SPM literature is Parametric Empirical Bayes (PEB) as described here:
Chapter 12: Random Effects Analysis
I previously tried to implement PEB but encountered a few difficulties, and I'm not sure how long it will take to get a valid, stable version.
If you need to run unbalanced design analysis now please consider the following two options:
-
Use SPM12, which supports unbalanced designs using PEB
-
Use a non-parametric procedure in spm1d.stats.nonparam. The F statistic will not be correct for this analysis, because as you point out the MS terms are generally not correct without a correction. Regardless, in non-parametric permutation analysis it's generally valid to use any test statistic you like, so the F statistic as computed should be fine. If you try to use the code as is a similar unbalanced error will be generated as a warning that the F statistic is incorrect, but it should be possible to modify the source code as needed.
Todd
It is mentiod above that FDR is already implemented but "requires a bit of clean-up prior to public release". Is it ready for usage by now? I'd like to use it for adjusting p-values for scalar field post hoc test after conducting a vector field test.
The Python version of FDR is 95% complete, but the MATLAB version had not yet been started. Both should be ready for release by summer 2019, when a related paper is ready for submission. Alternatively, if you're interested in writing your own code, note that FDR routines don't require very many commands, and the two approaches below can be readily adapted to 1D data:
Chumbley JR, Friston KJ. False discovery rate revisited: FDR and topological inference using Gaussian random fields. Neuroimage. 2009 Jan 1;44(1):62-70.
Genovese CR, Lazar NA, Nichols T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage. 2002 Apr 1;15(4):870-8.
@0todd0000 Thank you for the references! I finally had a chance to read them. Am I right that in order to implement the FDR one would need to extract the p-Values of all clusters (including non-significant ones) and then procede as "usual" in the case of multiple testing in the scalar case?
I don't think FDR can be used with clusters, because p values are undefined when there are no clusters. Instead try assembling uncorrected p values for each continuum node. If the data array shape is (J,Q) where J is sample size and Q is the number of continuum nodes, then there will be Q separate p values, which are produced by separately conducting common 0D tests at each node.