Lead: Kelly Garner; Co-authors: Christopher Nolan, Abbey Nydam, Zoie Nott, Howard Bowman, & Paul Dux
CC BY-SA
An accurate quantification of effect sizes for an experimental manipulation has the power to motivate theory, and to reduce misinvestment in scientific resources by informing power calculations during study planning. Such a quantification could theoretically be achieved by a meta-analysis. However a combination of publication bias and small sample sizes (~N = 25) hampers certainty that such an analysis would yield a non-erroneous estimate. We sought to determine the extent to which each of these caveats may produce error in effect size estimates for 4 commonly used paradigms assessing executive function and implicit learning (attentional blink (AB), multitasking (MT), contextual cueing (CC), serial response task (SRT)). We combined a large dataset with a bootstrapping approach to simulate 1000 experiments across a range of N (13-313). Beyond quantifying the effect size and statistical power that can be anticipated for each study design, we demonstrate that experiments with lower values of N lead to problematic information loss, potentially biasing power calculations. Furthermore, we show that for the CC and SRT, a meta-analysis of experiments with lower N is unlikely to ever converge on the true effect size, owing to underspecification of the mapping between theory and statistical model. We conclude with practical recommendations for researchers and demonstrate how our simulation approach can yield theoretical insights that are not readily achieved by other methods; such as identifying when qualitative individual differences exist in response to an experimental manipulation.
Data:
Preprint:
Data taken from the Executive Function and Implicit Learning (EFIL) project run by Abbey Nydam and Paul Dux for the Team Honours Thesis in 2019. Uses data from 313 1st-year psychology participants on a battery of cognitive tasks including:
- Attentional Blink
- Multitasking paradigm (also referred to as single and dual task [SD])
- Contextual Cuing
- Serial Reaction Time Task
Running of code in this repository requires the 'data' folder from [insert link] be downloaded and stored locally in the top level directory of this repo. Code to run the analysis is in 'R' (see that folder for further info). To generate the manuscript document, knit the .Rmd file in 'doc'.
The code for the appendices of the paper can also be found in 'doc'