Experimental design and the GLMM

A while back, I announced on twitter that I wanted to put together a book showing how to use strategies from the generalized linear mixed model (GLMM) to analyze experimental data. This is the public-facing home for that project.

Welcome.

I should unpack a couple terms. I’m using GLMM in an expansive way to include not only single- and multilevel models using familiar likelihoods like the Gaussian and binomial, but I’m also thinking in terms of distributional models where all parameters in the likelihood function can have a linear model and of distinctly non-linear model types including splines. By experimental data, I mean fully randomized true experiments (between persons, within them, and both) and also quasi-experimental variants, where some of that control is lost for various pragmatic reasons. I plan to aim this textbook at students and researchers within the social sciences. Given my background in clinical psychology, the content will bend in that direction. However, I am keen to bring in examples from other areas within psychology and perhaps other social sciences, as well.

Books of this kind often use examples from the scientific literature, and this will too. However, I intend to go further and want to provide students with data sets from or based on the examples. If you’re going to learn how to analyze data from a randomized controlled trial, then why not practice with data resembling one of those trials? Which brings me to my main point:

You can help me write this book!

The primary purpose of this repository is to collect suggestions for studies with data you would want to learn how to analyze. I am particularly eager to include studies whose authors used open science practices, such as open data. Yet since these practices aren’t widespread in the social sciences, studies using older closed practices are also of interest.

Some of the research designs I’d like to include:

posttest-only control group designs
pretest-posttest control group designs
Solomon four-group designs (possibly the three-group variant, too)
factorial designs
crossover designs
control-group designs with many measurement occasions
post-test only designs with a large number of conditions
quasi-experiment variants of all of the above
single-group posttest-only designs
single-group pretest-posttest designs
multiple baseline designs (primarily across persons, but also across settings or behaviors)
AB designs
ABA designs
BAB designs
ABAB designs (ABABAB, ...)
multiple-treatment designs (ABAC, ABCBCA, ...)
changing-criterion designs

A collaborator and I are in the early-stages of a study that will use an encouragement design and will leverage the instrumental variable approach in the analytic strategy. Kristoffer Magnusson has written interesting things on the perils of ignoring therapist effects. Which is all to say, I’m open to including studies that range from methodically simple to intimidating.

Some of the data types I’d like to explore fitting:

approximately-continuous data with the Gaussian and Student-t likelihoods
bounded count data with the binomial likelihood
unbounded counts with the Poisson and negative-binomial likelihoods
non-negative continuous data with the lognormal, exponential, gamma, or Weibull likelihoods
continuous data with clear lower and upper limits with the zero-one-inflated beta likelihood
nominal data with the softmax approach
ordinal data with the cumulative probit

Just like with real-world data, the examples will contend with complications like

missing data
multiple sources of nesting
unequal variances (by group, persons, or both)
zero inflation

Content areas I’d love to get study suggestions from:

behavior-therapy classics
cognitive/developmental interventions
computerized reaction-time tasks (e.g., Stroop)
COVID-19
data vis
education interventions (e.g., DARE)
HIV/AIDS trials
large-scale replications
marriage and other relationships
meta-science
multi-site trials
non-human animal research (dogs, rats, chicks...)
PTSD trials
sex research
substance use trials

Other interesting topic areas are okay, too. For any of these topic areas, the studies could be gold-star examples of their method, or they could have substantial flaws. They could have large effect sizes, or they could show little difference among conditions. They could be citation classics, or something you just published last month.

Other things the book will contain:

data sets saved as external files for all analyses in the book
code for all analyses and figures in the book
discussions on topics like
- effect sizes,
- simulation-based power analyses,
- common pitfalls like Simpson’s paradox, the ecological fallacy, and the table 1 fallacy
discussions on priors, such as
- weakly-informative priors,
- weakly-regularizing reference priors,
- priors based on previous research,
- prior-predictive simulations

I'm leaning towards including light discussions of the potential-outcomes framework. I'm also thinking about using DAGs to (we'll see).

Things this book will avoid or minimize:

Bayes factors
finite mixture models (e.g., growth mixture models)
- I will, however, include mixtures in the form of zero-inflated likelihoods and so on.
introductory programming
introductory statistics
latent variables
NHST
the type-I/type-II error framework