Read on Twitter
Finally got around to reading this and have a few comments. It's a very good introduction to modeling distributions, rather than point estimates, with multilevel modeling approaches. It illustrates nicely how this approach can solve some problems in individual diff research 1/n https://twitter.com/Nate__Haines/status/1297912972707352576
It's also written in a very approachable way and I would recommend it to many folks who are just starting with this type of work. On a more conceptual level though, I do have some issues with how it's framed. 2/n
Two approaches are contrasted as qualitatively different - the more traditional "summary statistic approach" in which we simply calculate and contrast means of different conditions using simple statistical models; and the "generative approach" in which we fit a "model" to the 3/n
whole distribution of responses, and then compare parameter estimates for the fitted distribution. Here's the thing - these are not qualitatively different approaches, as presented in the paper. Both fit a statistical model to the data. Where they differ is in how well the 4/n
model is specified. Case 1) is equivalent to fitting an equal-variance normal model in which we ignore the variance parameter and only focus on contrasting the means. Case 2) explicitly fits a shifted lognormal model in which more distribution parameters are allowed to vary 5/n
Thus the difference here is not that of a "different approach" but of specifying a distribution model that is more appropriate for the data. The equal-variance normal model is just a bad fit for RT data. So lesson#1: fit *appropriate* models explicitly, rather than implicitly 6/n
This might seem like a nitpick, but it's important to note that the two are not qualitatively different in light of claims in the paper that the "generative modeling" approach somehow "provides an explicit mechanism to explain the observed changes in data". The paper talks 7/n
a lot about mechanisms, but there is absolutely nothing mechanistic about fitting a lognormal distribution. This is a measurement model. The line between mechanistic/process models and measurement models can be blurry (e.g. diffusion models), but in principle, a mechanistic 8/n
model would provide an explanation for why responses follow the distribution fit by the generative model. I don't think that the benefits of the current approach to improve test-retest reliability estimates has anything to do with mechanisms, but with 1) specifying a better 9/n
model for the data distribution, & 2) by reducing parameter uncertainty due to shrinkage in multi-level modeling. Both are great & as the paper shows, they lead to demonstrable improvements in parameter estimation. But I'd steer away from claims about mechanistic inferences 10/n
The generative model approach improves the reliability of the statistical inferences from the data, but on its own does little to resolve the "theory-description gap" that is identified in the beginning of the paper. For that to happen, you need an explicit specification of 11/n
the cognitive mechanisms that relates the substantive theory to the predictions it will make about the data. Those can be integrated with generative multi-level models as connecting the substative model with data as in @AdamOsth paper https://sciencedirect.com/science/article/pii/S0010028517303158 That's my take 12/12
PS: I want to stress that I think this is a great paper and I enjoyed reading it. My reservations are specifically about some of the mechanistic claims, but its strengths as a clear and comprehensive tutorial for improving individual differences research are undeniable
