Read on Twitter
(Thread) I fully support the aims of this research & generally agree w/authors’ stated positions re: the field's problems & reform needs. But I think some of the conclusions are overstated given the survey specifics. 1/n https://twitter.com/siminevazire/status/1351101217896886274
Disclosure: I took survey & answered as honestly as possible. I struggled w/“proportion of field engaging in X” questions. We cannot *know* so questions essentially ask for guesses (subjective priors) w/high level of specificity (0-100). Hence, uniform dist is unsurprising. 2/n
Most important to me is frequent statements like “Criminologists use…” & “Criminologists have a QRP problem” are problematic – even if (likely) correct. Why? Present tense is unwarranted in tweets & throughout paper. Questions retrospectively ask if “ever engaged in” acts. 3/n
IMO, this is a big deal. First, “ever” questions will maximize prevalence estimates relative to preferable alternatives (e.g., recent retrospective; current acts; future intent). Second, there was no attempt to measure behavior change, e.g., post-replication crisis. 4/n
For me, there are things I *had* done prior to replication crisis – things that were normative & often required by reviewers/editors as preconditions to publication. But, due to awareness & reforms from replication crises, can now refuse w/potent justifications. 5/n
Cases-in-point: Reviewers have demanded moving interesting exploratory results from discussion (as exploratory) to front end (HARKing); others have required dropping presumably less interesting results in favor of their preferred edits (underreporting). 6/n
To make claims re: what criminologists (currently) do or re: the field having a QRP problem, need to know extent to which QRPs are *still* problems today, given today’s heightened awareness, shared language re: QRPs, & growing knowledge re: reforms. 7/n
As survey is skewed toward mid/senior scholars, how many participants selected into survey bc they are aware of QRPs, self-reflective re: past actions, & actively avoiding these acts today? Maybe few, or perhaps many/most – we simply cannot know from this study. 8/n
Paper acknowledges this: “…positive relationship between OSPs and QRPs may be due to changing research practices… whereby criminologists who once used QRPs… now use OSPs.”… Our survey did not ask when [they] used [QRPs], only whether… (p.28). 9/n
The positive corr btw self-reported QRP & OSP behavior alongside negative corr btw QRP & OSP support may be more than evidence of individual behavior change – it may also be indicative of some sample selection (e.g., on transparency & reform aims). 10/n
(An aside: Why moralistic language – e.g., “making good” or “going straight” on bad behavior? Survey already aimed at choir, topic preaches to choir; now a sin & atonement sermon for the choir too? Feels like a lot of assumptions beyond data packed in here.) 11/n
So, results show non-negligible proportion of respondents (item avg~.30) reported ever engaging in 10 measured QRPs at some in past. May oversample on transparency & OSP support; behavior items do not indicate extent to which QRPs have changed/are a problem today. 12/n
However, IMO, attitudinal/belief questions (i.e., “Support for QRPs”) are more informative. Even given likely selection, a non-negligible proportion of respondents (~20%?) report criminologists *should* engage in QRPs “often” or “very often.” 13/n
If these largely represent sincere responses then, to me, this finding represents the strongest evidence in study that Criminology still has a QRP problem *today.* Disclosure: Akin to the study authors (I suspect), this finding aligns with my priors re: the field’s problems. 14/n
In fact, I suspect that a (hypothetical & impossible) survey w/better response rate, sincere reporting, & devoid of sample selection on transparency, QRP/OSP support, & behavior would likely show even greater problems. But, for now, let’s take care to stick w/what we know. 15/15
