2011: Joseph Simmons, Leif Nelson, and Uri Simonsohn publish a paper, “False-positive psychology,” in Psychological Science introducing the useful term “researcher degrees of freedom.” Later they come up with the term p-hacking, and Eric Loken and I speak of the garden of forking paths to describe the processes by which researcher degrees of freedom are employed to attain statistical significance. The paper by Simmons et al. is also notable in its punning title, not just questioning the claims of the subfield of positive psychology but also mocking it.
Pointer from Alex Tabarrok.
I am pretty sure that at some point prior to 2011 when I criticizes macro-econometrics I said that the degrees of freedom belong to the researcher rather than to the data. That is a minor note.
More important, I think that John Ioannidis deserves a mention. Yes, Gelman is focused on research in the field of psychology and Ioannidis focused primarily on epidemiology, but his paper Why Most Published Research Findings are False strikes me as a milestone worth including in the timeline.
Gelman’s post is mostly about the tension between insiders and outsiders in the academic world. The insiders’ chief weapon is the peer-reviewed journal article. The outsiders’ chief weapon is the blog post. If, like me, your heart is with the outsiders, you will find Gelman’s post bracing.
I should note that in my high school statistics class last year, I had an autodidact student who, among other things, was very familiar with the term p-hacking and the related literature. This gives me hope that as the generations turn over in academia, things might improve. As Max Planck is said to have remarked, science advances one funeral at a time.
P-hacking was discussed in a popular online/youtube science show called Veritasium. There are several of these what you might call edutainment 5-minute format science shows, and they are all pretty good – Scishow, One Minute Physics, Vsauce, PBS Spacetime, many others. The target audience ranges from 10 to adult, but the content is often a bit fluffy for adults.
I incorporate both Gelman and Ioannidis into the training I have been composing for intelligence analysts on reading the scientific literature. They each have a significant sphere of influence.
Of course there is also this which suggests top researchers misinterpret p-values:
http://andrewgelman.com/2016/07/26/29552/
They seem to treat them dichotomously rather than continuously (i.e., is p less than 0.05 or not), focus on them when irrelevant, and ignore other relevant information such as effect sizes.
As someone commented on the post, “apparently the very introduction of a p-value makes people confused about whether 8 greater than 7.”
[sorry for the multiple posts…the less than sign garbled everything]