Which of the following was not discussed as a contributor to the replication crisis?

The replication crisis is a big deal. But it’s a problem in lots of scientific fields. Why is so much of the discussion about psychology research?

Why not economics, which is more controversial and gets more space in the news media? Or medicine, which has higher stakes and a regular flow of well-publicized scandals?

Here are some relevant factors that I see, within the field of psychology:

1. Sophistication: Psychology’s discourse on validity, reliability, and latent constructs is much more sophisticated than the usual treatment of measurement in statistics, economics, biology, etc. So you see Paul Meehl raising serious questions as early as the 1960s, at a time in which min other fields we were just getting naive happy talk about how all problems would be solved with randomized experiments.

2. Overconfidence deriving from research designs: When we talk about the replication crisis in psychology, we’re mostly talking about lab experiments and surveys. Either way, you get clean identification of comparisons, hence there’s assumption that simple textbook methods can’t go wrong. We’ve seen similar problems in economics (for example, that notorious paper on air pollution in China which was based on a naive trust in regression discontinuity analysis, not recognizing that, when you come down to it, what they had was an observational study), but lab experiments and surveys in psychology are typically so clean that researchers sometimes can’t seem to imagine that there could be any problems with their p-values.

3. Openness. This one hurts: psychology’s bad press is in part a consequence of its open culture, which manifests in various ways. To start with, psychology is _institutionally_ open. Sure, there are some bad actors who refuse to share their data or who try to suppress dissent. Overall, though, psychology offers many channels of communication, even including the involvement of outsiders such as myself. One can compare to economics, which is notoriously reistant to ideas coming from other fields.

And, compared to medicine, psychology is much less restricted by financial and legal considerations. Biology and medicine are big business, and there are huge financial incentives for suppressing negative results, silencing critics, and flat-out cheating. In psychology, it’s relatively easy to get your hands on the data or at least to find mistakes in published work.

4. Involvement of some of prominent academics. Research controversies in other fields typically seem to involve fringe elements in their professons, and when discussing science publication failures, you might just say that Andrew Wakefield had an axe to grind and the editor of the Lancet is a sucker for political controversy, or that Richard Tol has an impressive talent for getting bad work published in good journals. In the rare cases when a big shot is involved (for example, Reinhart and Rogoff) it is indeed big news. But, in psychology, the replication crisis has engulfed Susan Fiske, Roy Baumeister, John Bargh, Carol Dweck, . . . these are leaders in their field. So there’s a legitimate feeling that the replication crisis strikes at the heart of psychology, or at least social psychology; it’s hard to dismiss it as a series of isolated incidents. It was well over half a century ago that Popper took Freud to task regarding unfalsifiable theory, and that remains a concern today.

5. Finally, psychology research is often of general interest (hence all the press coverage, Ted talks, and so on) and accessible, both in its subject matter and its methods. Biomedicine is all about development and DNA and all sorts of actual science; to understand empirical economics you need to know about regression models; but the ideas and methods of psychology are right out in the open for all to see. At the same time, most of psychology is not politically controversial. If an economist makes a dramatic claim, journalists can call up experts on the left and the right and present a nuanced view. Ta least until recently, reporting about psychology followed the “scientist as bold discoverer” template, from Gladwell on down.

What do you get when you put it together?

The strengths and weaknesses of the field of research psychology seemed to have combined to (a) encourage the publication and dissemination of lots of low-quality, unreplicable research, while (b) creating the conditions for this problem to be recognized, exposed, and discussed openly.

It makes sense for psychology researchers to be embarrassed that those papers on power pose, ESP, himmicanes, etc. were published in their top journals and promoted by leaders in their field. Just to be clear: I’m not saying there’s anything embarrassing or illegitimate about studying and publishing papers on power pose, ESP, or himmicanes. Speculation and data exploration are fine with me; indeed, they’re a necessary part of science. My problem with those papers is that they presented speculation as mature theory, that they presented data exploration as confirmatory evidence, and that they were not part of research programmes that could accomodate criticism. That’s bad news for psychology or any other field.

But psychologists can express legitimate pride in the methodological sophistication that has given them avenues to understand the replication crisis, in the openness that has allowed prominent work to be criticized, and in the collaborative culture that has facilitated replication projects. Let’s not let the breakthrough-of-the-week hype and the Ted-talking hawkers and the “replication rate is statistically indistinguishable from 100%” blowhards distract us from all the good work that has showed us how to think more seriously about statistical evidence and scientific replication.

What factors contribute to the replication crisis?

What is the main problem with replication in psychology?

How did the replication crisis start?

Why the replication crisis is a crisis?