Social science has a complicated, infinitely tricky replication crisis

Enlarge / The appropriate reaction to a pile of psychology textbooks, likely full of unreplicable results. (credit: flickr user: CollegeDegrees360)

For scientists, getting research published in the journal Nature is a huge deal. It carries weight, prestige, and the promise of career advancement-as do the pages of its competitor, Science. Both have a reputation for publishing innovative, exciting, and high-quality work with a broad appeal. That reputation means that papers from these journals make up a substantial portion of day-to-day science news.

But the prestige of these journals doesn't exempt them from problems that have been plaguing science for decades. In fact, because they publish such exciting and innovative work, there's a risk that they're even more likely to publish thrilling but unreliable papers. They may also be contributing to a scientific record that shows only the "yes" answers to big questions but neglects to mention the important-but-boring "no" results.

Colin Camerer, a behavioral economist at the California Institute of Technology, recently led a team of researchers in trying to repeat 21 social science studies from Science and Nature, successfully replicating 13 of them. The results, published yesterday (in Nature, naturally), may also hint at how our focus on positive results is biasing the literature. They also paint a complicated picture of the replication crisis in social science and illustrate how infinitely tricky the project of replication is.

Read 34 remaining paragraphs | Comments