Spotting Bad Science

Please pipedot, we can do better than that.

Over the past decade or so, I have come to think that published bad science is shockingly common. I have tried to ruminate on the reasons for this and, while it is easy to identify many contributing factors in the way scientific communities and institutions work, I think the main reason is that science has two completely different functions in human life that work against one another. On the one had, science can be about challenging our own preferred, learned, or expert hypotheses (a la Feynman: "Science is the belief in the ignorance of experts" . On the other hand, science can be about presenting and defending our own preferred, learned, or expert hypotheses (a la Scientific American: Ask the Experts ). Both functions serve useful purposes in our lives at different times, but each can look bad from the perspective of the other. i.e., challenges to authoritative science can look bad and receive short thrift because we frequently have a lot of confidence in the authoritative accounts, but authoritative accounts can look bad when we stop prodding them for the ways they might be wrong or limited. Who gets to decide when it is appropriate to defend or challenge an authoritative account on some topic is a central issue. In my opinion, the commonness of published bad science is a consequence of spending too much attention focused on being right about nature and not enough attention focused on how we might be confirming our assumptions or biases.

At this point the meme has so thoroughly permeated the culture that I really wish every mention of "correlation is not causation" (or "correlation does not imply causation," or any of its variants) came with a warning label explaining that in most cases, there's a whole lot more to the analysis than that.