Science, values and the limits of measurement

Metrics play a growing role in managing research. But to understand their limitations, we need to draw on the humanities.

There is a particular form of proof that is applied both by mathematicians and by critics of using metrics in research assessment. Proof by contradiction seeks to prove something, say that the square root of two is an irrational number, by first assuming its opposite and then proceeding to demonstrate an internal contradiction. It follows that the assumption is proven untrue.

In the case of metrics in research assessment, the common proof that they don't work runs as follows. First we assume that metrics do work, so they should rank articles and researchers correctly. Now we look at a set of articles (or researchers or departments) which self evidently fall into this rank order. But the metrics get the ranking wrong. Therefore by contradiction metrics don't work. QED.