The shrinking of the big data promise

Markets do not have much patience for a commitment to techniques that don't deliver. Unfortunately, spy agencies aren't subject to this discipline

"Regression to the mean" is one of the subtlest concepts in statistical literacy - and yet it's terribly simple. In plain English, "regression to the mean" is the idea that normally, things are pretty normal. That is, if you observe something abnormal - a high fever, a high share price, a long, unseasonable stretch of sunny or rainy days - then chances are that all will soon fall back within normal range. If you're sick, you'll probably get better (this is why so many quack cold remedies "work" - they take your mind off the passage of time while you wait for regression to the mean to assert itself).

Regression to the mean applies to commerce. An advertising technique that works very well today will probably work less well over time, as repetition and competition eat away at its novelty. Look at those "ghost ads" on the sides of faded Victorian buildings: apparently, it was once profitable to advertise soap with a slogan like "It makes you clean."