It's been almost ten years since two Google engineers published a paper describing the
architecture of Map Reduce , a framework for simplifying the development and deployment of algorithms that process terabytes or petabytes of data across a cluster of commodity servers. The Open Source community soon responded with
Hadoop , a Map Reduce work-alike, and in the following years it seems that most large IT organizations, and many startups, have jumped on the bandwagon pitching the virtues of Big Data, Hadoop and/or NoSQL as a revolutionary set of techniques for capturing actionable trends and correlations from the firehose of real-time data (clickstream, Twitter feeds, Facebook likes, server logs, sensor and surveillance data, mobile call events, and of course, all the stuff the NSA looks at).
Tim Harford of the Financial Times points out that this
methodology is subject to various types of sampling bias , even in cases where the more enthusiastic proponents claim to be 'observing the entire population, not just a statistical sample'. First, data collected from social media or smart phone apps is heavily biased by the user profile of those technologies, whch is disproportionately young, affluent, and urban or suburban. Harford mentions the famous case of the Literary Digest, a well-established magazine that forecasted a landslide victory for Alf Landon in the 1936 US Presidential election, based on
a massive poll of one out of five eligible voters - whose contact information was pulled from telephone subscriber lists (Landon lost the election to Franklin Roosevelt, who carried all but two of the 48 states; Literary Digest ceased publication soon afterwards).
Second, people adjust their behavior over time with respects to various topics in the news. The sudden increases in flu-related searches that made Google Flu Trends look very prescient five winters ago, turned into a debacle when Google used similar data to warn of a severe flu outbreak four years later; but the flu season turned out to be average when the curated data from the CDC finally came in.
What about the famous anecdote about
Target finding out that a teenage customer was pregnant before her dad did? Maybe so, says a researcher quoted in Harford's article, but there's an issue with false positives. The world likely didn't hear about other Target customers who got pregnancy-related marketing materials they wouldn't have any use for.
The Canadian Broadcasting Corporation (CBC) has reported on a potential case of cyber-bullying in Beloiel, Quebec . In a modern twist on high-school interactions, two girls took turns beating and mauling a boy in their school parking lot while the other videotaped the confrontation on a cellphone. The boy chose not to fight back. When posted to Facebook, the video got thousands of views.