With the Brexit vote looming, campaigns on both sides are clamouring to shout the loudest, and with the most authority. In light of both sides bombarding us with figures, statistics, research and costs, Tim Harford really hits the nail on the head here. The implicit trust that cold, hard numbers used to invoke is no more.
As the general populace becomes a little more wary, it is more important than ever that we back up our assertions. The sample size must be reasonable, and the results will be scrutinised.
In the world of PR and marketing, it's time we realised that a pretty chart won't quite hack it anymore.
An unscrupulous researcher can grind through the myriad combinations until a statistically significant pattern appears. But, says Gelman, there is no reason to think such unethical behaviour is common. More likely, researchers gather the data, look informally at the patterns they see, and only then choose a few hypotheses to test. They will tell themselves — correctly — that they’re being led by the data. That’s fine. But nobody should take seriously a test of statistical significance that emerges from such a research process: it will bring up fluke after fluke.