Statistically Significant is Only Half of “Interesting”

Q: If we are dealing with really large samples, even extremely small differences turn out significant using standard statistical tests.  How should I think about this?

A: Recognize that there are two important factors and that statistical significance is only one of them.

The other factor is the size of the effect.  In a large experiment, it’s quite possible to detect effects that are so small that no reasonable person would care about them.

For instance, people are almost left-right symmetric, but not quite.  Hearts are tilted over, left brains are a bit different from right brains, most people are right handed and the right arm is more dextrous than the left.  So, what if we do an experiment measuring the metabolic rate of cells in the left and right big toe. I’ll bet that if you measure a million people, you will find a significant difference.  But, suppose that we measure a statistically significant difference of 0.03% between the cells from the left and the right.  Will you get excited, or will it be just one more of those little differences between the left and right halves of the body?

Or, take IQ. I’ll bet that IQ depends on shoe size. But I’ll also bet that the difference between people with big feet and people with small feet is well under 1 point, so who cares?  [Especially since the typical person-to-person difference is about 20 points.]  Would it make you feel better if you knew that your above-average IQ was enhanced (a tiny bit) by your large feet?

Darn near everything in humans is correlated with darn near everything else.  That’s because the toe bone’s connected to the foot bone…  Or, more relevantly to psychology and linguistics, all the mental processing for grammar happens inside the same skull, and there are neurons that connect almost any two regions of the brain.  Given that anatomy, it’s hard to imagine that there is any grammatical effect that’s not affected by (e.g.) your love for waffles.

So, when you do an experiment, remember that the interesting effects are the ones that are both significant and relatively large.