How to Lie with Statistics is an excellent, old book that puts you on guard against all the classic ways to mis-use statistics to confuse. One of their examples is something I remember from my childhood TV-watching.
Four out of five dentists recommend Trident® sugarless gum for their patients who chew gum
or
Four out of five dentists surveyed preferred Crest toothpaste.
and there seems to be a modern version, too, dating to 2006 or so:
Four out of five dentists would recommend Crest Pro-Health over other brands
Now, these are impressive statistics. Too impressive. Those claims may actually have been true, but I would be deeply skeptical until I saw their methodology.
Let’s look at the first one. This is actually a plausible statement since Trident® was the most heavily advertised sugarless gum at the time. It would be a sorry dentist who would recommend gum with sugar, so without strongly competing gums, people (i.e. dentists) would jump to the name that they were most familiar with. But, even if there was strong competition, it would be easy enough to design a survey that would produce the desired result. If I wanted to be dishonest, I’d do it like this (Note: these questions were not part of the actual survey, which could very well have met strict standards.):
- Put a small logo somewhere. I’d do a telephone survey, and I’d mention My Sugarless Gum® early on in the phone call, just to plant it in the minds of the “subjects”.
- Have a few questions to emphasize the importance of sugar in gum. Make the “subjects” say that sugar is bad, bad! Bring that to the front of their minds.
- Ask the “subjects” to list several brands of gum, and for each brand, ask if it is sugarless. Give them yes/no/don’t know options. You’re hoping that the “subjects” will be uncertain about most brands, but that they’ll be sure about My Sugarless Gum ® because the “sugarless” part is a central part of ypur advertising.
- Then, you’re ready. Ask subjects to choose a gum that they would recommend, if one of their patients asked. Make it hypothetical so it doesn’t seem serious and important. People will jump for the name that they’ve recently encountered, and voilà! You have a result.
Or, of course, you could also survey groups of five dentists until you get the answer you want…
Now, if I get a survey like that, I’m going to hang up. I have better things to do with my life than be an unpaid part of someone else’s advertising campaign. But, if I hang up, then they can easily ignore my opinion. In fact, they have no choice to ignore me, since they won’t even know what my opinion is.
And that thought gives us one more tool for writing dishonest surveys:
- Set up the survey so that it’s easy for people who disagree with you to drop out.
Your sample of “subjects” will then be enriched in the views that you want to report.
Needless to say, only creeps and cads use these techniques intentionally. Or, people who are overly desperate to keep a job that allows them no self respect, I suppose.
NB: I’ve since run across a good blog post on this topic.