Surprise and Experiments

Sherlock Holmes once said, by way of Conan Doyle, “… when you have eliminated the impossible, whatever remains, however improbable, must be the truth.”  He was wrong.  Not in the strict, logical sense that the truth must be somewhere among whatever remains, but rather, he was wrong to pretend that this was a useful way to solve difficult problems.  To work, this maxim requires a God’s eye’s view, or perhaps the equivalently good viewpoint of a fictional detective.

For humans, it fails because when we apply it, our notion of “…whatever remains…” is limited by the human imagination but the correct answer isn’t.  We are not omni-imaginary.  So, when we have eliminated the impossible  what remains — if anything — are only the possible things that we have been able to think up.  So, will get it wrong if we fail to imagine the correct answer: the list of remainders might be empty.

Doyle’s maxim also fails if enough evidence cannot be had.  Obviously, if we have no evidence at all, then anything we imagine is possible.  With incomplete evidence, there may be more than one remainder, but there is only one actual solution for the crime.  Each piece of evidence we collect limits the possibilities and if we had all the evidence, there would be only a single Sherlockian remainder, and it would be the truth, maybe.

In the real world there is always some missing information; something that has not yet been seen or a measurement that is too difficult or too expensive to make.  So, since he has a good imagination, Sherlock Holmes will usually have more than one alternative theory.   And, indeed he does, for much of each story, until the last crucial bit of evidence comes in and he can solve the crime.  That is one of the fictions in the stories, of course.  Sherlock Holmes always gets the necessary evidence, but real policemen and real scientists need to struggle along with less data than they want or need.

Sometimes we don’t ever get that crucial evidence that gets you get down to a single remaining possibility.  Then we have several options, only one of which will match the real world.   That is not good, but at least we know that we don’t have a complete answer yet.

What is worse is a combination of missing information and a limited imagination.  This combination happens almost all the time, because humans have a limited imagination and incomplete evidence is the norm.  The problem here is that we can get down to a single remaining theory and it can be wrong.  All it takes is a failure to imagine the right answer.  Then, we work on the case (or research problem).  We would start with a list of many possible answers.  After much hard work eliminating the impossible, we would have a single answer remaining on the list, and it would be very tempting to stop.

But, if we stop, we will never realize that we had failed to imagine the correct answer.  The only way to know that we missed something is to try to empty the list of possibilities, and the only way to empty the list is to keep collecting evidence.  If Mr. Holmes stops the moment he has reduced the list to a single suspect, he will put someone innocent in jail if his imagination has been incomplete.  But of course, that never happened.

So, Doyle’s maxim works in imaginary worlds that are bounded by the human imagination, though it is not to be trusted in the real world.  However, one shouldn’t blame Conan Doyle: it was good philosophy for the time.  It wasn’t until the 20th century that philosophers like Karl Popper and T. S. Kuhn started trying to describe how science ought to work (Popper) and the ways it often works among real scientists (Kuhn).  Doyle can hardly be faulted for not imagining how his maxim could fail.

Another way of looking at the problem with Doyle’s maxim is that sometimes the real world will surprise you and
produce something that you never would have imagined. And that is exactly why linguistics needs to have a strong experimental basis.

Consider a field that is led by theorists.  Theorists are essentially the storytellers of science.   They imagine, much like Holmes, possible ways that the world might work.  But, unlike Holmes, they are not fictional heroes, and sometimes their imagination will fail.  In their defence, theorists have a tougher task than Holmes ever did: Holmes’ opponents were human and their plans came from human imaginations.

Nature is not under the same constraint.  People cannot naturally imagine the effects of Relativity; even Einstein didn’t.   Answers to straightforward questions like “What would a cube look like, if you passed it at 90% of the speed of light?” were mis-imagined by a generation of scientists.  Until the 1970s, half a century after  Relativity was formulated, we thought the cube would appear squashed along the direction of motion.  Only when someone applied the mathematics and actually computed what it would look like, rather than relying on their imagination, did we find that it would appear rotated rather than squashed.  The textbooks were wrong and we were surprised because human imagination failed.

Or, look at the theory of evolution.  Darwin’s insight was based on the evidence of his eyes; no fancy technology was required.   So, any well-travelled person in the preceding century could have seen the same kind of animals and environments that Darwin did.  There was nothing to keep such a person from seeing variation from animal to animal within a breed, recognizing that some animals would be a better match to their environment, that these animals would have a better chance of raising offspring, and so the next generation would be different.   Animals that were badly adapted to the environment wouldn’t raise many offspring, but animals that fit their environment well would have more offspring.   As a result, the next generation would be — on average — better adapted to its environment.

Anyone could have realized this, and realized that small changes could build up from generation to generation until the accumulated changes are large and obvious — until the animals have become a new species.  Until Darwin and Wallace came by (in the mid-1800s), everyone’s imagination had failed.  So, imagination can fail.  That fact shouldn’t be a surprise, although I suppose it would have surprised Conan Doyle.

Would this essay have surprised Holmes?  That’s actually a very metaphysical question.   Since Holmes lives in the London of Doyle’s imagination, if Doyle imagined something, so could Holmes.  On the other hand, if Doyle didn’t imagine something, it wouldn’t exist in Holmes’ London, so Holmes wouldn’t ever need to imagine it.    Clearly, then, no essay would have surprised Holmes.