Surprise and Experiments

Sherlock Holmes once said, by way of Conan Doyle, that “… when you have eliminated the impossible, whatever remains, however improbable, must be the truth.”  He was wrong.  Not in the strict, logical sense that the truth must remain, but rather, he was wrong to pretend that this is a useful way to solve difficult problems.  To work, this maxim requires a God’s eye’s view, or perhaps the equivalently good viewpoint of a fictional detective.

For humans, it fails because when we apply Holmes’ maxim, our notion of “…whatever remains…” is limited by our human imagination, but the correct answer isn’t.  We are not omni-imaginary.  So, when we have eliminated the impossible  what remains — if anything — are the possible things that we could imagine.  So, will get the wrong answer if we can’t imagine the correct answer.

Doyle’s maxim also fails when we don’t have enough evidence.  Obviously, if we have no evidence at all, then anything we imagine is possible.  With incomplete evidence, there may be more than one remainder, but there is only one actual solution for the crime.  Each piece of evidence we collect limits the possibilities and if we had all the evidence, there would be only a single Sherlockian remainder, and it would be the truth, maybe.

In the real world there is always some missing information; something that has not yet been seen or a measurement that is too difficult or too expensive to make.  Since he has a good imagination, Sherlock Holmes will usually have more than one theory.   And, indeed, in many stories, Holmes had alternative theories, until the last crucial bit of evidence came in and he solved the crime.  That is one of the fictions in the stories, of course.  Sherlock Holmes always gets the necessary evidence, but real policemen and real scientists need to struggle along with less data than they want or need.

Sometimes we don’t ever get that crucial evidence that gets you get down to a single remaining possibility.  Then we have several options, only one of which will match the real world.   That is not good, but at least we know that we don’t have a complete answer yet.

What is worse is a combination of missing information and a limited imagination.  This combination happens almost all the time, because humans have a limited imagination and incomplete evidence is the norm.  The problem here is that we can get down to a single remaining theory and it can be wrong.  All it takes is a failure to imagine the right answer.  So, as we work on the case (or research problem), we would start with many possible answers.  After much hard work eliminating the impossible, we would have a single answer remaining, and it would be very tempting to stop and declare success.

But, if we stop, we will never realize that we had failed to imagine the correct answer.  The only way to know that we missed something is to try to empty the list of possibilities, and the only way to empty the list is to keep collecting evidence.  If Mr. Holmes’ imagination is imcomplete, and he stops the moment he has reduced the list to a single suspect, he will put someone innocent in jail.  But of course, that never happened.

So, Doyle’s maxim works in imaginary worlds that are bounded by the human imagination, where imagination can be complete.  However, it is not to be trusted in the real world.  But, we shouldn’t blame Conan Doyle: it was good philosophy for the time.  It wasn’t until the mid-20th century that philosophers like Karl Popper and T. S. Kuhn started trying to describe how science ought to work (Popper) and the ways it often works among real scientists (Kuhn).  Doyle can hardly be faulted for not imagining how his maxim could fail.

Another way of looking at the problem with Doyle’s maxim is that sometimes the real world will surprise you and produce something that you might never have imagined. And that is exactly why science and society need to have a strong experimental basis.

Consider a field that is led by theorists.  Theorists are essentially the storytellers of science.   Much like Holmes, they imagine possible ways that the world might work.  But, unlike Holmes, they are not fictional heroes, and sometimes their imagination will fail.  In their defense, theorists have a tougher task than Holmes ever did: Holmes’ opponents were human and their plans came from human imaginations.

Nature is not under the same constraint.  People cannot naturally imagine all the effects of Relativity; even Einstein didn’t.   Answers to straightforward questions like “What would a cube look like, if you zoom past it at 90% of the speed of light?” were mis-imagined by a generation of scientists.  Until the 1970s, half a century after  Relativity was formulated, we thought the cube would appear squashed along the direction of motion.  Only when someone applied the mathematics and actually computed what it would look like, rather than relying on their imagination, did we find that it would appear rotated rather than squashed.  The textbooks were wrong and we were surprised because our human imaginations failed.

Or, look at the theory of evolution.  Darwin’s insight was based on the evidence of his eyes; no fancy technology was required.   So, any well-travelled person in the preceding century could have seen the same kind of animals and environments that Darwin did.  There was nothing to keep such a person from seeing variation from animal to animal within a breed, recognizing that some animals would be a better match to their environment, that these animals would have a better chance of raising offspring, and so the next generation would be different.   Animals that were badly adapted to the environment wouldn’t raise many offspring, but animals that fit their environment well would have more offspring.   As a result, the next generation would be — on average — better adapted to its environment.

Anyone could have realized this, and realized that small changes could build up from generation to generation until the accumulated changes are large and obvious — until the animals have become a new species.  Until Darwin and Wallace came by (in the mid-1800s), everyone’s imagination had failed.  So, imagination can fail.  That fact shouldn’t be a surprise, although I suppose it would have surprised Conan Doyle.

Would this essay have surprised Holmes?  That’s actually a very metaphysical question.   Since Holmes lives in the London of Doyle’s imagination, if Doyle didn’t imagine something, it wouldn’t exist in Holmes’ London, so Holmes wouldn’t ever need to imagine it.    So, I imagine that this essay would not have existed in Holmes’ London, so it wouldn’t have surprised Holmes.