Xkcd, the comic, often runs numerically oriented cartoons. But, recently, in the xkcd blog, he announced a real problem that calls a fair amount of real linguistic work into question.
The problem seems to be those useful result counts provided by Google
may not be at all reliable.
He writes:
The “number of results” count that Google gives when you search is clearly fabricated. This is clear for a few reasons. When Google says this:
You can tell that it’s wrong first by scrolling to the end of the results. When you get to page 32, it suddenly becomes:
This doesn’t usually matter, since nobody looks much past the first few pages of results, but it’s annoying if you’re trying to use the number of results as a measure of something.
But, the problem is that these numbers have actually been used for linguistic research. It is vastly easier to use Google’s numbers than to collect a corpus of data and count the occurrences on your own. Please read the blog post for more details.
Note added 14 Feb 2011:
This problem was around in 2005 also. See these blog posts by Jean Véronis 1, and Mark Liberman: 2. It looks like anyone who wants to use Google counts for linguistics research needs to prove them