Greg Kochanski

What is a bug?

So many people, especially in the free software community have a mistaken view of bugs. Many developers have this view: It's not a bug if it works according to the documentation.

But, that view is wrong on several counts.

So many documents, so little time.

First of all, most users don't actually read any more documentation than they absolutely have to. Doubtless, some of you are thinking "Lazy slobs! I worked hard to write that documentation and if you don't read it, you get what you deserve!"

But before judging your fellow man harshly, consider a few facts. On my Linux box (running Debian), there are 515 "README" files in the documentation directories, totaling to about a megabyte (that's a thousand pages of README!). Add in the files ending in ".txt" and ".html", and you get 8610 files, totaling 79 megabytes, or about 80,000 pages.

Printed, that's a stack of documentation something like 80 meters high, laid side-by-side, it would stretch for 30 kilometers. That's all the way around Manhattan, or most of the across the English Channel. At 10 seconds per page, for a high-speed skim, that would take about a month to read, allowing for some sleep and meals. And, that's ignoring the FAQ and NEWS and examples.

Most of us have something else to do with our lives than read the documentation. About 20 years ago, computer systems crossed the line of complexity where a human could reasonably be expected to understand everything in detail. Consequently, real software needs to be useable even if the users haven't read the documentation.

Software is psychology, not math.

Second, one has to realize that proper way to judge software is not correctness against its documentation, but how useful it is to the typical user. Certainly, it is good if the software agrees with the documentation, but that is only a means to an end. The whole point of writing a computer program is to help someone do something.

Inconsistent documentation is bad because it confuses the user, but that is just one of many ways for software to be bad. Even if the documentation were fixed to be perfectly consistent with the program, there would still be many possible ways for the software to be bad. (A simple example is a window manager program that paints all graphical elements as the same shade of blue. Even if the documentation reveals this to the user, it will still be rather hard to find the windows...)

So, usefulness is a better criterion for good software than "correctness of the documentation," because if you can use it, who cares if the documentation is correct? The one implies the other. Correct documentation only implies usefulness if (a) the user reads it, (b) the documentation is written in a way the user can understand, and (c) the user's problem is covered by the documentation.

One can claim that all of computer science is really a single extended user-interface experiment. The goal is to figure out how to make chunks of silicon understandable and useful to people. So, perhaps computer science is a variety of experimental psychology. The bits form the maze, and users play the role of white rats.

Design Documents can be Wrong.

Matching software to the documentation is really a way of saying that the software implements its design. That's nice so long as the design is good. But can one write a bad design? Of course! In fact, few designs are so good that they cannot be improved.

So, if the code matches the documentation, that just means they have the same design, for better or worse. Any design flaws in one will be reflected in the other. A perfect implementation of a bad design will still not be very useful. Think of a car with square wheels on the blueprints; implementing the design means you'll do something even worse than the Edsel.

Designing for Users

Any software that requires documentation is, to be blunt, either a highly specialized tool for experts only, or written by egotists who believe that their pet project is the only software than users need. OK, so I exaggerate. I do read some documentation, and don't grudge reading some documentation for a program that I find useful. Still, every bit of necessary documentation reading uses precious time and is a mark against the program.

The trick is for the program to minimize surprises and to be actively helpful.

One final observation is that very little free software is designed for the naive user. That's perhaps understandable, as the people who write it are experts. What I find is odd, though, is that they were all naive once, and I'd expect that they would remember what it was like. Perhaps many do, but equally clearly, many have forgotten.

This is not to say I'm not sometimes grateful for hard-to-use annoyingly picky software that I have to struggle with. It can be worth the work, and I appreciate that the authors (especially of free software) have made valuable contributions to humanity by releasing it. Even if I don't find it worthwhile, it may be a life-saver to someone with the right combination of knowledge and needs. Moreover, some of the developers work in the evening, and have sacrificed time to do it. Still, it would be better if people would say "I'd like to improve that, but I don't have time," instead of saying "It's not a bug, it's in the documentation."