April 24, 2007
The Meaning of Error
I am sitting at a meeting of a National Academy of Science Committee on Identifying the Needs of the Forensic Science Community. Forensic scientists addressing the committee have said that it is not a "false positive" or not an "error" when a test (such as microscopic hair comparison or ABO blood typing) is "correct" in the sense that it gives the best result it can (two hairs really are indistinguishable under the microscope, the blood really is type A). Judge Harry Edwards, the co-chair of the committee, disputed this terminology, saying that if more discriminating mitochondrial DNA testing correctly establishes that the hairs actually come from different people, then the microscopic comparison was an error.
Who is right? Well, both. The laboratory has not erred in the sense that it has applied the test correctly. This is an internal perspective on the process. Judge Edwards also is right. The test has erred from an external perspective. If a court convicts an innocent man because the microscopic features of his hair match, that is a substantive error. Would the forensic scientists insist that an eyewitness who looks carefully and has a good memory but nevertheless misidentifies an assailant has not erred?
The point is that there are different sources of possible error. If we are interested in the error rates of a properly performed test, then the external perspective is appropriate. Such a test has a measurable sensitivity and specificity, and we need to know these statistics to evaluate its validity and utility. If we are interested in proficiency or reliability, however, the internal perspective applies.