August 26, 2006

Helping Legal Actors with Bayes’ Theorem

The importance of understanding base rates and Bayes’ Theorem cannot be overstressed, particularly in the case of many types of medical and scientific testimony. The importance of base rates is seen in the following problem: A disease occurs in 1% of the population, and a test has been developed which has an 80% accuracy rate (i.e., if you have the disease, there is a 80% chance the test will pick it up), and a 9.6% false positive rate (i.e., if you don’t have the disease, there is a 9.6% chance of getting a positive result anyway). Sam tests positive for the disease. What is the probability that Sam has the disease?

The general inclination is perhaps to say 90.4%, because the false positive rate is 9.6%. This conclusion, however, is wrong because it does not account for the rarity of the disease in the general population. (As doctors are often trained to think, if you hear hoofbeats, think horses, not zebras.) Using Bayes’ Theorem—and here I will spare the reader the mathematical details—one can show that the probability that Sam has the disease is 7.8%. Intuitively, this is because given the rarity of the disease, it is more likely that Sam is actually one of the false positives than one of the people with the disease. Short of being a math genius, however, crunching the numbers is extremely difficult to do intuitively, and merely plugging values into Bayes’ formula has a certain mystical quality that might make jurors (or judges) skeptical.

Psychological research by Gigerenzer & Hoffrage, however, suggests that people find analyzing the problem from a frequentist perspective far easier than from the probabilistic perspective shown above. We can see this by transforming the example above to series of frequencies: A disease afflicts 10 out of 1000 people in the population. For people with the disease, 8 out of 10 will have positive test results. For people without the disease, the test will still (erroneously) yield a positive result 95 out of 990 times. Sam tests positive for the disease. What is the probability that Sam has the disease?

The answer follows far more simply. Out of a population of 1000 people, 8+95=103 people will test positive. And of these 103 people, only 8 actually have the disease, so the probability that Sam has the disease is 8/103 = 7.8%.

Gigenrezer and Hoffrage write that only 16% of people presented with the percentages got the correct answer, compared to 46% of those presented with the frequencies.

This research is just one example of the many ways that psychology can help jurors and other legal actors grapple with scientific data and make better decisions. The underlying math is the same, but as we lawyers know, presentation is often half the battle (if not more). I’m indebted to Michael Bishop and J.D. Trout’s fascinating book, Epistemiology and the Psychology of Human Judgment (Oxford 2005) for calling my attention to this work and providing the above example. In their book, Bishop & Trout defend a new interdisciplinary approach to epistemology that takes advantage of current psychological research and argue how epistemology should be reconceived to help people make better everyday decisions.

--EKC

August 25, 2006

The Perils of Profiling

Professor Bernard Harcourt has an interesting op-ed in today's New York Times regarding the Transportation Security Administration's intention to expand behavioral profiling at the Nation's airports.  [See here]  He questions the validity of such techniques and recommends a straightforward thorough search of each passenger as a more effective solution.  I agree.  But I would add that there is another danger to adoption of such profiles.  They, in fact, tend not to narrow the population worth targeting for greater intrusion, they expand it.  Experience with behavioral profiles of the past, especially drug courier profiles and hijacker profiles, indicates that just about any behavior eventually finds its way into the profile's definition.  This means that law enforcement personnel are granted a large measure of discretion under the cover of a purportedly objective test.  For instance, courts have upheld searches of suspected drug couriers on the basis of a DEA profile that listed suspicious behavior as including (1) getting off an airplane first (United States v. Millan (8th Circuit)), (2) getting off an airplane in the middle (United States v. Buenaventura (2d Circuit)), and (3) getting off an airplane near the end (United States v. Mendenhall (S.Ct.)).  Other suspicious activity identified in court opinions includes (1) carrying a large heavy bag, (2) carrying little or no luggage, (3) paying for the ticket in cash with large bills, (4) paying for the ticket in cash with small bills, (5) paying for the ticket in cash with a combination of large and small bills, (6) leaving the airport by taxi, (7) leaving the airport by hotel courtesy van, (8) leaving the airport by private transportation, and so on.  The reader, I am sure, gets the point.  The problem with behavioral profiles is that they actually permit screening personnel to pursue hunches and intuition, possibly based on illegitimate if not unconstitutional bases, but to hide behind scientific claims of accuracy.  In the past, courts have not demonstrated either the inclination or the capacity to demand valid data to support reliance on behavioral profiles.  It is not that such profiles could not prove to be an effective law enforcement tool, but that they never are proved to be effective.  If TSA goes forward with its plans to step-up use of behavioral profiles, courts must demand the data that would support their continued use.
-- DLF

August 24, 2006

What's wrong with "near-match" DNA database searches?

This week, USA Today reported that the FBI will perform “near-match searching” of CODIS, the Convicted Offender DNA Index System. One might wonder why would police want to know the names of individuals whose DNA profiles fail to match those from samples from crime scenes or rape victims? The answer is simple genetics -- a near match implies that a close relative might be the source of the crime-scene DNA. Police in the United Kingdom have scored several significant successes with the technique. In the words of one official there, "We're having a positive effect on cases (and) preventing additional victims.  What's wrong there?"

Plenty, says Tania Simoncelli, who studies DNA database issues for the ACLU. In her view, “It's the worst kind of privacy intrusion.” Really? Worse than breaking into a suspect’s home and ransacking it for evidence?  Worse than installing a hidden video camera in a bedroom?  Worse than reading my mail or eavesdropping on my telephone calls?  Worse than breaking into a psychiatrist’s office to obtain a doctor’s files and notes?

OK, maybe there are worse invasions of personal privacy, and the ACLU is just engaging in the equivalent of campaign rhetoric here. The question is still what, exactly, is wrong with “near-match searching”? What valid privacy interest can a brother of a convicted offender assert to stop the police from using lawfully acquired information to develop a lead that points to him as a possible culprit? Surely the police can show a mugshot of the convicted offender to a woman who has been raped. If the woman reports that the man who raped her looked a lot like the photo, what is wrong with investigating to see if the brother, who closely resembles the mugshot, might be the rapist? Or suppose that the police question a suspect, who says, “I didn’t do it, my no-good brother did.” The brother might well be upset when the police come knocking on his door, but it is hard to claim that the intrusion on his “privacy” is unwarranted.

What is different when the investigative lead comes from the brother's DNA rather than his face or his mind? --DHK

The Continuing Monkey Wars

The New York Times is reporting today that evolutionary biology has been removed from the list of acceptable fields of study that are eligible for low-income federal grants.  [See story here]  Although the Department of Education claims that it was a clerical error, albeit an extraordinarily convenient one (given the Administration's beliefs about science), the omission is deeply troubling.  The error, no doubt, will be quickly corrected.  But it can be chalked up as one more instance of this Administration's scientific illiteracy and its seeming desire to impose that disability on the Nation.  For an excellent book on this subject, see Chris Mooney's "The Republican War on Science."
-- DLF

August 23, 2006

Handwriting "EXPERTISE"??

This past Saturday, August 19th, the Jacksonville Daily Progress reported the "findings" of a local "handwriting expert" that John M. Karr probably is guilty of the killing of JonBenet Ramsey, based on his analysis of several of Karr's writings and the ransom note left at the scene of the crime.  (Attentive readers will also recall that at least one other handwriting expert had identified the ransom note as having come from the hand of JonBenet's mother.)  According to the story,

Local handwriting expert Don Lehew, of Mt. Selman, has obtained several samples of John M. Karr’s handwriting, as well as, a copy of the Ramsey ransom note, and according to his analysis, they are all written by the same person. Lehew compared the ransom note to an old yearbook entry by Karr and an employment application Karr filled out when applying for a teaching job.
According to Lehew, several writing features on the yearbook entry were identical to the ransom note, including the unusual curvature of Karr’s “D”s and “L”s, and the odd angle of his “S”s.
“In both samples the slant of the writing is mixed; sometimes it slants right, and sometimes it slants left,” Lehew said.

But the story gets better.  This guy, according to the story, can not only identify Karr as the author of the ransom note, but it his expert opinion (based on research, no less) that Karr's writing indicates that he is a pedophile:

“Over the last three years, by collecting samples of known pedophiles, I have created a profile, by identifying the things that are common in their handwriting,” Lehew said. “I’ve looked at writing samples from 23 or 24 known pedophiles to create this profile. I’ve been doing this research for about three years.”
According to Lehew, his profile cannot identify if a person is a pedophile, only that they possess the propensity to be or become one.
“I can’t look at someone’s handwriting and say, “That’s a pedophile.” What I can do is look at it and say, of all the handwriting samples that I have looked at, about 10,000 of them in the past 25 years, this one closely approximates and relates to the handwriting of known pedophiles.”
Based on Lehew’s pedophile profile, Karr’s writing style displays several of the hallmarks of a child molester.
“Triangles or figure eights in the lower zone, retracing, leading in from below the baseline, these are all behaviors that have been observed in convicted pedophiles,” Lehew said. “He (Karr) has the tendency — he scored high on the pedophile chart.”

And Lehew is not just some self-styled expert.  According to the story, he is a member of the American Board of Forensic Examiners and teaches questioned document courses at Handwriting University.  He has also appeared numerous times in both state and federal court and is "certified by the Department of Public Safety to teach questioned document examination to law enforcement throughout Texas."

I could add an editorial comment here, but it hardly seems worth the effort.  I will simply refer readers back to Donald Kennedy's Science editorial (see August 19th post), in which he asked whether "Forensic Science is an Oxymoron."  The answer, clearly, is sometimes yes!
-- DLF

August 22, 2006

Fame, the Founding Fathers, and Modern Psychology

The New York Times has an interesting article on "the fame motive," in its Science Times section today.  [see full article]  The article observes that psychologists have recently begun to study fame's motivating forces and how some people can be overwhelmingly devoted to its attainment:

For most of its existence, the field of psychology has ignored fame as a primary motivator of human behavior: it was considered too shallow, too culturally variable, too often mingled with other motives to be taken seriously. But in recent years, a small number of social scientists have begun to study and think about fame in a different way, ranking it with other goals, measuring its psychological effects, characterizing its devoted seekers.

The article reminded me of the debate between Anti-federalists and Federalists at the founding of our Republic.  The Federalists, men like James Madison and Alexander Hamilton, who advocated ratification of the Constitution and a strong central government, recognized man's "love of fame," an insight associated with David Hume.  For the Federalists, this love of fame would motivate men to participate in government, but had to be checked and balanced, since it could lead men to abuse the power they held.  One prominent Federalist, John Stevens, for example, criticized the Anti-Federalists' "talk of virtue as the spring of action."  He considered such a basis for government "too feeble."  He said that human nature was self-interested and motivated by ambition:

Can any man, who has a tolerable acquaintance of human nature, imagine that men would so eagerly engage in public affairs, from whence they can hope to derive no personal emolument, merely from the impulse of so exalted, so pure, so disinterested a passion as patriotism, or political virtue?  No!  It is ambition that constitutes the very life and soul of Republican Government.  As fear and attachment insure obedience to Government, so does ambition set its wheels in motion.

Madison, who shared Stevens' views, observed that "Respect for character is always diminished in proportion to the number among whom the blame or praise is to be divided.  Conscience, the only remaining tie, is known to be inadequate in individuals: In large numbrs little is to be expected from it."    Madison's observations of psychology led him to support a government structured by checks and balances, one that would not invest too much authority in any one department.  In The Federalist, No. 51, Madison penned these famous words:

But what is government itself, but the greatest reflection on human nature?  If men were angels, no government would be necessary.  If angels were to govern men, neither external nor internal controls on government would be necessary.  In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself.

The Federalist vision of human nature, of course, prevailed.  (I should point out, just to be clear, that the "Federalists" of the founding generation had little in common with the "Federalist Society" of today.  The latter advocate positions that are more closely associated with the Anti-Federalists.)  We tend to forget, however, how much of the behavioral observations of the founders influenced the form of government they established.  Morevover, many of the debates about human nature should continue to inform how we understand and interpret the relations between the branches of the federal government and the dynamic between that government and the states.  Modern psychology is unlikely to definitively answer such basic and profound questions, but modern research methods can help shed light on these ancient debates.  And, perhaps, some of this scientific work might even be relevant to concrete questions we have today, such as how much power should be invested in the Executive Branch in these treacherous times that we live in.  For Madison and many of the other founders, the answer was clear.  In Hamilton's words:

Political writers have established it as a maxim, that, in contriving any system of government, and fixing the several checks and controls of the constitution, every man ought to be supposed a knave; and to have no other end in all his action, but private interest.  By this interest, we must govern him, and by means of it, make him co-operate to public good, notwithstanding his insatiable avarice and ambition.

Hence, while we must invest great power in our elected representatives, we should never forget the basic motivating forces of human nature.  Perhaps the most basic operating principle of the American Constitution is that it checks ambition with ambition through the checks and balances inherent in the division of power.  As the founders understood, Investing too much authority in any one branch, no matter how virtuous we might consider those occupying the offices of that branch, is contrary to the original plan of government and likely to result in the abuse of power.
-- DLF