EvidenceProf Blog

Editor: Colin Miller
Univ. of South Carolina School of Law

A Member of the Law Professor Blogs Network

Sunday, November 4, 2012

Eco-Dome: The Ecological Validity Problem With Most Polygraph Studies

Ecological Validity

The extent to which the conditions simulated in the laboratory reflect real life conditions. Using an experimental laboratory-based research method, as has been the tradition in Cognitive Psychology, rigorous control for confounding variables is put in place and the ideal is that the researcher can study the onlythe phenomenon of interest. By manipulating variables (so-called independent variables) in the experimental setup and observing the changes that result (measured in the change of the dependent variable) the researcher can infer causality: If (independent) variable X is changed, (dependant) variable Y also changes. By rigorously controlling for confounding variables and confining the experiment to the laboratory, the results can very well lack applicability and generalisability with regards to the richness of everyday life. Ecological validity refers to an acknowledgment of the fact that human action is situated and highly contingent on contextual factors/variables. To obtain 'valid' results, humans should be studied in the richness of their natural environment

As Adam B. Shniderman notes in You Can’t Handle the Truth: Lies, Damn Lies, and the Exclusion of Polygraph Evidence, 22 Alb. L.J. Sci. & Tech. 433 (2012), one of the main problems with studies on the accuracy of polygraph exams is the lack of ecological validity.

According to Shniderman,

Courts applying the Daubert standard often find that the polygraph fails to meet the standards of validity testing and having a known or established error rate. Courts upholding the exclusion rely on two general arguments relating to reliability (1) the error rate is unknown/debated/unacceptable, and (2) the studies of polygraph's error rate lack ecological validity and thus are insufficient to establish reliability. Several courts have taken a "generalist" approach to uphold the exclusion of polygraph evidence, meaning they simply state, as if an obvious and unavoidable conclusion, that polygraph evidence is too unreliable for court. For example, the court in State v. Wright held, "[g]enerally, the results of polygraph examinations are inadmissible because the reliability of the polygraph is questionable." In Scheffer, the Supreme Court's first case involving the ban on polygraph evidence, the Court upheld a per se ban on polygraph evidence in military courts. In his concurrence, Justice Thomas, along with three of his colleagues, held that "there is simply no consensus that polygraph evidence is reliable. To this day, the scientific community remains extremely polarized about the reliability of polygraph techniques." The opinion mentions S. Abram's book, The Complete Polygraph Handbook, which reports the overall accuracy of the Control Question Technique at approximately 87 percent, while other scientific studies suggest that the accuracy of the technique is "little better than [chance]." Particularly, the opinion cites the Office of Technology Assessment's 1983 report that states that the error rate for the polygraph is significant. The report, which reports the error rates claimed in twenty-eight contemporary studies of the polygraph, offered error rates ranging from 0 to 83 percent. While this report ignores numerous other studies, Justice Thomas gave the report its place in the history books as part of the rationale for excluding polygraph evidence. Other courts, and polygraph opponents, have criticized lie detection validation studies for lacking ecological validity.

As I have frequently noted, polygraph exams test stress, not truth.

The instrument typically used to conduct polygraph tests consists of a physiological recorder that assesses three indicators of autonomic arousal: heart rate/blood pressure, respiration, and skin conductivity. Most examiners today use computerized recording systems. Rate and depth of respiration are measured by pneumographs wrapped around a subject's chest. Cardiovascular activity is assessed by a blood pressure cuff. Skin conductivity (called the galvanic skin or electrodermal response) is measured through electrodes attached to a subject's fingertips.

Based upon the way that polygraph exams work, you can see why most polygraph studies lack ecological validity. Many of these studies are done on college (psychology) students with nothing or at most a few bucks at stake. Such a study might conclude that polygraphs are 87%, 90%, or even 95% accurate. But what does this tell us about the accuracy of polygraph exams on criminal defendants?

These defendants are often subjected to pretrial incarceration. They often have limited resources. They frequently have an overburdened and inexperienced public defender. Their liberty is often at stake. Their life is sometimes at stake. Etc., etc. It is easy to see why there are few false positives when carefree college students are tested. But what about jittery criminal defendants with everything to lose? (And what about sociopathic criminal defendants who are among the 5-10% generally recognized as untestable?).

So, my question is: Has there ever been or could there ever be a polygraph exam study with anything approaching ecological validity? I don't see how such a study would be possible. The closest example that I could find was described by David T. Lykken in Reply to Raskin and Kircher, 27 Jurimetrics J. 278, 280 (1987):

Raskin and Kircher prefer to rely upon a series of laboratory studies by Raskin and his associates, studies in which student volunteers are required to commit a mock crime and then to lie about it on a polygraph test. One of my four criteria for an acceptable study of polygraph accuracy excludes such experiments on the grounds that they do not adequately simulate the serious emotional concerns experienced by people undergoing lie detector tests in real life. Recently, however, three laboratory studies have been reported in which greater verisimilitude has been achieved. Two of them used the simple device of permitting each volunteer subject to privately decide whether to commit the 'crime,' which entailed a larger reward if he got away with it, or to accept a smaller payment so that he could be truthful on the subsequent test. The third study, done in a maximum security prison in Canada, achieved ecological validity by giving the inmate volunteers a reason to fear failing the lie test: they were led to believe that their failure might cost their fellow inmates a much desired prize and thus incur their wrath. One way we can tell that these recent laboratory studies were successful in eliciting genuine concern on the part of the subjects is that, in each case, about 40 percent of the 'innocent' subjects failed the polygraph test, just as in the better field studies.

So, in this study, in this study, there was a 40% false positive rate for inmates fearing that failing a polygraph could lead to incurring the wrath of their fellow inmates. What do we suppose the false positive rate would be for defendants facing a murder wrap and life imprisonment or the death penalty?



| Permalink

TrackBack URL for this entry:


Listed below are links to weblogs that reference Eco-Dome: The Ecological Validity Problem With Most Polygraph Studies:


Post a comment