The Case of Racial Bias by Judges in Court Rulings

In Race & Gender of Judges Make Enormous Differences in Rulings, Studies Find, Edward A. Adams, reports in the ABA Journal on recent research discussed at an ABA Midyear Meeting program entitled “Diversity on the Bench: Is the ‘Wise Latina’ a Myth?.” Opening, big splash, paragraph:

A judge's race or gender makes for a dramatic difference in the outcome of cases they hear—at least for cases in which race and gender allegedly play a role in the conduct of the parties, according to two recent studies.

According to one of the cited ELS studies, Myth of the Color-Blind Judge: An Empirical Analysis of Racial Harassment Cases, Pittsburgh law prof Pat Chew and Carnegie Mellon business school prof Robert Kelley, Adams offers this summary:

In federal racial harassment cases, one study found that plaintiffs lost just 54 percent of the time when the judge handling the case was an African-American. Yet plaintiffs lost 81 percent of the time when the judge was Hispanic, 79 percent when the judge was white, and 67 percent of the time when the judge was Asian American.

Adams reports that at the program, Chew

"said she found 'the rule of law is intact' in the cases she reviewed. Judges—no matter which side they ruled for—took the same procedural steps to reach their decisions, she said.

But judges of different races took different approaches “on how to interpret the facts of the cases,” she said.

Pressed on whether the rule of law could actually be considered intact when outcomes varied so much depending on the race of the judge, she replied: "It’s always made a difference who the judge was. We’ve long known, for instance, that a judge’s political affiliation makes a difference."

Anything Proven Empirically? On whether Chew and Adams have proven anything empirically in their study is another matter. Unlike Law & Economics, Empirical Legal Studies, while promising, still has a long road ahead before establishing accepted statistical methods and analysis. Publication of these studies in law reviews (Chew and Adams' study can be found at 86 Washington University Law Review 1117 (2009)) instead of peer-reviewed journals isn't helping the situation. Do you think the student editors really understood what they were reading when they decided to publish the article? For that matter, did the audience at the Mid-Winter Meeting program?

Problems with the Chew and Adams Study. About this study David Cohen published the following comment to the ABA story:

Empirical legal research is in its infancy and, I think, should be encouraged.  But no good end is served by publishing sloppy work.  The first study mentioned, by Professors Chew and Kelley should not have been published in its current form.  I highly doubt that it could have been published in a peer reviewed social science journal used to publishing quantitative research.

There are lots of problems with their analysis and how they present it, but the three most important points (if I were reviewing their manuscript) are these:

1. They are looking at judges nested within circuits, and circuits make a difference to how plaintiffs must prove their case. Most discrimination lawyers will agree, I think, that they’d rather be in some circuits than others.  If, for example, we assume that plaintiffs win more often in liberal and urband circuits, and that black judges are disproportionately located in liberal and urban circuits, then even if black judges decide cases for plaintiffs at the same rate as other judges in their circuit, it would still appear that black judges nationally were deciding cases disproportionately for plaintiffs.

Using logit ignores the variance at the circuit level—that is, how much law and practice differences between the circuits affect the outcome of discrimination cases. The authors really need to use a multi-level statistical method. HLM, a popular multi-level software package, can handle logits.

2. The right way of presenting regression results is to run multiple models, starting with your control variables, then adding your variables of interest, and then, if you want, interaction variables. This tells you whether adding your variable of interest actually makes a significant difference to your model. (In technical terms, you can report delta-F and delta-R-squared.)  The authors never do this and there is some reason to think that the judge’s race would not, if analyzed properly, make a significant difference. They seem to be saying that, when analyzed alone, the R-squared (that is, how much of the variance between outcomes is explained by the judge’s race) is very low (0.03). Since, as explained below, this is likely to be too high due to omitted relevant variables, it might be that judges race does not add anything to their explanation of outcomes.

3. When you leave a relevant independent variable out of a regression analysis, the variables you put in appear more significant than they really are, depending upon how much they correlate with the omitted variable. That’s because some of the variance properly attributed to the omitted variable will end up being wrongly attributed to the included variable. So, if we think, for example, that judge’s race and political affiliation are correlated, then leaving out one will make the other appear more significant than it really is. All but one of the analyses the authors present leave out variables that the authors consider relevant (and all ignore which circuit the judge is in, which I consider relevant).

The one analysis that includes all of their relevant variables? Finds that judges race is not significant (p = 0.1). So the best evidence from this study (which is still not actually well-enough done for us to rely on this finding) is that judge’s race is not a significant factor and, at best, makes a very small difference in outcomes. 

(The authors try to argue that p = 0.1 actually means something—but that is an argument that is mostly rejected by the good peer-reviewed social science journals today. 10 years ago, you could sometimes argue that p = 0.1 was “marginally significant.”)

Still, the difference in plaintiff’s success does seem large and there might be something going on here. I’m not saying that judge’s race has no effect, I’m saying that this study doesn’t support any conclusion.  I wish that we could see a properly done multi-level analysis, because the effect might be to explain some of the unexplained variance, which would actually lower the p value of judge’s race, potentially making it significant. On the other hand, if black judges do tend to be disproportionately liberal or disproportionately urban, it could be that race has even less effect than the authors found.

(Emphasis added.)

Ah ... Here's how I will know when ELS is producing reliable results -- when peer-review journals start publishing dry-as-bones ELS research results like Law and Economics articles now do, then ELS will have moved beyond its current infancy stage. [JH]