Crowd Control - Bar Exam Style

With the move to the Next Generation bar exam, here's an interesting chart, produced using data from the ABA and printed by the State of California, that might caution about placing too much trust in numbers to do the hard work of measuring competency to practice law.

So let's take a closer look. What do you see?  On the horizontal access, we see cut scores for the jurisdictions varying from low too high.  On the vertical access, we see attorney discipline rates corresponding with those jurisdictions. Except for the two outliers, North Dakota and South Carolina, the chart suggests that bar exam cut scores have no apparent relationship in lowering rates attorney discipline claims.  Or, as the State of California put it:

What the scatter plot shows is that attorney discipline – as measured by private and public discipline per thousand attorneys – appears to have no relationship to the cut score. With so many states using 135 for their cut score, the details of the Figure can be somewhat difficult to tease out. The big picture, however, is clear. At a cut score of 135 the rate of attorney discipline ranges from a low of 1.9 per thousand in West Virginia to 7.9 per thousand in Tennessee. Looking across the entire range of cut scores we see strikingly similar rates of attorney discipline in states with cut scores from 130 – Alabama – all the way to 145 – Delaware. California’s rate of discipline (2.6 per thousand) is just over one-half (55 percent) the rate of discipline in Delaware (4.7 per thousand). Given the vast differences in the operation of different states’ attorney discipline systems, these discipline numbers should be read with caution. But based on the data available, it raises doubts as to whether changing the cut score would have any impact on the incidence of attorney misconduct. As with the research conducted by professors Anderson and Muller, this measure of “misconduct” is admittedly limited to cases where misconduct is detected, reported, and sanctioned. There is however currently no better measure of the actual incidence of attorney misconduct or, more importantly, of public protection.[1]

Granted, as California recognizes, attorney misconduct is perhaps a poor proxy for whether bar exam cut scores relate to attorney competency.  But make not mistake.  Bar exam cut scores, by nature of their very arbitrariness, likewise are imperfect proxies for measuring attorney competency.  

Why is this important?  

Well because state supreme courts are now in pitched conversations about whether to adopt the s0-labeled "Next Generation" bar exam.  The chart below suggests caution because it's easy to place confidence in numbers, more confidence that they deserve, like a friend who has betrayed one's trust. As a person trained in mathematics, count me as a skeptic.  Just picking a number out of "thin air," as the range of cut scores suggest, doesn't compute, in my book, as the proper way to judge whether one is competent to practice law.  So as jurisdictions contemplate big changes to a possible next exam format in several years, let's hold them accountability for the math.  Our students and the public at large deserve nothing less. (Scott Johns).

[1] Final Report 2017 Studies, The State Bar of California, available at: https://www.calbar.ca.gov/Portals/0/documents/reports/2017-Final-Bar-Exam-Report.pdf (last accessed Sep. 16, 2021).