Sunday, May 5, 2019
- Robert Kuehn (Wash U), Law School Specialty Program Rankings: Is the Tail Wagging the Dog?
"In previous years, the person rating other schools was directed to choose the top 15 programs in that specialty area. U.S. News would then list the top 12 to 20 programs (and even more behind a paywall) based on how often the school was listed by raters. This year, U.S. News asked survey participants to rate specialty programs on a 5-point scale of Outstanding (5) to Marginal (1), mirroring the much-criticized method used to assess the academic reputation of schools among peers and lawyers/judges."
"A glance at the new rankings shows noticeable bunching of schools ranked highest by U.S. News also at the top of most specialty programs and an overpopulation of lowest ranked schools at the bottom. Is this because the highest ranked schools also happen to have the highest quality specialty programs and vice-versa for the lowest ranked schools? Or, might the over-representation at the two ends be due in part to a halo effect where the academic reputation of a school is influencing the rating of that school’s specialty program? This effect could be especially pronounced this year, when raters were asked to not just opine on the quality of 15 specialty programs but to express an informed opinion on 200."
Conclusion: "U.S. News explained that its goal in moving to the new scale was to rank more schools (five to six times more) and provide better understanding of the scope of the differences between schools. Yet when there is strong evidence that many raters may cue off a school’s overall reputation and not independently assess the quality of its specialty program, U.S. News’ rank ordering of school specialty programs down to 192 is of dubious validity and should also call into question the use of this method in its annual law school ranking."
I can only speak about the legal writing rankings, but they look fairly accurate to me. They do not reflect any "halo" effect at all. Take a look for yourself.
In many ways, legal writing profs are a tightly-knit group. They are very aware about what is going on through the annual meetings and the list-serve. They are also good at evaluating which legal writing programs are the top ones.