The Opaqueness of Bar Passage Data and the Need for Greater Transparency

There has been a great deal of discussion lately over at The Faculty Lounge regarding declines in law school admissions standards, declines in bar passage rates, and the general relationship between LSAT scores and bar passage. Much of this discussion is clouded by the lack of meaningful data regarding bar passage results.  In this blog posting I will delineate several questions that just cannot be answered meaningfully based on the presently available bar passage data.

The national first-time bar passage rate among graduates of ABA-accredited law schools fell significantly in 2014. According to the NCBE’s statistics, the average pass rate from 2007-2013 for July first-time test-takers from ABA-accredited law schools was 83.6%, but fell to 78% in 2014. (2015 data won’t be available until next Spring when it is released by the NCBE.)

While there might be some reasons to believe these results were somewhat aberrational given that the objective criteria of the entering class in 2011 was only modestly less robust than the objective criteria of the entering class in 2010, and given the ExamSoft debacle with the July 2014 bar exam, the results are concerning, given that the objective criteria of the entering classes in 2012, 2013 and 2014 showed continued erosion. As the last two years have seen declines in the median MBE scaled score among those taking the July bar exam, the changes in entering class credentials over time suggest further declines in median MBE scaled scores (and bar passage rates) may be on the horizon.

In 2010, there were roughly 1,800 matriculants nationwide with LSATs of 144 or less. In 2012, there were roughly 2,600 matriculants nationwide with LSATs of 144 or less. In 2014, there were roughly 3,200 matriculants nationwide with LSATs of 144 or less. Recognizing that law school grades will be a better predictor of bar passage than LSAT scores, I think it is safe to say that entering law students with LSATs in this range are more likely than entering law students with higher LSATs to struggle on the bar exam.  Because the number of those entering law school with LSAT scores of 144 or less has grown substantially (particularly as a percentage of the entering class, more than doubling from less than 4% in 2010 to more than 8% in 2014), many are concerned that bar passage rates will continue to decline in the coming years.

While there has been a great deal of discussion regarding declines in admission standards and corresponding declines in bar passage standards, this discussion is profoundly limited because the lack of meaningful bar passage data presently provided by state boards of law examiners and by the ABA and ABA-accredited law schools means that we do not have answers to several important questions that would inform this discussion.

1. What number/percentage of graduates from each law school (and collectively across law schools) sits for the bar exam in July following graduation and in the following February? Phrased differently, what number/percentage of graduates do not take a bar exam in the year following graduation?

This is a profoundly important set of questions as we look at employment outcomes and the number/percentage of graduates employed in full-time, long-term bar passage required positions. Given that only those who pass the bar exam can be in full-time, long-term bar passage required positions, it would be helpful to know the number/percentage of graduates who “sought” eligibility for such positions by taking a bar exam and the number/percentage of graduates who did not seek such eligibility. It also would be helpful to understand whether there are significant variations across law schools in terms of the number of graduates who take a bar exam (or do not take a bar exam) and whether those who do not take a bar exam are distributed throughout the graduating class at a given law school or are concentrated among those at the bottom of the graduating class. At present, however, this information simply is not available.

2. What is the first-time, bar passage rate for graduates from ABA-accredited law schools?

One might think this would be known as ABA-accredited law schools are required to report first-time bar passage results. But the way in which first-time bar passage results are reported makes the data relatively unhelpful. Law schools are not required to report first-time bar passage for all graduates or even for all graduates who took a bar exam. Rather, law schools are only required to report first-time bar passage results for at least 70% of the total number of graduates each year. This means we do not know anything about first-time bar passage results for up to 30% of graduates of a given law school. Across all law schools, reported results account for roughly 84% of graduates, leaving a not insignificant margin of error with respect to estimating bar passage rates.

People would have been flabbergasted if the ABA had required reporting of employment outcomes for only 70% of graduates. Now that the ABA is requiring reporting on employment outcomes for all graduates, there is no good reason why the ABA should not be requiring bar passage accounting for all graduates, requiring law schools to note those who didn't take a bar exam, those who took and passed a bar exam, those who took and failed a bar exam, and those for whom bar status is unknown.  (Up until recently, some boards of law examiners were not reporting results to law schools, but my understanding is that the number of state boards of law examiners not reporting results to law schools is now fairly small.)

Notably, for 2011, 2012, and 2013, the average bar passage rate for first-time takers from all ABA-accredited law schools based on data reported by the law schools was consistently higher than the data reported by NCBE for the corresponding years (2011 – 83.8% v. 82%, 2012 – 81.8% v. 79%, 2013 – 82.4% v. 81%. (Moreover, first-time takers are not measured equivalently by the ABA and by the NCBE. The ABA reporting requirement focuses on graduates who took any bar exam for the first-time. The NCBE defines as first-time takers any person taking a bar exam in a given jurisdiction for the first-time. Thus, the NCBE set of first-time takers is broader, as it includes some people taking a bar exam for the second time (having taken the bar exam in another jurisdiction previously).

3. What is the “ultimate” bar passage rate for graduates from ABA-accredited law schools?

Even though a number of commenters have noted that “ultimate” bar passage is more important than first-time bar passage, there is no publicly available data indicating the ultimate bar passage rate on a law school by law school basis for the graduates of each ABA-accredited law school. What number/percentage of graduates of a given law school who take a bar exam pass after the second attempt? What number/percentage of graduates of a given law school who take a bar exam pass after the third attempt? What number/percentage of graduates of a given law school never pass a bar exam? This information just is not publicly available at present.

While Standard 316, the bar passage accreditation standard, allows schools to meet the standard by demonstrating that 75% or more of those graduates who sat for a bar exam in the five most recent calendar years passed a bar exam, this “ultimate” bar passage data is not publicly disseminated. Thus, while first-time bar passage data is limited and incomplete for the reasons noted above, “ultimate” bar passage data on a law school by law school basis is actually not available.

The modest amount of information available on “ultimate” bar passage rates is not very helpful.  The LSAC National Longitudinal Bar Passage Study contains some analysis of "ultimate" bar passage rates, but it focused on the entering class in the fall of 1991, which it described as being “among the most academically able ever to enter” law school based on entering class statistics (page 14), a description that could not be used with the classes that have entered in the last year or two or three. It also does not contain any information about "ultimate" bar passage for graduates of individual law schools.  In addition, Law School Transparency has recently received some information from at least one law school that has requested anonymity. Much better “ultimate” bar passage information is needed to better inform many of the discussions about the relationship between entering class credentials and bar passage.

4. How can we compare bar passage results from one jurisdiction to another?

Most state boards of law examiners do not present data regarding bar passage that allows reasonable bases for analyzing the results in ways that provide meaningful insight and a meaningful basis for comparison. Fewer than one-third of states publicly provide information in which a delineation is made between first-time takers and repeat takers on a law school by law school basis and only a few of these provide information about MBE scores on a school by school basis. Accordingly, it is very difficult to make meaningful comparisons of year-over-year results in the months following the July bar exam, because data is rarely reported in a consistent manner. The NCBE does provide statistics annually (in the spring) which includes a delineation of bar passage rates by state based on first-time test takers from ABA-accredited schools, but the NCBE does not provide MBE scores on a state by state basis (although it seemingly should be able to do this).

Conclusion

There is a need for much greater transparency in bar passage data from boards of law examiners and from the ABA and ABA-accredited law schools. It well may be that some law schools would be a more meaningful investment for "at-risk" students, those whose entering credentials might suggest they are at risk of failing the bar exam, because those law schools have done a better job of helping "at risk" students learn the law so that they are capable of passing the bar exam at higher rates than graduates of other law schools with comparable numbers of at risk students. It may well be that some jurisdictions provide "at risk" students a greater likelihood of passing the bar exam.  At the moment, however, that information just isn’t available. Much of the disagreement among various commentators about the relationships between admission standards and bar passage rates could be resolved with greater transparency – with the availability of much better data regarding bar passage results.