Is the Bar Exam Necessary?

Nearly every lawyer has a bar exam story.  For some, it was sleepless nights studying.  Others experienced trauma during the preparation period.  I walked under the wrong bird on my way into the exam.  Ask any lawyer, and they will tell a story.  Is the collective experience enough for continued administering the current bar?  The current crisis necessitates asking whether the current bar exam is an appropriate assessment tool.  Brian Tannebaum published this week that the answer is no.  He said the bar should be cancelled forever. (hat tip to James McGrath)

The bar exam debate reached the boiling point this year due to COVID-19.  The 2015 addition of Civil Procedure to the MBE and subsequent crash in national MBE scores raised red flags about the test.  In front of over 100 experts in learning and student achievement, a NCBE representative (current president) ignored and ungracefully punted when questioned about cognitive load theory.  The continued decline in scores caused numerous questions about the MBE.  Now, the COVID-19 crisis could prevent safely administering the bar exam, and state bars must answer the question whether bar exams are necessary for licensure.

The NCBE obviously says the bar exam is critical to licensure.  In a white paper some would describe as insensitive during the current crisis, the NCBE promoted the virtues of the bar exam.  The exam protects the public by ensuring a level of competence by everyone licensed.  You can read their white paper here. 

I am skeptical of the white paper arguments though.  Aside from the obvious financial bias, the white paper conflates bar exams protecting the public with their version of the bar exam protecting the public.  They espouse the MBE's greatness through reliability and validity.  Their argument is an objective MBE is fair, and thus the best assessment.  The argument is persuasive if fairness also related to the practice of law, but unfortunately, I believe the MBE only tangentially relates to competence to practice law. 

For a paper I want to publish, I contacted over a thousand former students and attorneys in OKC to take a simulated MBE provided by a bar review company.  Approximately 20 agreed to take the exam.  The practice experience ranged from 1 year to over 15 years.  Unsurprisingly, 0 (zero) people passed the simulated MBE according to the Oklahoma MBE cut score at the time (135).  A litigator passed the Evidence section, but no one else passed the subject they practiced the most.  ADAs and PDs failed the Crim Law section.  Transaction attorneys failed Contracts.  As attorneys' experience increased, MBE scores decreased.  I had Superlawyers, local award winners, and Superlawyer Rising Stars take the test.  The results were the same for everyone.

The argument the bar exam protects the public assumes the exams tests what a lawyer does for the public.  Unfortunately, the MBE fails to test minimum competency to practice law, which is the threshold for licensure in virtually every state.  It tests competency to answer MBE questions which allude to legal concepts within the questions.  I know the criticisms of my research.  It isn't statistically significant and the test isn't a real MBE.  Finding lawyers to voluntarily take a bar exam isn't an easy task.  However, I fully believe the result would occur on a larger scale.  I encourage bar examiners, Supreme Courts, and anyone advocating for the MBE (including lawyers at the NCBE) to take a simulated MBE.  I encourage the NCBE to make a real version of the MBE available for the test and independent score verification.  If a large majority of participants pass, then the MBE tests minimum competence to practice law.  However, if they don't, then maybe the wisdom from learning experts about cognitive load, spaced repetition/the forgetting curve, and other learning science can be used to create a test that truly assesses the minimum competence to practice law.     

COVID-19 is accelerating an ongoing feud over the bar exam, especially the MBE.  This debate will continue, so I encourage everyone to evaluate the information related to the bar.  We can use this time to either modify or solidify the current licensure process.  Previous generations' experience should not be our guiding principle for an exam.  Even if I am wrong, now is the time for boards, Supreme Courts, and the NCBE to evaluate the way we license attorneys.  Confidence in what we are doing actually protects the public should be paramount.

(Steven Foster)