Thursday, February 25, 2021
This week the Association for Academic Support Educators ("AASE") published Best Practices for Online Bar Exam Administration. AASE President, DeShun Harris, says that the best practices advocate for "procedures that ensure a fairer test for online test takers." The organization, established in 2014, urges state high courts and bar examiners to adopt these procedures. The AASE Bar Advocacy Chair, Marsha Griggs, says "many of the best practices that we identified are things that bar examiners are already doing." Yolonda Sewell, Vice President for Diversity, adds that in addition to the great strides that bar examiners have made in deploying an online exam, we seek to make sure that the online administration does not unfairly disadvantage any bar applicant on the basis of skin tone, race, gender orientation, biophysical conditions, disability, need for test accommodations, or socio-economic resources. The Best Practices are aimed to level the playing field, both among applicants of varied backgrounds, and between the online and in-person versions of the exam."
One of several effects of the COVID-19 pandemic, was that bar examiners and bar applicants questioned the wisdom and feasibility of administering in-person exams in the traditional large group format. In response to COVID-19 limitations, the first online bar examinations in the United States were administered between July and October 2020.
With but a few exceptions, the online exams were remotely proctored using artificial intelligence technology provided by a commercial vendor. As the exam dates approached many issues surfaced surrounding the use of facial recognition software and remote proctoring. One prominent issue was the number of complaints voiced from students who are people of color, asserting that the software did not recognize them. During and after the exam, other complaints sounded, ranging from data breaches, and poor technical support, to "flagging" hundreds or thousands of applicants for alleged cheating or "testing irregularities." At the extreme, some applicants reported having to sit in their own waste—as the exam instructions warned applicants about being out of view of the camera except during scheduled breaks—for fear of failing the exam. Additionally, there were reported issues with the technical delivery, submission, and scoring of the Multistate Performance Test, and jurisdictional scoring errors that wrongly identified applicants who earned passing scores as exam failures, and falsely notifying others who failed the exam that they had passed.
AASE lauds the efforts of bar examiners at the local and national levels for their flexibility and willingness to provide options for remote administration. While we defer to the proven expertise of the test-makers in determining matters related to exam content, scoring, accommodations and character and fitness eligibility, we add our collective expertise in assessment delivery, performance application, and enhancement pedagogies for non-traditional test takers. We recognize that online bar exam delivery will outlive the pandemic and current circumstances. We also believe all who play roles in the process of creating and delivering a bar exam, want the exam to be fair and effective. In light of those dual goals, we think the time is ripe for adoption of additional policies that are more than performative gestures toward a more diverse legal profession.
(Association of Academic Support Educators)
Monday, April 27, 2020
For the times they are a-changin’. -Bob Dylan
The times certainly have changed. Almost overnight, every facet of daily life has transitioned to online delivery. Telehealth and telemedicine are becoming the primary source for doctor-patient interaction during the pandemic. Law school classes are online. College classes are online. K-12 primary education is online. Church and religious services have moved to online formats. My grocery and organic farm-to-table products —gone online. Court hearings, also online. I can buy a car, entirely online. I can have legal documents notarized online.
But I cannot take the bar exam online. At least not yet.
The COVID pandemic has tested our resolve and our ability to utilize available technology. Almost every aspect of the legal profession, from court proceedings and probate administration, to law enforcement and legal education, has mobilized for remote administration. Bar examiners at the state and national levels should hang their heads in shame for not harnessing the available technology to deliver the existing exam remotely. It is an embarrassment of epic proportions that those at the helm of legal licensure are so behind the times that the pipeline for entry to the legal profession could be closed until further notice.
Relentlessly tethered to tradition, those insistent that 2020 law grads take an exam that may not be offered until early 2021 have either dropped the ball or are hiding it. It is fundamentally unfair to require an exam for licensure and at the same time withhold that exam from licensure candidates. The cries for diploma privilege and supervised practice options have sounded around the world. To which bar examiners and high courts have responded with either feigned indifference or a proposed solution that is no more than a band-aid for a gaping wound.
To become attorneys, bar candidates should not have to risk their health or the health of their vulnerable loved ones to the spread of the coronavirus. Even today, there are still more unanswered questions than answers. The majority of U.S. jurisdictions have made no announcement as to whether they will offer an exam in July or not. A number of states have canceled the July exam, but still have not announced definitive information about the date or form of the replacement exam. Candidates across the country remain in the dark as the bar exam becomes an archaic qualifier for competence. If the bar examiners hold fast to the pencil and scantron method of testing, we can expect to see it go the way of the pay phone, the answering machine, and the 8-track tape.
Two states, California and Massachusetts, have alluded to an online exam, but with little detail. It remains unknown what role the National Conference of Bar Examiners (NCBE), who produces the multistate exams used in all states except Louisiana, will play in the online exam. If the NCBE can provide an online exam for two states, why not do so for all UBE jurisdictions? And why make candidates in other states suffer the risk of exposure to COVID or career delays by withholding the online exam?
If the NCBE has not developed an online exam, we must ask “why not” and "where has it been for the last two decades?" And we must not accept “test security concerns” as a viable response. Test security is no less of a concern to law school faculty, and to those who administer admissions exams. Yet all law school exams and the LSAT will be offered online in May 2020. The MPRE (another NCBE exam) and other professional licensing exams are already online.
Whether the bar exam effectively assesses one’s competency to practice law is a reoccurring question that will continue to resurface. At a time when virtually every state, except maybe Utah and Wisconsin, is under fire for indecisiveness and poor communication regarding the fate of would-be July 2020 bar takers, bar examiners are justifiably under scrutiny. As is the bar exam. The future of the exam is in the examiners’ hands. We’ve only to watch and see if they’ll respond like Blockbuster or Netflix.
(Marsha Griggs© 2020)
Saturday, March 28, 2020
There is so much that goes into the making of a bar exam. There are layers of research, accountability, and quality control involved in the drafting of the questions. There is beta testing of the exam content. There is scoring, rescoring, and equating. And there are levels of exam security that rival Area 51. The parties involved range from statisticians to politicians, who cautiously weigh input from the podium, the bar, and the bench. To top it all off, the job of bar examiner – at least at the state level – is a modestly compensated appointment that is held all the while keeping a day job of managing a law practice, or ruling from the bench. Too little appreciation is shown to our almost volunteer bar examiners in times of rest and high passage rates. So, I sincerely and thankfully shout out bar examiners everywhere who discharge an office of such societal importance. And I use the term bar examiners in the collective to include every role, from essay graders to the character and fitness investigators, from the licensure analysts to the admission administrators and honorable members of the board.
Bar examiners have to operate independently and make decisions about scoring and bar admissions that will be unpopular to some. But the examiners must make decisions, and it is the failure or delay in reaching a particularly important decision that has placed examiners under fire across the country. That decision: what about the July 2020 exam?
It is understandable to the legal and lay public that a law license is a privilege not to be indiscriminately awarded. It is equally clear that security protocols must be in place to maintain the integrity of the exam. What is not understandable is how some examiners can fail to make adjustments in the face of the extreme and dire circumstances of the COVID pandemic. In less than two weeks’ time, the nation’s ABA-accredited law schools went entirely online, trained faculty (many with limited technology skills) for online teaching, and adopted pass-fail grading. There is simply no excuse for bar examiners to not be just as creative and as willing to implement emergency protocols for the prospective July 2020 examinees.
This week 1,000+ students, representing all of New York’s law schools, petitioned the New York State Bar Association’s Task Force on the New York Bar Examination for an emergency diploma privilege. Days later, New York canceled the July exam. Adding ambiguity to injury, the exam has been rescheduled to the fall, but no date is provided to examinees who need to make study, travel, and lodging plans for the two-day exam. Are you kidding us? It’s almost like the examiners are not listening. At all.
A reasonably prudent person will interpret the New York decision as a signal for other states to follow. New York is considered highly influential, as its 2016 adoption of UBE was followed by Illinois, Maryland, North Carolina, Ohio, Tennessee, Texas, and others. The 2020 bar takers are not asking the examiners to give away law licenses without merit. They —joined in large number by law faculty, deans and alumni— are asking for necessary emergency licensing measures. They are asking examiners to think outside of the traditional bar exam box. They are asking that fairness, humanity, and the chance to earn a living be prioritized over security worries. They are asking the examiners to listen.
Excerpted from An Epic Fail, Volume 64 Howard Law Journal _____ (2020)(forthcoming).
Monday, January 13, 2020
The Uniform Bar Examination (“UBE”) has juggernauted from an idea to the primary gateway for entry into the practice of law. To the resounding support of law graduates and law schools, a supermajority of states has abandoned individual state law exams for a uniform exam written by a private entity. The UBE is the exam of the future and I anticipate that at least three more states will have adopted the UBE by year end. The UBE remedies many voiced complaints about varying degrees of exam quality and exam difficulty across states. Perhaps the most touted feature of the UBE is score portability.
UBE takers may "port" or transfer their scores into other UBE states, thus, relieving examinees from the arduous chore of having to sit anew for a bar exam. However, the promise of score portability is allusive at best. Transfer procedures vary by state. The fees to transfer one’s UBE score may be as high as $1700, possibly more than the cost of taking the bar exam in the transferring state. For a majority of students who exit law school burdened with student loan debt, these transfer costs will make the promise of portability unrealizable.
According to attorney and bar prep professional Ashley Heidemann, “the UBE is not as portable as law students are led to believe.” Heidemann feels that the promise of portability is highly deceptive to law students who believe that a widespread uniform exam means that once licensed, UBE attorneys will be able to transfer into other states at any time. “The biggest misconception students have,” says Heidemann, “is that UBE scores can be transferred to a different UBE jurisdiction at any time. In reality, UBE scores are only good for generally two to five years, meaning one cannot transfer a score from one state to a different UBE state after their specified time period is over.”
Even staunch supporters of the UBE seem to think that the UBE has not yet reached its greatest potential. UNLV Professor Joan Howarth advocates for a uniform cut score, citing that a six point score differential could effectively exclude hundreds of bar takers from the practice of law. Melissa Hale, Director of Academic Success and Bar Programs at Loyola University Chicago School of Law says, “I’d love to see a more uniform process [regarding admission and transfer policies].” Hale, who sees the UBE as an improvement over predecessor exams and self-identifies as pro-UBE, wants to make sure that students understand the score transfer process and that it is “not without hurdles.”
As more and more states adopt the UBE, academic support professionals will need to stay in the know and keep students informed about the true costs and limitations of score portability. That is — until or unless a uniform cut score becomes a reality. Stay tuned, we may be closer than we think!
 Marsha Griggs, Building a Better Bar Exam, 7 Tex. A&M L. Rev 1 (2019).
 Interview with Ashley Heidemann, President, JD ADVISING LLC (Mar. 25, 2019).
 Joan W. Howarth, The Case for a Uniform Cut Score, 42 J. LEGAL PROF. 69, 72 (2017).
Monday, December 2, 2019
Follow (v): To act according to an instruction or precept; to pay close attention to; to treat as a teacher or guide.
While in law school, I never connected with any of my professors on social media. Let's pretend that's not because social media tools were not yet sufficiently developed to allow me to do so. Fast forward into the information age where I've seen healthy discussions about whether law professors should encourage students to "follow them" on Twitter and other social mediums. Ultimately every professor has the right to their own individual preferences and likewise, their students have the freedom to decide whether and how to interact with their professors online.
Many professors are kind enough to freely spew out words of wisdom as regards exam preparation, and the beauty of Twitter makes these gems available to all. University of North Carolina School of Law Professor O.J. Salinas tweeted some words of wisdom that I wish I had access to as a first-year (or even second-year) law student. Professor Salinas shared:
"Law students (particularly 1Ls): Finals are here. Remember to support your conclusions w/ analysis. Apply the law to the facts of the hypo for every issue you spot. Conclusory answers (conclusions w/out analysis) don’t get you a lot of points (if any). The facts of the hypo are your friends. The facts are there to help nudge you (sometimes quite directly) to your analysis. If you are stuck on the exam and don't know where to go, first take a couple of deeps breaths. Then re-read the call of the question. Then revisit the facts. As you revisit each line of the facts, ask yourself: Why is this fact here? Have I applied this fact to any laws that we have covered in class? Does this fact or could this fact relate to something that we have covered in class?
Finally, make it easy for your prof. to read your exam. Aim for clear & concise writing. Short sentences. Paragraph breaks. Headings/subheadings. Walk the reader through your prediction by providing effective/complete legal analysis. And don't presume your reader knows anything. You can do this!"
I have a list of professors that I follow. Many of whom I know only through online interactions. I am grateful to be able to follow their wisdom and shared experiences. I benefit regularly from our exchanges. My daily takeaways include teaching tips, common struggles, and concise study and writing advice for my students. Thanks Professor Salinas for your exam writing wisdom. I remain a follower.
Monday, November 4, 2019
Logically it makes no sense that, in today’s world, failing at something because you tried will tarnish you with a negative social label. . . . [T]o continue evolving, the stigma associated with failure has to be shaken off and be replaced with positive personal development. When you fail at something, hopefully you can recogni[z]e why and where you failed, so that next time you can move forward accordingly. – C. Montcrieff
Bar takers in all but one state have received results from the July 2019 bar exam. Although California examinees may have to wait another week for results, with increased MBE scores reported nationally, bar passage rates (overall) are deliciously higher than recent past exams. What better way to transition to the semester wind down than with news of newly licensed attorneys joining the ranks of your alumni rosters!
I am elated and overjoyed for my students who find their names on the bar pass list. I understand the sacrifice, the grit, the fear, the pressure, the exhaustion, and the anxiety that are necessary conditions precedent to bar passage. I actually get teary-eyed as I scroll through the social media feeds of newly minted attorneys that contain expressions of joy and gratitude for the obstacles they overcame and support they received.
My joy is tempered by the heartache I feel for those who fought so valiantly and fell short of the state cut score. It never ceases to amaze me how a day that brings elation can, at the same time, end in devastation. Those of us doing ASP work must manage that range of emotions altogether in the same day. We collect data and publish articles on interventions that lead to bar success in licensure candidates with known failure indicators. We are experientially trained to manage bad news and to earnestly encourage unsuccessful students to try anew. But how does the reality of our calling square with the purpose of our profession?
We must examine the role and reality of stigma in bar exam failure and determine where, how, and if, it fits into the notion that diversity in the legal profession is not solely about racial and socio-economic inclusion. The diversity promoted by effective academic support programs includes intellectual disparities, physical and emotional disabilities, linguistic variations, and learning differences.
The definition of academic and bar success is changing. Success for some may be sitting through a two-day exam without the testing accommodations relied upon during law school. For others, it can be completing an exam scribed in a language other than the test-taker's native tongue. For many bar takers who graduated in the bottom quartile of their law school classes and/or with low entering LSAT scores, success may be coming within 5-10 points of a passing score, that all published statistics said that they could not achieve.
I dare not suggest that legal educators dismiss or ignore bar failure, but I challenge the status quo about how we frame bar failure as part of professional identity formation. Moved by the MacCrate Report, law teachers have become more intentional about teaching, and have begun to support law students’ professional identity formation inside and outside of the classroom.1 I see no reason for that support to end with the bar examination. As we normalize struggle2, we must communicate bar failure as a temporary status and not as an indelible component of one’s professional identity.
1 Susan L. Brooks, Fostering Wholehearted Lawyers: Practical Guidance for Supporting Law Students' Professional Identity Formation 14 U. ST. THOMAS L.J. 377 (2018).
2 Catherine Martin Christopher, Normalizing Struggle, ___ Arkansas L. Rev. ___ (2019).
Tuesday, September 24, 2019
Last year, one of my international students brought to me a response she had written to a mid-tern exam question. She was wholly perplexed, because the professor had given her a low score on this particular response, and yet, even in looking at the notes the professor had written on her paper, she could not fathom where she had gone wrong. Bizarrely, the more the two of us discussed her essay, the more confused I became about why she had written what she had written. Finally, and wholly by accident, I stumbled across the source of the trouble. At one point the exam question referred to someone being "served", and my student had not recognized this usage as being connected with "service of process". The latter term she understood, but she read the off-hand and abbreviated statement that "X was served" as some form of hospitality, not legal action. ("Have some tea!") This was partly because English was her second language, and undoubtedly also partly because she did not grow up watching movie and TV shows in which frumpy anonymous operatives walk up to the protagonists, slap envelopes against their chests, and say, "You've been served!" For much of our discussion, it had not even occurred to me that this could be a source of confusion, and of course there was no way the student could have known it herself.
I thought about this episode last week, when I was attending a conference hosted by the NCBE, in which some of the presenters were discussing the ongoing evolution of the development of MBE and MEE questions. Part of that evolution includes the elimination, or at least minimization, of the use of terms whose meaning was not tied to the practice of law and might not be recognized by all of the examinees. An example given involved a torts question involving a car that had been damaged in a collision. In the original question, the defendant was identified as "Union Pacific", and it was apparent that the rest of the question was written with the assumption that examinees would recognize Union Pacific as a company that operated railroads, and that therefore the collision under consideration was between a car and a locomotive. The newer, improved version of the question simply referred to the defendant as "a railroad company", thus providing the information needed for proper analysis to all examinees.
Discussion at that point livened up a bit, as presenters and participants brainstormed about other terminology that question writers should considered changing in order to make their questions more accessible. These tended to fall into a few categories:
- References to people, businesses, locations -- generally, things that could be identified with proper nouns -- that might be recognized by some people (but not all people) as possessing some characteristic relevant to the legal analysis. For example, a question that named Gregory Hines as a plaintiff in a case in which his feet were injured might reflect the expectation that examinees would recognize Hines was famously a dancer, and that therefore a foot injury might generate greater damages to him than to an average person. A question that mentions "Reno" might rest on the assumption that everyone knows Reno is in Nevada and gambling is legal there.
- References to technology, fads, or news items from two or more decades ago that most of us who were alive and adult at that time would instantly recognize, but the significance of which might be totally lost on people currently in their 20s. A question that depends on the operation of an answering machine or the effect of a slap bracelet may only be accessible to a portion of the testing population.
- Specialized terms for everyday objects that nevertheless are not commonly used in conversation. A question that depends on knowing the difference between a banister and a balustrade, or between a lintel and a gable, is probably going to lose a portion of the examinees.
It can be hard, when writing exam questions or practice questions, to resist the temptation to make a clever reference or to give examinees the chance for a moment of recognition. But our tests are not supposed to be tests of any vocabulary but legal vocabulary. If an examinee misses the opportunity to demonstrate that he knows the appropriate rule, and can apply it skillful to relevant facts, because he did not have access to the full meaning of the fact pattern so that he could recognize the issue that leads to that rule, then the examinee has been unfairly denied a chance to shine.
Sunday, July 28, 2019
Social media timelines are aflutter since the California Bar Examiners released, days early, the question order and subjects for the July written exam. After someone “inadvertently transmitted” test information to “a number of deans of law schools,” the CA examiners disclosed the same information to all registered July 2019 California bar takers. The internet remains undefeated and the information now hovers in the public domain accessible to us all for comment and critique. The CaliLeaks, as I refer to them, sent ripples of shock, resentment, and gratitude throughout the community of future, past, and present bar takers.
Dear California Bar Examiners, you did the right thing. You responded to a mistaken disclosure by disseminating the same information to all bar takers, to prevent any actual or perceived unfair advantage. You made a mistake and you owned it. There is a lesson in every mistake and I hope that other bar examiners, and especially the NCBE, with its foot on the jugular of all but a few states, will learn from yours.
In an ideal scenario, the premature and selective leak of confidential information to some law deans would not have occurred. No student should be disadvantaged in terms of familiarity with the exam content, inside knowledge, or the opportunity to pass. We now know the identities and school affiliation of the receiving deans. I am naive enough to believe that respected academic leaders would not compromise the integrity of the bar exam by sharing confidential information about its content. I am also cynical enough to recognize the good reason of those who question whether bar takers from some schools may have received information days before bar takers from other schools. Notwithstanding the many unanswered questions, California's disclosure (the one to all of its bar takers) is something that could have and should have happened long ago.
For goodness sake, the bar exam is based, at least in theory, on fundamental legal principles learned in law school. Knowing the general subject area to be tested is not a dead giveaway to the question content. Bar examiners in Texas have provided general subject matter information for decades. It is a preposterous notion that knowing the subjects that will be tested will lead to a flood of unqualified lawyers. Consider the law school final exam as the loosest conceivable model. Law students know to expect Property questions on their Property final exam, but it still leaves them to their own devices to prudently review the full scope of course coverage from possessory estates and future interests, to conveyances, recording acts, and landlord-tenant rules. Disclosure of the tested question areas should not be Monday morning tea, instead it should be the norm in bar examination. Telling would-be lawyers what they need to know to be deemed competent to practice law isn’t a blunder or a gracious act. It is the right thing to do.
I challenge any lawyer, law student, or law professor to imagine the futility and frustration of completing a full semester of required first-year courses, spending weeks preparing for final exams, and then not learning until the beginning day of final exams which courses will be tested and which will not. As unthinkable as this notion may be, this precisely describes the current practice of bar examination in most states and under the UBE. Time will tell if California’s leak leads to a more reasonable exam process and to less arbitrary bar failure rates. If it does, then others should follow suit. We need a better bar exam and California’s error could be an accidental step in the right direction.
Thursday, February 7, 2019
Recently, I heard a discussion suggesting that bar passers do things differently in the final two weeks than those who are not successful on the bar exam. That got me thinking about what I've been seeing, at least anecdotally, in my 10-plus years working with students in preparing for their bar exams.
First, both groups tend to work extraordinarily hard in the last two weeks before their bar exams. So, what's the difference? It must be in the type of work that the two groups are doing. In short, during the final two weeks, it seems to me that bar passers tend to ramp up their practice with lots and lots of MBE questions and essays [while also creating super-short compact homespun study tools (2-3-page outlines, flashcards, or posters)]. In contrast, people who find themselves unsuccessful tend to focus on creating extra-bulky study tools and trying to memorize those study tools with very little continued practice of MBE questions and essays. In brief, one group is continuing to practice for the exam and the other group is focused on memorizing for the exam.
But, here's the rub:
It’s a perfectly natural feeling during the final two weeks of bar prep to want to focus solely (or mostly) on creating perfect study tools and trying to perfectly memorize all the law.
But, according to the educational psychologists, there’s something called “useful forgetfulness.” You see, when we jam packet our study tools with everything, we aren’t learning much of anything because we haven’t had to make any hard decisions about what to let go (what to “forget”). We’re just typing or handwriting or flowcharting like a scribe. But, when we purposefully decide that we are only going to make a super-short “starter” study tools (knowing that we can always add more rules as we work through more questions during the next couple of weeks), our decisions about what to put in our super-short study tools (and what to leave out) means that we actually empower ourselves to know both what we put in our study tools (and what we left out).
As a suggestion, tackle two subjects per day – one subject that is essay-only and one subject tested on both the essay and the MBE exam. Starting with one subject in the morning, using the most compact outline that your commercial course provides (and referencing the table of contents for each subject), create a super-short study tool with the goal of completing your study tool in 2 hours or less.
Here’s a tip:
If you think that you need a rule, don’t put it in because you can always add more later. Instead, only add a rule that you’ve seen countless times over and over. Just get it done. Move quickly. Don’t get stuck with definitions of elements, etc. Stick with the big picture umbrella rules. Think BIG picture. For example, be determined to get through all of contracts in 2 hours (from what law governs to remedies). As a suggestion, have just one rule for each item in the table of contents for your commercial bar review outline. Don't go deep sea diving. Stay on the surface. Then, in the remainder of the morning, work with your study tool through a handful of practice essays. In the afternoon, repeat the same tasks using a different subject (creating a snappy study tool and working through a few essays). Finally, in the evening, work through mixed sets of MBE questions.
In the last week before the bar exam, with most of your starter study tools completed, focus on talking through your study tool (for about one hour or so) and then working through lots and lots essay problems and MBE questions. As you practice in the last week, feel free to add rules that come up in practice essays and MBE questions to your study tool. As I heard one person explain it, your study tool becomes sort of a "bar diary" of your adventurous travels through essays and MBE questions (thanks Prof. Micah Yarbrough!). In short, you've created a study tool that has been time-tested and polished through the hard knock experiences of working and learning through lots of bar exam hypothetical problems.
So, for those of you taking the February 2019 bar exam, focus on practice first and foremost because you aren't going to be tested on your study tool. Rather, you're going to be testing on whether you can use your study tool to solve hypothetical problems. And, good luck on your bar exam! (Scott Johns).
P.S. For those taking the Uniform Bar Exam, there are 12 subjects as grouped by the bar examiners (I think there are 14 subjects in California, depending on how you count subjects):
* Business Associations (Corporations, Agency, Partnership, and LLC)
* Secured Transactions
* Federal Civil Procedure
* Family Law
* Wills & Trusts
* Conflicts of Law
* Constitutional Law
* Criminal Law & Procedure
Sunday, December 2, 2018
Thank you to Sandra L. Simpson, Co-Director of the Institute for Law Teaching and Learning at Gonzaga, for her email about a post written by Lindsey Gustafson of University of Arkansas Little Rock on the ILTL pages reviewing a 2016 article by Elizabeth Ruiz Frost entitled "Feedback Distortion: The Shortcomings of Model Answers as Formative Feedback." The review can be found here: Article Review. (Amy Jarmon)
Tuesday, November 13, 2018
I have learned probably hundreds of tips, tricks, and techniques to improve one's performance on examinations. But there is only one that I learned with ten million people watching.
In 2005, I took the Florida Bar Exam -- my second bar exam, after passing the DC Bar Exam seven years earlier. When I returned to my car, the lone message waiting for me on my cell phone was not the expected call from my family. Instead, it was Glenn, from Culver City, California, calling to inform me that I had been selected to be a contestant on Jeopardy! -- the fast-paced quiz show in which contestants vie to answer 61 questions in 22 minutes.
The taping was to be in a month, and so I went right from cramming for the bar to cramming for trivial warfare. I knew there was no way I could study every possible subject that might come up on the show. At the same time, I felt like I ought to be "training". Today, there are websites that archive years of Jeopardy! clues, and old episodes on demand on Netflix, but these weren't available in 2005, so my main source of practice was watching the daily broadcast of the show at 7:30 p.m. And, perhaps because I felt that it was a rather precious resource, I decided that I wasn't just going to casually sit on the couch and shout out responses with the contestants. I decided that I was going to act like a contestant. Each contestant stands behind a podium and holds in one hand a pen-sized electronic button, and the first person to press that button after host Alex Trebek finishes reading the clue gets the chance to give the response -- famously, in the form of a question (e.g., "Who is George Washington?"). So, for a month, I tried to simulate their actions. I watched the show standing up, behind a living room chair. I held a clickable ballpoint pen, and practiced pressing the top button after Trebek finished reading each clue, and only then did I allow myself to call out a response in the form of a question. From time to time, I would feel a little goofy doing this, thinking, Isn't the show really about what you know? But I kept at it, because it seemed like the only way to really practice.
Finally, I arrived in California for the taping. Jeopardy! tapes five episodes in one day, a couple days every few weeks, so on the day on which I was scheduled to tape, I was herded into the studio with about a dozen other contestants. We spent a few hours signing documents and having make-up applied and learning all the rules and, most important and exciting, playing a few practice rounds on the set to familiarize ourselves with the equipment. I noticed some of the other contestants -- all clearly bright and as delighted as I was to be there -- seemed slightly awkward behind the podium. We all knew intellectually what to do, of course; we had all been fans watching the show for years, and we had just received a thorough briefing on what was expected of us. Even so, some contestants struggled to push their electronic button at the right time -- pushing it before Trebek was done talking would lock you out so that you could not answer, but if you waited too long, someone else would get in before you. Others got the hang of the button, with concentration, but then could not remember the responses they were trying to give. And there were times when contestants would press the button correctly, and give the right response, but forget to give it in the form of a question.
But when I went up on stage to practice, it was like I was standing back in my living room. I had practiced the timing of pushing my pen button so many times that, when it came time to press the real thing, I did not even have to think about it. I rang in quickly, focused entirely on recalling the information needed, and then gave the answer automatically in the form of a question. It worked in practice, and it worked in the actual taping. Yes, the show is about what you know, but it's important that nothing hinder you from demonstrating what you know. I won four games, and eventually came back to be a finalist in the Tournament of Champions.
In the years since, I have learned that what I had stumbled onto is known as "simulation training". It is a kind of practice that is not unlike the physical training that athletes do to develop muscle memory and automatic responses. In the context of quiz shows and law examinations, though, what makes simulation training particularly useful is not just the physical skills that it develops. What makes it useful is that it frees up mental space and focus for more complex thought. Not having to think about when to push the button and how to phrase my answer enabled me to devote full attention to reading the clue and retrieving the correct response.
Practicing to take examinations -- whether final exams or Bar exams -- can provide the same kind of simulation training, under the right conditions. Of course, students should write practice exams for other very good reasons, like improving legal analysis and uncovering weaknesses in subject matter knowledge, because law examinations should also be about what you know. But there is an added benefit when practice exams are done under conditions that imitate expected exam conditions. There are dozens of details and stimuli that students encounter consistently during an actual exam that, if unfamiliar, can demand valuable thought or cause detrimental distraction: dressing comfortably, locating a seat, timing bathroom use, logging into ExamSoft, calculating timing targets, contending with silence or noise, reading and following directions, cutting and pasting text, properly submitting responses, etc. Encouraging students to incorporate attention to these elements during their practice work, even when they are not really necessary, can help them improve performance, not because performance depends on finding a proper seat, but because being able to do so with almost no thought allows them to devote their mental energies to the tasks that really need them. Exam performance is about what you know, but it is important that nothing hinder you from demonstrating what you know.
Tuesday, October 30, 2018
To lawyers, law students, and professors, the IRAC formula is as commonplace a tool as yellow highlighters or The Blue Book. Some may tout or prefer one of its dozens of variations, particularly in specific situations, but at heart, they all do the same basic job of providing a reliable structure for building an argument. It may take some time for students to internalize that structure and use it consistently. Once they do, however, some students lean on it heavily, as a way of making sure all the expected components of their analysis (Issue, Rule, Application, Conclusion) are included. Other students may see it with more anxiety, as a set of expectations imposed by certain professors; they may worry that if they don't use IRAC, they won't receive full credit in their essay responses.
In either case, students can sometimes be stymied when trying to adhere to IRAC format in an essay test response that requires multiple pieces of analysis, like a rule with multiple elements. For example, trying to fit a discussion of a negligence claim into one big IRAC paragraph -- as some students may feel they are required to do -- may start off well, as the student correctly identifies the question of negligence as the issue and the requirement to show duty, breach, causation, and damages as the rule. But then the application section may become messy, as the student tries to write about each element. If more than one element depends on tricky or subtle facts, or if there are multiple arguments and counterarguments to some elements, then the student may struggle to control multiple threads of analysis, without additional structure, in an enormous paragraph that spreads over two or three pages. The student may lose some of those threads, and so might the reader.
This is an unsurprising consequence of the emphasis on sticking to an overall IRAC format: students, for comfort or consistency, might feel compelled to turn every argument into a unitary IRAC. This may be less of a problem for long-term projects, like a legal research and writing memo, where a student may be given more instruction about formatting and will have opportunities to rewrite and edit their essays. But on a timed assignment, like a final exam, the urge to create one big IRAC argument -- or the fear of not doing so -- can slow students down and inhibit clarity.
One way to help students improve their relationships with IRAC is to point out that a well-reasoned argument can have layers of IRACs built into it. The Application portion, after all, is where the meat of the analysis appears, and if that analysis requires that the student examine multiple elements, each element could be discussed in its own separate sub-IRAC paragraph. To use the negligence example:
Issue: Negligence claim
Rule: Duty, Breach, Causation, Damages
Rule1: [e.g., Obligation to act as reasonably prudent person under circumstances]
Application1: [Application of rule to specific facts]
Conclusion1 re: Duty
Conclusion2 re: Breach
Conclusion re: Negligence claim
This layering of IRACs allows students to take advantage of the order imposed by the format, while still providing the flexibility to address separate sub-issues separately. Theoretically, the layering could continue indefinitely, if certain elements have sub-elements to consider:
Rule3: Actual cause and Proximate cause
Issue3A: Actual cause
Conclusion3A re: Actual cause
Issue3B: Proximate cause
Conclusion3B re: Proximate cause
Conclusion3 re: Causation
This layering of IRACs may not always be the most artful way to organize a legal discussion, but in an exam situation in which students are trying to maximize speed, completeness, and clarity simultaneously, it can provide an efficient way for them to put together a complex analysis.
Thursday, September 20, 2018
According to the American Bar Association (ABA), citing to Law.com and TaxProfBlog editor Dean Paul Caron, the national average score on the MBE multiple-choice portion of the July bar exam dropped to its lowest level in 34 years. http://www.abajournal.com; https://www.law.com; http://taxprof.typepad.com. The National Conference of Bar Examiners (NCBE) reports that the July 2018 MBE average score was just 139.5, while for the July 1984 exam, Law.com reports that the MBE average score was likewise low at 139.21. http://www.ncbex.org/news; https://www.law.com.
In an article by Law.com, the President of the NCBE - Judith Gundersen - is quoted as saying that "they [this summer's lower MBE scores] are what would be expected given the number of applicants and LSAT 25th percentile means of the 2015 entering class." https://www.law.com. In other words, according to the NCBE, this summer's low score average is the result of law school admissions decisions based on the NCBE's appraisal of 25 percentile LSAT data for entering 2015 law students.
Nevertheless, despite the NCBE's claim, which was previously theorized by the NCBE back in 2015 (namely, that bar exam declines are related to LSAT declines), previous empirical research found a lack of empirical support for the NCBE's LSAT claim, albeit limited to one jurisdiction, one law school's population, and admittedly not updated to reflect this summer's bar exam results. Testing the Testers.
As an armchair statistician with a mathematics background, I am leery of one-size-fits-all empirical claims. Life is complex and learning is nuanced. Conceivably, there are many factors at play that might account for bar exam results in particular cases, with many factors not ascribable to pure mathematical calculus, such as the leaking roof in the middle of the first day of the Colorado bar exam. http://www.abajournal.com/news/article/ceiling_leaks_pause_colorado_bar_exam.
Here's just a few possible considerations:
• The increase to 25 experimental questions embedded within the set of 200 MBE multiple-choice questions (in comparison to previous test versions with only 10 experimental questions embedded).
• The addition of Federal Civil Procedure as a relatively recent MBE subject to the MBE's panoply of subjects tested.
• The apparent rising incidences of anxiety, depression, and learning disabilities found within law school populations and graduates.
• The economic barriers to securing bar exam testing accommodations despite longitudinal evidence of law school testing accommodations.
• The influence of social media, the internet age, and smart phones in impacting the learning environment.
• The difficulty in equating previous versions of bar exams with current versions of bar exams given changes in the exam instrument itself and the scope of subject matter tested.
• The relationship among experiential learning, doctrinal, and legal writing courses and bar exam outcomes.
Consequently, in my opinion, there's a great need (and a great opportunity) for law schools to collaborate with bar examiners to hypothesize, research, and evaluate what's really going on with the bar exam. It might be the LSAT, as the NCBE claims. But, most problems in life are much more complicated. So, as a visual jumpstart to help law schools and bar examiners brainstorm possible solutions, here's a handy chart depicting the overall downward trend with respect to the past ten years of national MBE average scores. (Scott Johns).
September 20, 2018 in Bar Exam Issues, Bar Exam Preparation, Bar Exams, Encouragement & Inspiration, Exams - Studying, Exams - Theory, Stress & Anxiety, Study Tips - General | Permalink | Comments (0)
Thursday, June 28, 2018
It's sweltering in much of the USA. And, the heat is only getting hotter for the many recent law school grads preparing for next month's bar exam.
So, I thought I'd offer a few "hot" tips on how to enhance one's learning this summer based on a recently published study entitled: "Smarter Law School Habits: An Empirical Analysis of Law Learning Strategies and Relationship with LGPA," by Jennifer Cooper, adjunct professor at Tulane University, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3004988
As detailed in the article statistically analyzing study tactics and learning, Professor Cooper found that two particular study strategies are positively correlated with law school grades.
The first is elaboration, i.e, explaining confusing concepts to others. So, be a talker this summer as you prepare for your bar exam. In short, be a teacher...be your teacher!
The second is the use of practice questions to learn. So, grab hold of every opportunity you have this summer to learn by doing. Take every mock bar exam you can. Work through every bar exam practice problem available. Be tenacious in your practice. Learn by doing!
Finally, as documented by Professor Cooper, beware of reading and re-reading. It might make you feel like you are learning, but there is little learning going on...until you put down the book and start working on problems for yourself. And, that particularly makes sense with the bar exam...because...the bar exam is testing the "practice of law" not the "theory behind the law."
So, throughout this summer, focus less on reading and more on active learning - through lots and lots of practice problems and self-taught elaboration to explain the legal principles and concepts - as you prepare for success on your bar exam next month. (Scott Johns).
Monday, April 9, 2018
Routines are critical for me to get anything done in a day. I wake up at the same time every morning. I hit snooze 1 time, read my daily devotional after the next alarm, then start my shower routine. I turn the coffee pot on at the same time, grab breakfast, and have “shoe race” with my kids before driving them to school on the same route. The days I follow a solid routine at work with to-do lists, I am more focused and accomplish more. Sound familiar?
My routine and habits help me get through law school and overcome struggles. I knew what I planned to accomplish and finished my tasks even when life was difficult. I tell students every semester that having a routine makes doing additional MBE questions in face of failure, navigating life circumstances, and accomplishing anything else much easier, especially when confronting obstacles during studying. However, I didn’t know much about the research on habit formation until recently. The research could help all of us working with students.
I started listening to The Power of Habit by Charles Duhigg recently while driving, and so far, I love the book. It is a great combination of explaining habit research and providing anecdotal stories of how the research worked in particular situations ranging from large corporations to individuals. I plan to purchase a desk copy to highlight and take notes.
Law students could benefit from the research. The early parts of the book discuss creating and modifying habits. People have cues and rewards for situations, and changing the routine or response to the cue while still receiving the reward helps habit formation or modification. I am already thinking about how I can teach specific responses to certain cues to help 1Ls build habits for law school and reinforce the habits right before the bar exam. Individual meetings may be the best way to inculcate routines, but I am also thinking about how I could integrate the information into my classes.
The section I am listening to right now is about willpower. Research indicates people can increase willpower, and small gains in willpower in one area of life can spillover to other areas. The willpower discussion overlaps with Angela Duckworth’s Grit research. The book indicates willpower can be built with pre-programmed responses to challenging circumstances, which creates routines. Starbucks receives high customer service reviews because they developed training programs for routine responses. Employees use a specific tactic when rude or angry customers come to the counter. Even if an employee is tired, upset, or life is going poorly, the pre-programmed response provides the willpower to help the customer in spite of the rudeness. Response routines can drastically improve willpower.
Students need pre-programmed responses to challenges. Many of us encounter students who dislike professor feedback on assignments, perform poorly on oral questions, or fail another set of MBE questions. Telling students to overcome the obstacle and not worry about the performance may be true but probably not specific enough to help. Helping students determine a clear roadmap for the response is what will help the next time. When faring poorly on the MBE, help them come up with a routine, which could include decompression, analysis, positive response, and another set of questions. We all know it is easy to continue when everything is going well. Responses planned before challenging events are more likely to help overcome those events. Just as lawyers do, plan for the worst.
I can’t wait to finish the book. I encourage everyone to listen or read it if you get a chance.
Tuesday, January 23, 2018
As you review your fall semester exams and set goals for the spring semester, ask yourself: Does my exam look like the work of someone who should be given a license to practice law? When you graduate, you are likely to accept a position with either a plaintiff’s firm, prosecutor’s office, a defense firm, or the court. And, you may find yourself switching from one side to the other throughout your legal career. Therefore, you have to convince your professor that you are capable of handling the legal matter from any chair in the courtroom. The good news is that a well-constructed IRAC answer does just that!
The Rules Section
Q. Where do the jurors learn all of the legal rules and instructions during a trial?
A. From the judge, especially during the final jury instructions.
You have to show the professor that you know and understand the rules of law associated with the litigation. At a real trial, it will be your job as the judge to properly instruct the jury as to the law. Similarly, if you are one of the litigants, then you must protect your client’s interests by ensuring that the judge properly carries out his or her duty. Therefore, both the judge and the litigants need to be well-versed on the legal standard. Show the professor that you know the rules that are related to the specific dispute. Just as a Judge wouldn’t burden a jury with unnecessary rules that don’t relate to the instant case, you too, should be selective in the rules offered to your professor. Include enough rules to help the jury/professor understand the legal standard, while omitting any unnecessary or irrelevant bits of information.
The Application Section
Q. Where do the jurors get all of their facts during a trial?
A. From the witnesses who testify and the exhibits offered into evidence during the litigation.
Q. Whose job is it to tie everything together, i.e. convince the jury that the facts offered at trial actually amount to a crime or tort or breach of contract, under the law as explained by the judge?
A. From the lawyer representing the injured party.
This section shows the professor that you can argue on behalf of your client. The professor essentially gives you a trial transcript when they draft a fact pattern or hypothetical for the exam. It is your job, as the attorney, to dissect that transcript, identifying each piece of helpful (or hurtful) testimony. Then you must compile all of those facts into a thorough and thoughtful argument on behalf of your client, making sure to discuss each of the legal factors or elements important to your case. Just like all good attorneys, you will likely spend more time discussing the hotly disputed elements, while quickly dispensing with the more obvious ones.
Would you ever stand up in a jury trial and say “Clearly the defendant intended to kill the victim,” and then simply sit back down? Of course not! So, don’t make that type of conclusory argument on an exam either. Instead, take the professor step-by-step through each fact or witnesses or piece of documentary evidence that supports your argument, just like you would during a real trial.
If you follow these litigation techniques when you draft your spring exam answers, you’re bound to get a verdict in your favor! (Kirsha Trychta)
Monday, October 30, 2017
I mentioned last week that 1Ls are likely starting to think hard about outlining for their podium courses. With the end of October approaching, students need to focus some of their precious time on preparing for their final exams. It takes a while for some students to shift their focus. But, those students who take time to prepare for final exams may often feel more confident and less stressed come the end of the semester. And a more confident and less stressed student may be better able to focus and demonstrate to the professor what he/she knows about the doctrinal subject come December.
One way students can to start feeling more confident and less stressed is by organizing their class notes around big picture rules in an outline. Students can insert into the outline various hypotheticals that test these big picture rules. The professor in the Socratic class could have generated these hypotheticals. They could also be pulled from other sources, like law school study aids or from the casebooks’ Notes and Decisions. Or, better yet, students can try to generate the hypotheticals on their own.
An outline can take many shapes or forms. What’s important is that each student focuses on what helps him/her best understand the material. What’s also important is that students try to create their outlines on their own. It’s cliché—but, a huge part of the learning process is synthesizing all the materials that each student has available to him/her and putting it down in the outline. Working with the materials and thinking about how and why the materials fit into the doctrinal course can help solidify or create a better understanding of the material. And who doesn’t want a better understanding of the material before finals? (OJ Salinas)
Tuesday, October 3, 2017
For my fourth and penultimate pumpkin post, I recommend that novices regularly remind themselves of their original goal.
In Lesson #2 I explained how I came to establish my personal goal for the growing season, namely to get a single healthy pumpkin to the weigh-off scale. Now that the weigh-off is less than two weeks away, I'm starting to secretly hope I win the Rookie of the Year award. Consciously I know that I have not done as much work as some other new growers, but that fact hasn't stopped me from wanting to win the award. My original goal was to grow a pumpkin, and I've done just that. Yet, I have the distinct feeling that I'm going to be (irrationally) disappointed with my ranking on the leaderboard at the weigh-off event.
All this ranking-focused-thinking got me wondering about my students and their first-year grades. At the beginning of the semester, I asked my criminal law class (which, by design, also includes my AEP students) to tell me what they most wanted out of the fall semester. The top two responses: "to survive" and "to pass." Only a handful of students offered more specific aspirations, like learning about murder, gaining confidence, performing pro bono work, or learning "how to write." It seems that most of my rookie students and I had the same mindset at the outset of our respective endeavors: to survive the new experience. So, does that mean that most of my students will start dreaming of sitting at the top of the leaderboard in December, even though that wasn't their original goal, and even though they may not have put forth the amount of effort needed to achieve a high ranking?
Admittedly, I don't have an answer; rather I'm making an observation about novices. Nonetheless, I do plan to discuss the theory with my students. I suspect that a candid discussion about my own illogical (and last minute) desire to be "the best" may help reframe my students' thoughts and expectations with regard to their own fall grades. In a school with a mandatory grade curve, there can only be handful of "A"s in each class. But, a law school "B" can be equally worthy of celebration--especially if the original goal was just "to survive." (Kirsha Trychta)
Here is "Presley" in mid-September, weighing an estimated 400 pounds.
Monday, October 2, 2017
I mentioned last week that students don’t have to wait until final exams at the end of the semester to find out whether they have a good understanding of what their doctrinal professors are teaching. Since most law school classes don’t have traditional periodic tests, I encouraged students to use their professors’ various “what ifs” and “how abouts” to test their understanding of key rules and concepts that the professors are covering in class.
Students: If you are able to answer the professors’ hypotheticals—whether out loud or in your head—you are positioning yourself well to answer the professors’ hypotheticals on their final exams.
A final exam is often just a mixture of a bunch of hypotheticals in one or two large stories. The hypotheticals test your recollection and understanding of key rules that you have covered throughout the semester. The hypotheticals also test your ability to identify and apply significant facts within the hypotheticals to your key rules. This application of law to facts is legal analysis. The better your legal analysis is on a final exam, the more likely you will get a better grade.
But, I know the Socratic class can often be an intimidating and difficult experience, particularly for many 1L students. I know it is not easy sitting in a Socratic class worrying about getting called on—I’ve been there, and I didn’t particularly like it. I disliked the Socratic class so much that I wanted to quit law school after my first year (That story is for another blog post; but you can read a little more about my law school experience here.)
I feared speaking up in the Socratic class because I didn’t want to be seen as incompetent. I worried too much about what my professors or my peers might have thought about me during that moment right after the professor called my name in class. I worried about getting the professor's question wrong. I worried about appearing nervous. I worried.
It took me a long while to adjust to the type of teaching in the Socratic class. It took me a long while to realize that it didn't matter if I was nervous or got a question wrong--what mattered was how I did on the final exams.
So, I wanted to do what I could to prepare for the final exams. I tried to do a lot of preparation outside of class. I read my cases. But, I also used study aids to help give me context for what I was reading. The study aids also provided me with a bunch of hypotheticals where I could practice my legal analysis.
I practiced my legal analysis within the confines of my safe apartment where I didn’t have to worry about others “judging” me if my voice cracked or was shaky or when I didn’t answer a question correctly. I trained myself on issue spotting and applying law to facts so that I could feel more confident not only in the Socratic class, but on the final exams as well. And things turned out okay for me. The guy who wanted to quit law school after his 1L year is now teaching in a law school.
It’s funny how things turn out. And things can turn out well for you, too. Try to engage with your professors’ hypotheticals. If you are not fully able to engage with the hypotheticals in class, look for ways to engage with hypotheticals outside of the potentially intimidating classroom. Like anything in life, the more you practice, the better you will get. And you have an entire semester to practice for your big day (and it won't matter on that big day whether your voiced ever cracked in class or whether you got a question wrong when the professor called on you). (OJ Salinas)
October 2, 2017 in Advice, Diversity Issues, Encouragement & Inspiration, Exams - Studying, Exams - Theory, Learning Styles, Miscellany, Stress & Anxiety, Study Tips - General | Permalink | Comments (0)
Monday, September 25, 2017
We are several weeks into the Fall semester. 1L students are starting to get a little better handle on what law school is all about. If they didn’t know this already, they are starting to realize that law school is much different than college.
There are no boldface words and glossaries in the law school casebooks. The Socratic class is not filled with a professor lecturing at passive students for the duration of class. And there are few, if any, written “chapter tests” during the semester so that students can assess their understanding of the material.
But, there are many opportunities throughout the semester where students can assess whether they are picking up what they should pick up in the course. These opportunities happen every day in class as a result of the often-dreaded Socratic method (and I dreaded it when I was a 1L--but, that story is for another blog post).
The professors’ many “what ifs” and “how abouts” give students opportunities to test their understanding of the relevant law; they are given chances to apply this law to many factual scenarios—which, in turn, help the students become better issue-spotters and legal analysts. And, as we all know in the ASP world, the more issues a student is able to spot and analyze on a law school final exam, the more likely that student will gain more points on the professor’s final exam rubric.
So, students: Try to engage with the professors’ hypotheticals in class—even when you have not been cold called in class to verbally answer the questions. Try to answer the questions to yourself in your own head. If you can’t come up with an answer to a hypothetical, write the question down on your notes and revisit that question after class or on the weekend when you review what you have covered in class for the week. You may not have come up with the answer in class. But, that doesn’t mean you can’t come up with the answer on the final exam--when it really counts!
One of the many differences from college and law school is that you don’t have several formal written tests throughout the semester; you often only have one exam at the end of the semester per course that often dictates your entire semester course grade. Try to prepare for that final exam every day in class when you engage with the professors’ hypotheticals, and practice the legal analysis skills that will help make you a better law school test-taker and, eventually, lawyer. (OJ Salinas)
September 25, 2017 in Advice, Encouragement & Inspiration, Exams - Studying, Exams - Theory, Learning Styles, Miscellany, Professionalism, Stress & Anxiety, Study Tips - General | Permalink | Comments (0)