Monday, November 7, 2022
After last week’s onslaught of students with legal writing questions (and some tears), I was hoping to be less busy this week. Yet, this morning I received several requests for appointments from students who just got back their first midterms as 1Ls, so a calmer week is not in the cards here. I have to admit that I spent some quality time this morning trying to decipher why a doctrinal professor wrote question marks on some parts of an exam answer, but check marks on others. I did this without a grading rubric to look at-and to be honest, it was, at best, purely speculative. So here are some things you can do with students when discussing their midterms that does not involve the use of a crystal ball or calling your psychic friends (you can save these resources for determining what might be on the final…):
- Send the student from whence they came: not to their parents, but to the professor who placed the check and question marks on the exam. They might know what they meant-and most likely have a better idea than you do. Or even the TA, who most likely speaks the professor’s language fluently. Tell them to ask for the rubric-or even to just see it.
- Remind students about IRAC. Sometimes the reason a student got a B- on an exam that the rest of study group got an A on is (hypothetically) because they didn’t outline any rules upfront but rather let them accumulate throughout the answer.
- Remind students that we are not mind readers. Yes, your professor knows the rules they are testing, but no, they don’t know what you know unless you tell them. So, tell them, even it is seems obvious, or they think it should “go without saying.” Nothing should go without saying.
- A corollary of the prior rule is do not leave any analysis in your head. Yes, the answer is clear sometimes, but again, explaining why it is clear is where the points come from. The journey > destination.
- Be sure to tell students that midterm exams are a gift. A midterm means that the stakes are lower than just having a final for the entire assessment of the course and understanding what your professor is looking for is a huge amount of helpful information. This also makes the doctrinal professors who give them (and grade them!) incentive to continue this important practice.
- Be sure to remind students where they can find practice essay and multiple-choice questions. Law School exam success is like getting to Carnegie Hall: practice, practice, practice! A reminder of where the exam helpful resources are is always helpful. We have a hidden place on our law library website that is honestly full of great resources, but finding it is a little like looking for the room of requirement in Harry Potter. Since everyone needs these resources, be sure to share the links you know about with students!
As to the other midterms, please go vote. Or be proud that you already have.
"Our lives begin to end the day we become silent about things that matter."
- Martin Luther King, Jr
Thursday, September 15, 2022
I heard a recent joke that goes something like this, in a conversation between an insurance agent and the insured homeowner:
- Agent: Hello.
- Insured: Hi. I'd like to report a theft from my house.
- Agent: I'm so sorry to hear the news. Let me take a look at your policy.
- Agent: Okay, tell me more. Did your house also catch on fire?
- Insured: Oh, no. Just a theft.
- Agent: Well, in that case, I'm so sorry. You're not covered.
- Insured: What do you mean I'm not covered? My policy says right here that it is fire and theft protection.
- Agent: Well, that's precisely right. You see, you bought fire and theft protection, not fire or theft protection. So, since you didn't also have a fire, you aren't covered. It's as clear as day.
All kidding aside, contracts are often like that, as is much of law.
So, as you study cases, statutes, and other legal materials, pay attention to the writing, the terms, and the connectors. Be curious. Think outside the box. Be on the lookout for ambiguities in the text because that's the heart of lawyering, precision. Parse the words, particularly criminal statutes. And, if you seen ambiguities, try to clear them up. And, don't forget to do the same on midterm exams and practice exams. That's because it's in the ambiguities in which the points are most heavily concentrated. And if you'd like more advice and exercise in how to become better at reading, check on Prof. Jane Griese's book on Critical Reading for Law School Success. It's the book that I wished I had had in law school. (SJ).
Thursday, June 30, 2022
You've heard the quip about "the chicken or the egg, which comes first?"
Well, as the joke goes, "I've just ordered one of each from Amazon, so I let you know tomorrow!"
That got me thinking about memorization.
Most bar takers are really concerned about memorization, particularly because most of their law school exams, unlike bar exams, were open book/open note exams. But take a look at the word "memorization." That's a word of action, of a process, of recalling something previously learned. In other words, at its root core the word "memorization" derives from creating "memories." So how do you create memories when it comes to learning rules of law?
Or, to ask it another way, which comes first, memorization or memories?
Well, I think that the answer to that question is in the question because it's memories that we memorize. So the key to memorizing is to work through lots of problems, to test yourself with your study tools, to practice retrieval practice, and, in short, to create lots of memories with the rules.
You see, memorization is just a fancy word for the process of experiencing memories through distributed and mixed practice over time. So, instead of worrying about memorization as you prepare for your bar exam this summer, focus on making memories (and lots of them). (Scott Johns)
Tuesday, April 26, 2022
Law schools have not yet fulfilled the Carnegie Report’s call for more formative assessment. One reason for falling short is conflicting narratives about what is “good” formative assessment. One specific narrative seems particularly troublesome: That the only legitimate method for providing formative assessment is for the instructor to sit down with each student and explain their errors. This post pushes back on that narrative.
- Self-critique is more effective than we appreciate.
Most would agree that individualized feedback from an instructor, the expert both on the subject and the way the student will be graded, is most effective. But especially when using a model answer or quality student answer, allowing students to compare their work against the ideal version, doing so not only assists with doctrinal comprehension, legal writing, and exam skills, but also builds metacognitive abilities. Having the capacity to determine one’s own weaknesses is crucially important, as demonstrated by countless studies showing the performance-enhancing effect of improving metacognition.
The assumption that all feedback must come from the instructor certainly undercuts the mission to improve students’ metacognition. When students find themselves professorless during bar study, they will scramble around helplessly if they have absorbed the legal education fable that only professor knows best. Moreover, new lawyers will certainly struggle in the early years of practice when they need to run to the partner/ division chief/ client to do the metacognitive work for them.
Although some students certainly will do a poor job of self-critiquing (“I mentioned res ipsa loquitur just like the model did! I should get full points!”), this is no reason to underappreciate self-critique. First, in my experience, most students DO figure out their weaknesses from this process. While before the self-critique process they think their C-minus should be an A-minus, seeing the student essay that booked the course tends to leave them thinking otherwise. Second, even if the student still does not see the problems, this is where academic support faculty come in. In partnering with doctrinal faculty, academic support faculty can meet with underperforming students and comment not on the law but on the student’s metacognition. This method distributes personnel resources in a way that makes robust feedback more possible, fosters metacognition, demonstrates to students the valuable connection between doctrinal and academic support faculty, and frees up time for doctrinal faculty.
- Calcifying the status quo.
A particular danger with the solely instructor-based feedback narrative is that it preserves the status quo. We all know that formative assessment is lacking in legal education. The principal argument against remedying that problem is that individualized feedback is so time-consuming that one can accomplish little else. When those inclined to pursue efficient formative assessment are then met with the chorus of voices claiming that self-provided feedback is inadequate, they throw the baby out with the bathwater, dismiss formative assessment, and turn back to the same one-final-exam process used since 1877. Therefore, creating this strict dichotomy between individualized feedback and self-provided feedback makes the perfect the enemy of the very good and leaves students with nothing instead of at least something.
None of this is meant to say that instructor-led feedback is unnecessary or inferior. Feedback from course instructors is crucial. But when that type of formative assessment is not feasible, self-critique is a solid option.
There is a lot more to discuss on this subject. Unlike almost all other graduate programs, why do we think that TAs providing feedback is an unspeakable heresy? Why do we almost never use summative assessment as formative assessment by improving the process of post-semester self-critique? (FYI, simply letting students see their exam answers does not accomplish this goal.) Why do we see testing only as formative and summative assessment but not as a learning tool in-and-of itself?
Unfortunately, the time constraints on writing about the time constraints of formative assessments are such that I have to stop tying now. Ironic.
Louis Schulze, FIU Law
 Self-critique without a model answer is possible, too, but I concede that having a model answer is preferable. To those who would avoid providing such an answer because doing so would take time to write or risk being imprecise, I would argue that a simple solution is to release the strongest student answer.
 Metacognition is the process of assessing one’s knowledge: Do I really know the felony murder rule, or do I just think I know it? As I tell my students, it is like hovering over one’s knowledge and objectively scrutinizing one’s real comprehension.
 See generally J.A. Gundlach & J. Santangelo, Teaching and Assessing Metacognition in Law School, 69 J. LEGAL EDUC. 156 (2019) (reporting on empirical study of first-year law students, finding that students who demonstrated strong metacognitive skills were more likely to perform well).
Monday, April 18, 2022
Today I am sitting on my chosen side of Beacon Street while the Boston Marathon is happening just down the road. I cannot even imagine running over twenty-six miles -- especially considering that my “running” occasionally consists of trying to catch the train in the morning and occasionally deciding that the next train would be fine too.
According to the organizers of the Marathon, there will be numerous places to get water. There will also be places on the route to get some regular or decaf “hydrogel” (which honestly sounds like cheating, right?). There will also be people on bicycles bringing up the rear so that the slower runners are observed and safe even when the roads are re-opened, as well as 26 medical checkpoints. Don’t even get me started on the security surrounding the event (which is entirely understandable), I’ve been hearing helicopters overhead for about a week now. The route is marked on the ground and with signs. People are on the edge of route cheering all the runners on. Overall, the marathon runners are being cared for, protected, and encouraged all along the way.
Are we similarly helping our students in their upcoming exam marathon? We know students have been training for these final exams for months, and like the Boston marathon runners, since it is spring, these students have completed a set of exams that qualified them to tackle this one. We are getting to the point in the semester where just preparing and training will have to give way to running the actual course. Here are some things we can remind students to do for themselves, so they are ready when the starting gun is fired:
- Remind yourself that you have studied and put in the time. While you might have some regrets about the way you have studied or the amount of time you have put in, there is no changing that now and focusing on that will take up valuable time and bandwidth.
- Do not pull all-nighters before exams. This is not a sprint. You do not have the time to recover from a missed night of sleep before the next exam (if there is one), and more importantly you will need to be able to think clearly during the exam. Think of sleep as carbs for the busy work your brain needs to do.
- Be sure you have the correct technical equipment before you get to the starting point. Have you uploaded the exam software? Have you done any tutorials or tests that are recommended? Is your charger working? Now is a much better time to assess your equipment readiness than ten minutes into the exam.
- Do you know the rules? Have you looked over the exam requirements/restrictions? Is it open book, closed book, a hybrid? Can you bring in an index card? Write on the code (if that can come with you), tab your materials, bring in your outline? This will impact how you study and how you plan to access material during the exam (from your brain or other sources). Do you know what to do when you need to use a bathroom during the exam?
- Do you know your exam numbers?? Do you know where to find them? Also, do you have your ID with you for checking into the exam? Take a picture of it now just in case….
- Do you have SHARPENED number 2 pencils or know where to find them-- and a decent eraser? If you somehow ruin all your pencils and erasers during the exam, do you know who to ask for assistance and where they are located in the building?
- Do you have hydration for the exam? Is your water bottle clean and filled? Do you know where the water filling station nearest your exam room is? We will, sadly, be offering hydrogel (which, now that I think of it, may be the active ingredient in my sunscreen…).
- Know that the folks cheering for you on your exam route are your ASP faculty. We are also the folks who will have chocolate, band-aids, tissues, and a place to stash the stuff you cannot get into your locker in an emergency. No, we cannot stand in the hall with banners and yell (distracting and all), but we are there. We know how hard you have worked to get here, and we are behind you 100%.
 Once I was proctoring an exam and the student forgot their school ID and showed me their gun permit as an alternate picture ID. I reminded them (and myself) that I was not grading that exam-just proctoring….
Thursday, March 24, 2022
Courtesy of the Colorado Supreme Court, here's a whole bunch of free hypotheticals that are great for law students, albeit a bit outdated, trying to get a handle on testing themselves through practice exams: https://www.law.du.edu/academics/academic-achievement-program#colorado-exam-essays
Monday, November 29, 2021
Every summer, our family rents a (dog friendly) house out on Cape Cod. Recently, we have been renting bicycles when we get there at a bike rental place called Idle Times. It isn’t fancy, but it is friendly--the name is welcoming and seems to be assuring us that we need not race or even labor much to get around on the bicycles. It is the kind of place where an old black lab lies in the overgrown seagrass and seems to will the kids trying out bicycles to go around him rather than move from his shady spot. It is idyllic-no false advertising involved. This past weekend (that started on a Wednesday-shouldn’t they all?) was also gloriously idle (aside from the cooking, cleaning, laundry, and latkes). I removed my laptop from the table (yes, the new one for those of you who have been following these posts) and didn’t return it to its spot until yesterday. And here I am on a Monday morning trying to jump start my professional brain after this lovely idleness.
Today is the last day of classes for us. While many might think that this is the beginning of a nice break for all academics, it is absolutely crunch time for ASP folks. There are students panicked about finals. They seem shocked that exams are almost upon us despite all the warning signs. I agree that by the time we develop our fall mojo, it is already Veteran’s Day-which was less than three weeks ago. Fall seems like a slow walk uphill to a sudden cliff, while spring semester seems like a cold, dark walk through a cave into the light.
Nonetheless, we are about to begin our "reading days." I’m not sure how much time between classes ending and exams beginning is just right; I don’t think there is a one size fits all time period, but our 1L students have around 2.5 days.
Here is (some of) what I advise students to do now and during these days and the exam period:
- Get out of the law school building (we are all in one building here). The air is thick with stress and every little whisper will make you think someone knows something you don’t about a class you are in. I point out to our students that we are (in the fall at least) out of sync with our undergraduate and business schools, so their libraries might be a better place to study if a library is your preferred spot. At least the din there won’t make you feel unnecessarily inadequate. In pre-COVID times I would also recommend a coffee place (away from school) or even my favorite, the café at the Museum of Fine Arts (excellent place to study and wonderful place to be when you need a break from it).
- Make an exam plan. Work backwards from your last exam and plan reasonable study schedules for each day. Remember to add a teaser of the exam after the immediate one into your plan-so if Civ. Pro is on Thursday, you can take an hour and review a little Crim because that is next and so on.
- Attend to your hygiene and health! Seriously, this is going to be a marathon, pace yourself and be sure to stay hydrated. Don’t take unnecessary pandemic risks right now. Showering is important even if the alternative can help with social distancing.
- Practice writing answers and doing multiple choice questions: while reading carefully will be an important part of your exams, you will still need to produce an answer. You should practice essays often enough that IRAC is a muscle memory. Do enough multiple-choice questions that you are not confused by slight changes in terminology (because…gasp…sometimes doctrinal professors do not write their own questions). Remember, a good way to be prepared for exams is to be a PERP: Prepared for class, Engaged in class, Reviewing after class and Practicing. Ok, now I can see why this didn’t catch on, PERP is just not going to happen. But there is still hope for fetch.
- Handle different subjects with different strategic approaches: Civil Procedure is linear and chronological; Contracts is transactional; Torts and Criminal law just beg for making a chart with all the people and causes of action involved and so on…
- Just get started: if you are lost on the exam, start with something you can answer to get the brain engaged and then go back. However, do not go back and change any multiple-choice answers if you have already made a choice-it will not end well.
- Get out again-after the exam, leave the building. Do not discuss it with other people. I know that talking about a shared trauma can be therapeutic, but this will not be. I promise. Think about what you have done well on this exam and then move on with your plan. As Timon famously says in The Lion King, “You gotta put your past behind you.”
- When all the exams are over, enjoy the idle time.
Thursday, October 21, 2021
Poet Robert Frost writes:
Thursday, September 30, 2021
Often times I see but I don't. Perhaps an analogy will explain.
It's bear season where I live. But the bears are awful hard to spot, despite their large size. It seems that their big paws tend to distribute weight so that they move with stealth-like grace as they forage among the mountain berries, shrubs and trees. They tend to make not much more noise than a trifling breeze or a bird at work building a nest.
But I have a secret weapon to spot the bears - my dog.
You see, a few weeks back, while hiking, Maisey came to a screeching halt, sat perfectly still, and sniffed the mountain breeze. A sniff here and a sniff there. I was like, "Come on Maisey, let's get going." But she sat, still.
After about 5 minutes of waiting silently, I finally noticed a slight rustle down the hillside from the trail. Not much of anything. But then another rustle and another and another, all ever so silent. Suddenly, I saw what Maisey had sensed all along before. A bear, foraging in the scrub oaks. For the next twenty minutes or so, I watched the bear slowly eat its way down the hillside before I finally lost sight. But the lesson wasn't lost on me. I would not have seen that bear by myself. I needed the sense of another, one with keener senses than me.
I think law school is bit like that.
As law students, we can re-read our papers or our notes or our midterm answers and not really see what we really wrote. It's sort of like we are blinded by our own senses, by our own sight.
However, much like my experience on the trail scouting for bears, as law students, we have available to us, just for the asking, people who have keener senses than us, finely tuned, who can take a look at our work and thus open us up to a whole other way of seeing and experiencing things. In short, we can turn to our faculty and academic support teams to help us - as learners - see what's really in our answers (and what isn't).
So, as law students, don't feel like you need to go it alone in law school, at all. Freely reach out to others for help. Let experts review your work. Get feedback from your professors and your ASP team at your law school. You'll be surprised at what you'll see. It probably won't be a bear, but I can guarantee that it will help you become a better attorney. And that's what we are here for -- for you. (Scott Johns).
Monday, September 20, 2021
One of the ways we support our students who are on academic warning or probation is to require them to take a second-year course in Legal Analysis and Methods. The title is vague enough to appear on a transcript without stigma to the student and, as a side benefit, it also gives us a lot of latitude in what we teach in the course. In my section of Methods, I teach study and exam skills as well as a smidge of legal writing, a dash of argumentation, and a bissel of statutory construction/interpretation. I also conference with students one-on-one towards the beginning of the semester to check in on an ungraded “getting to know you” assignment and to try to understand how they got stuck, I mean were fortunate enough to enroll, in this class.
I had a set “script” for these conferences. At the beginning of each conference, we discussed the ungraded assignment (there is written feedback for everyone as well). I thanked each student for doing a great job in our simulated legislature class last week (seriously, the Massachusetts legislature could learn from them). Then, I asked about the other classes they are taking to see where there might be stress points.
Finally, I ask about the elephant in the room, “How do you find yourself on Academic Warning/Probation?” I intentionally use the passive voice. If a student says they had some “personal problems,” I do not ask for details, I just ask if the issues are resolved (or resolving), and if our Dean of Students’ office is aware of them just in case they need some higher power intervention. If a student says they had issues on exams, I make a note of the type of exam it was for future classes on exam skills. Now granted, I knew some of the students coming into these conferences because we met regularly last year. Other than now knowing how tall they really are and confirming that they do indeed have legs, I didn’t need to hear how they got here, but I did need to know how they were doing now.
This year, like all years, I take notes of these meetings. As I flipped through the legal pad for these conferences after meeting with my 22 students, I saw one word show up at the end of my notetaking for every single student, “Zoom.” This was the always part of the answer to how they found themselves in academic trouble.
Zoom or remote learning wasn’t the whole problem for most students: it was Zoom plus. Students told me that last year was not academically successful because of Zoom plus: ADD, ADHD, anxiety, dyslexia, having COVID, having a family member with COVID, having a chaotic living situation, having a bad internet connection, and so on. But remote learning was, as one student put it, “at least 30-40% of the issue.” Everyone in the remote learning situation-those of us teaching and the students learning- were all trying our best. The bottom line is that remote learning does not work for everyone. These students were concerned that when they take the bar, they will not have learned enough in their first-year classes to get them into a passing range. They felt that they were building their law school houses on weak foundations. This is a valid concern. Going through two (or three for evening students) more years of law school feeling like you are perpetually trying to overcome a deficit will also take a toll on confidence.
I am not saying that remote learning is universally negative either. I had students last year that thrived in a remote learning environment, as well as students who were very nervous about returning in-person because of the pandemic. Remote learning allows broader access for students; I think that is the promise of remote learning going forward. A student can, for example, attend a law school in a place they cannot afford to move to (like Boston) or attend school when health or family issues might otherwise prove an insurmountable barrier. And this is not even close to a complete list of pluses.
Yet, the students who preferred remote learning are just simply not the students I am seeing in academic distress right now. I am not asserting that my 22 student class is a representative sample of all law students but they are mine to teach and I need to know where things fell apart for them before they came to me. The current in-person situation has pluses and minuses as well. Students report that are much happier to be back in-person--but also stuck in a position of navigating the 2L curriculum with a 1L understanding of law school culture. Some of them have spent less time in the building than the 1L students who came to school before classes started for orientation-- a few more cracks in the foundation that will need filling. One student thought that being called an upperclassman was laughable because they felt they had very little to offer the incoming class in terms of wisdom and “the ways” of law school. And yet, they hoped that the expertise they did have was, and would continue to be, obsolete. I hope so too.
As academic support folks, we know there have always been (and will most likely be) students who are in academic distress. Some have had family issues, relationship issues, a failure to understand the time investment etc., but it seems that today’s students have all of these troubles plus Zoom.
 Bissel means just a little bit in Yiddish, https://www.dailywritingtips.com/the-yiddish-handbook-40-words-you-should-know/
Sunday, September 12, 2021
A supermarket I go to occasionally has a little maze of seasonal items right at the entrance. This is a tricky way to entice you to purchase these colorful things at the beginning of your shopping trip when your cart is empty and/or perhaps a way to mollify any small child you have brought with you with something small to play with on the journey. Yesterday, when I arrived, I saw a sea of orange and black: candy as far as the eye could see. There were also some decorative scarecrows and rust colored “hardy mums,” (which I consider a challenge, but more on my lack of gardening skills in another post). In the middle of the maze, I saw a single display of popsicle shaped window clings, Whiffle bats, sunscreen, some mismatched kickboards, and s’mores skewers-all 50-75% off. Summer was on sale-despite the fact that there is at least another week of it on the calendar. I sighed and realized that we had actually already been in school for almost four weeks and the time had come for the seasonal shift from merely briefing cases to…(please read each of those periods as dun, dun, dunnn respectively): OUTLINING.
Now that students have covered at least one full topic in each class, the time has come for them to take those case briefs and carefully written class notes and knit them into a nice cozy outline for December exams or, more urgently, upcoming midterms. This is also a good time to start because it intersects with students learning how we use and talk about cases in their legal writing courses. The magic formula of how we use cases in memos and briefs: FHR (facts, holding, reasoning) is how they can incorporate the components of their case briefs into their outlines. This is really a win-win because they are practicing using the FHR formula for outlining in legal writing and vice versa.
Now, I know you have probably discussed outlining at least twice already with students. We do it in a pre-orientation module, during orientation itself and have a class on it planned for the coming week. The number of times I say, “your outline should be rules based rather than case based” could be a drinking game at this point (not that I condone drinking while outlining as either effective or efficient).
How can we best communicate the message that it is currently prime outlining season to our students? I thought of the buzzer at the beginning of a swim meet heat, a ribbon cutting or even a giant banner, “START OUTLINING NOW!!!!” Maybe I should stand in front of the law school with a sandwich board that says, “Ask me about outlining-I’m not just an ASP professional, I’m a client!” Maybe we should perform, “Outlining the Musical,” with such tunes as:
525,000 pages more,
How do you make sense of a course in the law?”
“Oh, it is time to start ‘lining,
Time to take a little of the briefs we’re writing,
Time to take time,
Because it’s already fall--exams are in just no time at all….”
(sincere apologies to Rent and Pippin).
Yet, we all know that no matter how or how often we sound the alarm at this point in the semester, we will still be talking with students who are just getting started in November. And while we will silently groan and do an internal face palm, we will advise those students to move as quickly as they can to ideally finish their hastily organized (but nonetheless helpful) outlines when classes end.
I expect that the next time I will need to think about getting students to begin outlining, the supermarket entrance will be aglow in red, pink and white: candy as far as the eye can see -- except for the candy canes in the center on sale.
Tuesday, August 10, 2021
Over the years, I have noticed that many legal educators and students have an imperfect understanding of the utility of using prior exams for practice. This misunderstanding usually holds that the purpose of such materials is for students to review the exams simply to see what topics professors test and methods with which they do so. In turn, faculty become leery of providing such materials, as doing so might create an unwarranted expectation on students’ part that their exam will test the same topics and use the same methods.
This impression is problematic. Both students and faculty are squandering the opportunity for students to use materials that will make them better learners, improve their performance in law school and the bar exam, and increase their knowledge and skills (both in classes and on the bar exam).
An important recent (methodologically sophisticated) study supports this claim. In Understanding the Metacognitive “Space” and its Implications for Law Students’ Learning, Professors Jennifer Gundlach and Jessica Santagelo found statistically significant evidence that: “Students who reported using active strategies at the end of the semester were more likely to succeed in the class … relative to students who never used active strategies.”
Faculty should better understand the use of prior exams and other materials that would allow students to practice rather than re-read over and over again. Although many law professors used re-reading and re-reviewing prior exams in their studies, their success quite possibly could have been despite and not because of those flawed methods. Faculty tend to have had an elite education, elite aptitude, and elite socio-economic condition opportunities for academic success. They thus had a great degree of wiggle-room in terms of the efficacy of their learning methods
Many of our students are not so lucky. If we admit students with fewer socio-economic opportunities and with non-elite academic credentials, we should not erect further obstacles to their success by assuming that the methods we used in very different circumstances will be effective for them.
Especially given the recent findings quoted above, we should not rely on the anecdote fallacy (that because one person had success with a method all will have such success) and the post hoc ergo propter hoc fallacy (that because certain study methods preceded success, those methods must have caused that success). Instead, we should rely on the empirical evidence that shows that active learning, including taking practice exams, fosters success more optimally.
(Louis Schulze, FIU Law)
Thursday, February 25, 2021
This week the Association for Academic Support Educators ("AASE") published Best Practices for Online Bar Exam Administration. AASE President, DeShun Harris, says that the best practices advocate for "procedures that ensure a fairer test for online test takers." The organization, established in 2014, urges state high courts and bar examiners to adopt these procedures. The AASE Bar Advocacy Chair, Marsha Griggs, says "many of the best practices that we identified are things that bar examiners are already doing." Yolonda Sewell, Vice President for Diversity, adds that in addition to the great strides that bar examiners have made in deploying an online exam, we seek to make sure that the online administration does not unfairly disadvantage any bar applicant on the basis of skin tone, race, gender orientation, biophysical conditions, disability, need for test accommodations, or socio-economic resources. The Best Practices are aimed to level the playing field, both among applicants of varied backgrounds, and between the online and in-person versions of the exam."
One of several effects of the COVID-19 pandemic, was that bar examiners and bar applicants questioned the wisdom and feasibility of administering in-person exams in the traditional large group format. In response to COVID-19 limitations, the first online bar examinations in the United States were administered between July and October 2020.
With but a few exceptions, the online exams were remotely proctored using artificial intelligence technology provided by a commercial vendor. As the exam dates approached many issues surfaced surrounding the use of facial recognition software and remote proctoring. One prominent issue was the number of complaints voiced from students who are people of color, asserting that the software did not recognize them. During and after the exam, other complaints sounded, ranging from data breaches, and poor technical support, to "flagging" hundreds or thousands of applicants for alleged cheating or "testing irregularities." At the extreme, some applicants reported having to sit in their own waste—as the exam instructions warned applicants about being out of view of the camera except during scheduled breaks—for fear of failing the exam. Additionally, there were reported issues with the technical delivery, submission, and scoring of the Multistate Performance Test, and jurisdictional scoring errors that wrongly identified applicants who earned passing scores as exam failures, and falsely notifying others who failed the exam that they had passed.
AASE lauds the efforts of bar examiners at the local and national levels for their flexibility and willingness to provide options for remote administration. While we defer to the proven expertise of the test-makers in determining matters related to exam content, scoring, accommodations and character and fitness eligibility, we add our collective expertise in assessment delivery, performance application, and enhancement pedagogies for non-traditional test takers. We recognize that online bar exam delivery will outlive the pandemic and current circumstances. We also believe all who play roles in the process of creating and delivering a bar exam, want the exam to be fair and effective. In light of those dual goals, we think the time is ripe for adoption of additional policies that are more than performative gestures toward a more diverse legal profession.
(Association of Academic Support Educators)
Tuesday, February 9, 2021
Here it is, Tuesday evening, and I am finally settling down to write another blogfest – this, like many weeks, despite having specifically placed this high enough up on my to-do list that I genuinely expected to be starting in the early afternoon. The problem – one I am sure we are all familiar with – is not the writing, but all the other things I had planned to finish beforehand, which took far longer than I had originally estimated they would. Fortunately, such difficulties are illustrative of this week’s topic of discussion – the planning fallacy and how to counteract it.
The planning fallacy is a simple psychological phenomenon: human beings’ predictions about the time needed to complete a future task are usually significant underestimations. In some cases, wild underestimations: for example, when construction began on the Sydney Opera House in 1959, it was expected to be completed by 1963, but the site was not actually finished until 1973. Daniel Kahneman and his partner Amos Tversky were the first to describe this phenomenon, more than forty years ago, and Kahneman writes about it in his wonderful book, Thinking, Fast and Slow. He explains it as a kind of optimism bias, a tendency of people to adopt the rosiest scenarios as they imagine how a task will proceed. Later scholars added other nuances to this explanation. One reason for this apparent optimism bias, for example, might be the self-serving human tendency, when considering similar past situations, to take personal credit for all the things that went right (and thus assume they will go right again in the future), but to attribute errors and delays to outside forces that they presume will not occur again.1 Nassem Taleb, in his book Antifragile, suggests it may not only be a psychological phenomenon, but also a consequence of a natural asymmetry: whenever circumstances or events cause a deviation from a well-laid-out plan, chances are far greater that the disruption will lead to delay than to expedition, so that the sum total of all deviations would always be expected to be postponement.
How many times have we seen the planning fallacy in action amongst our students? Just in the past month, I have met with returning students, vowing to perform better in this coming spring semester, who base this determination on unaccountably confident projections of all the steps they will complete to do so. I have worked with February bar examinees, noses to the grindstone, who despite their genuine efforts are finding themselves slipping behind their intended schedules. Not every student suffers from this bias, of course, and many of those who experience the bias don't actually suffer for it, either because they start with ambitious goals that leave plenty of leeway or because they find the extra time and energy to offset their underestimated projections. Still, every year brings a significant crop of students who do not perform as well as they might have, because they seriously underestimate how long it will take them to complete an essay test question, compile a useful outline, learn the rules governing a specific legal topic, research, draft, and edit a significant writing assignment, or attend to the demands of student organizations.
Fortunately, the psychologists and scientists who have studied the planning fallacy have suggested a few strategies that can be used to counteract it, and these strategies are easily adoptable -- or correspond to techniques already used -- by academic support professionals. In his book, Kahneman suggests the use of reference class forecasting -- that is, making predictions of the time needed to complete a task based not on a person's (or an entity's) internal sense of how long it should take them, but on observations of actual outcomes in prior similar situations. In other words, if I were going to build an opera house, I might start off by assuming I could get it done in a few years, but if I considered how long it took to build the one in Sydney (and of course in other locations), I should understand that it is likely to take more than a decade. Many of us do something at least adjacent to this with our students already -- providing them with estimates about how long they should expect to take to complete a case brief, for example, or to study for the MPRE -- but the idea of reference class forecasting suggests that it might be even more powerful to refer specifically to prior performances by other students. Instead of saying, "You should devote at least 24 hours," it might be more effective to say, "Last year, every student who devoted 4 hours a day, every Saturday and Sunday, for three weeks, completed this successfully."
Another suggestion is the use of the segmentation effect. It has been observed that a person's estimate of the total time it will take to complete a task will be longer -- and thus likely more accurate -- if they are asked to segment the task (break the task down into a number of sub-tasks), to estimate the time it will take to complete each sub-task, and then to add all those times together to come up with the total time.2 However, there is a cognitive cost to being mindful and particular enough to break complex tasks down into numerous sub-tasks, and, without help, this kind of approach may be hard to learn and sustain. Fortunately, this is just the kind of help we can give, especially to inexperienced students who may not be able to envision how a long-term task can be broken down, or even what all the steps involved might be. By providing students with a framework of what to expect, and encouraging them to think realistically about what it will take to build each part of that framework, we can help them to stay on track, or at least in the general vicinity of the track, by using the segmentation effect.
Finally, another tool that has been suggested to combat the planning fallacy is the implementation intention, a term coined by Peter Gollwitzer for a particular model of thinking about future actions. Encouraging people to think specifically about when, where, and how they will act towards their goal tends to make them more likely to move forward steadily, and in a timely way, towards them. For example, people who received a telephone call in which someone asked them what time they planned to vote, from where they would be heading to the polling place, and what they would be doing just before they left to vote -- all questions designed to prompt them to think about when, where, and how they would vote -- were more likely to vote than those who did not receive the phone call.3 The mental IF-->THEN statement (as in, "If I am aiming to take a practice exam, then I should get a copy of an old exam from the library on Friday") is the implementation intention that moves people apace towards their goals. This, too, is something that academic support professionals do, or can do. By querying students about the specifics of how they expect to achieve their long-term goals, we can induce them to map out their plans in advance, changing vague ambitions about what they would like to achieve into articulable steps (the implementation intentions) that they can follow methodically to their desired ends within the time they have available.
It is a natural human tendency to overestimate what can be done in a given period of time. By helping our students account for this tendency, even if we cannot help them complete everything, we can at least help them get in a position where they've done enough to succeed.
1(1995) It's About Time: Optimistic Predictions in Work and Love, European Review of Social Psychology, 6:1, 1-32,
Sunday, January 31, 2021
Some of you are getting your Fall grades, and for some of you this is the first time in getting law school grades. So let me assure you of something – your grades do not define you. If your grades make you happy, you should be proud. But they don’t define you. If your grades were not what you had hoped for, your grades do not define you. Your actions, the way you treat classmates, the cases you take on, the way you treat future clients and future colleagues – all of these things will define you. But your grades are not on that list.
Now, having said that, I’m well aware that grades are important, they help you get on things like law review, obtain clerkships and obtain your first jobs. So yes, they are important. But no, they do not define you.
So, if your grades are less than stellar, or not what you were hoping, pick yourself up and learn from your mistakes. The key is to not dwell on the mistakes, but learn from them. What can you do in the Spring to bring those grades up? Or, if you are happy with the grades, ensure that Spring grades are just as good? Try the following:
- Meet with your professors and review your exams. Even if you did fairly well, it’s worth looking over your exams with your professors. Talk about what you did well, and where you can improve. Many students focus on the fact that they correctly issue spotted, or came to the “correct” conclusion, when in reality, most professors are also looking at things like organization, and most importantly, how you came to the conclusions that you did!
Don’t be afraid to reach out. I talk to many students who are embarrassed, or a bit nervous. Your professors want you to improve and succeed. It is part of their job to review these exams with you, so please approach them!
A good starting point is this CALI lesson: How to Learn from Exams, by Melissa A Hale
- Take an honest look at this past semester, and self assess. This is incredibly important for your law school career, but also for your legal career. Learning to self assess performance is an invaluable skill that we all need.
- Did you read and brief all cases?
- How much time did you spend studying for each class?
- Did you meet with your professors during the semester, and talk about things you struggled to understand?
- Did you complete practice exams? If so, how many?
- Did you start outlining early or late?
Be honest with yourself in all of these questions. In addition ,think about other things that might have been happening, especially since we are currently in a pandemic:
- How was your mental health?
- Do you have test taking anxiety?
- Did you have a good place to study?
- If you are easily distracted, did you find ways to deal with that anxiety?
- Were there any life events that interfered with studying? Such as a break up, a death in the family, other personal turmoil? A health concern or health issue?
Again, this is a self reflection, so be completely honest. There is a CALI lessons that helps lead you through these issues, written by Renee Nicole Allen - Semester Self Assessment and Reflection - https://www.cali.org/lesson/18326
In addition, try the following:
Grit, Growth, and Why it matters, by Melissa A. Hale
Assessing Your Own Work, by Allie Robbins
Above all else, remember that while grades might open up some opportunities for some, even a few years into the future, they will not matter. And also remember, you can always learn and grow – whether you want to improve on something you did well, or learn from mistakes.
Tuesday, December 1, 2020
Now that Thanksgiving is past, most law students have started or are about to start preparing for final exams. With an entire semester of material to master, many try to prioritize what to spend their time on. Some concepts and rules, introduced early in the course, may feel tediously familiar through repetition, and students may feel they can afford not to spend time on them now, especially if they tested successfully on those points on the midterm. Experimental evidence suggests, however, that time could still be well spent, even on familiar material, if it is spent the right way.
Jeffrey D. Karpicke and Henry L. Roediger III, psychologists at Washington University, ran a series of tests of memory recall of lists of words, examining the effects of two distinct tasks that contribute to learning: studying and testing.1 Their first set of tests, similar to other experiments that had been done over the previous 40 years, sought to determine what combination of studying (in this case, visually reviewing lists of words to be memorized) and testing (writing down recalled words under time pressure) would produce the best learning. Some subjects were told to study a list of words, then were tested on their recall, then given another opportunity to study, and finally given one last test. These were identified as STST subjects. Other subjects were given disproportionate opportunities to study, or to test: either three study periods and one test (SSST) or one study period followed by three tests (STTT). The scientists then compared recall performance for different groups of subjects after the final test. Hopefully not surprising to either law students or Academic Success professionals, the subjects in the STST group had the best recall in the end. As other psychologists have observed, a mix of studying and testing produces the best learning.
What was new and interesting was the second phase of their testing, in which they variations on the STST pattern on new groups of subjects, and tested recall not just at the end of the four-step pattern, but also on an extra test given one week later. In addition to testing some volunteers using the original STST method (study, test, study, test), and to tweaking the order (but not the ratio) by giving some volunteers two study sessions followed by two test sessions (SSTT), the scientists tested a third set of volunteers by starting with the same size set of words in the first two steps (study, test), but then removing from the word list all the words successfully recalled at that point, and then asking the volunteers to study and then test using only the words not recalled in the first test. Using the reduced list of words was identified with a subscript "N", so this set of examinees was called "STSNTN". [This method is familiar to many who study (or recommend studying) with flash cards by removing from the deck each card you recall correctly, so that every new pass through the deck, you are only studying and testing yourself on the information you failed to recall the last time.] Finally, a fourth set of volunteers similarly started with a complete list of words, which they studied and were tested upon, and then spent a second study period studying only the reduced list (that is, again, they did not have to study any words they had already learned). However, on their second test, all of the words from the initial list were tested, even those that were not studied a second time. Thus, this was the "STSNT" group.
What Karpicke and Roediger discovered this time was that the STSNTN group clearly had the best recall after the second test -- in fact, among the fifteen subjects tested in this group, there was only a single instance of a word not being recalled. In other words, this group had the "fastest initial learning" of all, apparently because they focused both their studying and their testing on material that they had not previously learned. They learned the material more quickly than either group that studied and tested on all the words twice (STST and SSTT) or the group that studied only material that it had not previously learned, but tested on all the words twice (STSNT). And this has an intuitive appeal to those of us who have used flash cards -- if you focus on your gaps, you can overcome them more quickly, right?
However, the scientists also tested all the subjects one week later, asking them to recall all the words from their initial lists. And on this later test of long-term memory, the STSNTN group performed the worst. One might simply attribute this to those subjects having spent less total time and mental energy on the learning task, since they had a reduced word list for the second half of their learning. If that were the case, though, you would expect that the best long-term results would have been seen in the STST group, which studied and tested on full word lists twice. In fact, the long-term performance of the STSNT group was as good as, and in some cases better than, the performance of the STST group. This suggests that it does not matter so much whether you spend time studying material you already know, as long as make sure to continue to test yourself on that material along with the material you are continuing to learn.
So over the next few weeks, as our students work somewhere between diligently and frantically to prepare themselves for their final exams, it seems that the most efficient and effective use of their time will be to focus their study time (reviewing, rote memorization, consulting supplementary material, asking TAs and professors questions) on the things they are unsure of, but to continue to test themselves on everything in the syllabus.
1Jeffrey D. Karpicke & Henry L. Roediger III, Repeated retrieval during learning is the key to long-term retention, Journal of Memory and Language 57 (2007) 151-162.
Wednesday, November 25, 2020
To the Law Students, especially first years:
It's ok to take a break. I promise. I wanted to give you some great advice for exams, and since Victoria has written about so many tips for prepping for and writing the exams, my advice is to relax.
I know, I know, you have so much to do. But your brain really DOES retain information better if you give yourself breaks. I'm not suggesting that you don't do any work between now and Monday, but I am suggesting that you pace yourself, and do a few fun things for yourself as well. Netflix does have some great holiday movies coming out!
If you are a first year, it's also important to put things in perspective. Yes, grades are important, we can't get around that issue. But so is your mental and physical health. It's tempting to put so much pressure on yourself, and you need to realize that as long as you are doing your best, that's all we can ask of you.
Also, don't be afraid to ask for help. From your professors. From your Academic Support people. From a Dean of Students. We all want to see you succeed, and we will do what we can to make that happen. If you don't know how to ask for help, that's ok too. It's a good idea to start with someone like your Academic Support person, or your Dean of Students. Literally go in and say "I'm having a hard time, and I don't know how to ask for help." We will help you! Promise!
And speaking of help, if you are entitled to non standard testing accommodations, and especially if you have used them in other situations, please reach out and ask for them! For some schools, it might be late in the process, but it can't hurt to at least see what's possible.
Finally, practice! Don't go into your exams having never practiced one. The more practice exams you do, the better you will feel, and it WILL be reflected in your grades.
To my Academic Support Colleagues:
We need to take this advice as well! We are so good at advising our students on how to take care of their mental health, how important it is to take time for breaks, and to do things for themselves. But we don't always take our own advice. Hypocrites, the lot of us. The AASE programming board put on a fantastic workshop about how we should be helping ourselves, and each other, so take this as an important reminder. Especially, if you are like me, your list of things that must get done this weekend is a mile long. I'm going to try to take my own advice and do the best I can, while also spending some time with my husband, and maybe watching a Christmas movie or two! I hope all of you do the same.
Finally, I've been doing a month of gratitude. Each day, on facebook, I note one thing I'm grateful for. I started doing this a few years ago, as it is a really good way to boost serotonin and put things in perspective. This year I felt it was especially important, since this year has not been one of the best. So, I want all of you to know that I'm incredibly grateful for this community, and I'm thankful for each of you. Your wisdom and expertise, your friendship and support.
Happy Thanksgiving To All!
Monday, November 23, 2020
Continuing last week’s post, students looking to develop and/or refine their exam-taking plan of attack may want to consider the following:
- Reading and dissecting the fact pattern. Once you have selected a question, begin by reading the prompt closely. If you know what the “ask” is upfront, you can read with more intention. You also avoid the surprise of reading the fact pattern, reaching the prompt, and having to re-read the fact pattern again because the prompt asks something you did not expect. Next, read the fact pattern carefully. As you read, question everything! There are not a lot of “extra” facts in a law school fact pattern so ask yourself what your professor wanted you to take away from each sentence. Are the facts triggering an issue? Are they relevant to your analysis of an issue? Are they relevant to a counterargument? Does the framing of the facts create an ambiguity ripe for analysis? Remember, if an issue is triggered by the facts, you are expected to analyze it in your response. Pay special attention to names, dates, dollar amounts, and quoted/excerpted language. If your professor took the time to give someone a name, that name should probably appear somewhere in your answer. If your professor gives you a specific date or dollar amount, such details might have legal significance (e.g., statutes of limitation, timing for contract formation, statutory period for adverse possession, time to serve a responsive pleading, amount in controversy requirement, applicability of the statute of frauds for goods transactions). Finally, if your professor takes the time to tell you exactly what something or someone said, there is likely a reason why.
- Tracking issues. The facts will trigger multiple issues in each exam question so your attack plan should include a method of keeping track of the issues that you spot (e.g., by numbering, creating an issue list in the margins). As a rule of thumb, you should plan to spend roughly 15 minutes per hour that is suggested on your pre-write phase: reading, issue spotting, and “outlining” your answer (i.e. ordering the issues and creating an exam structure). Do not skimp on this time. Professors can tell when students do not outline their answers because such responses are often poorly organized. The exam-taking experience is also much more frenetic for the student.
- Tracking facts. Consider how you might create visual cues to identify the specific facts that are relevant to each issue you spot. You could use a color-coding method, numbering method, symbol method, etc. to identify the facts from the fact pattern that are relevant to your analysis of each issue. Do not overcomplicate this process—choose a method that is efficient and functional. Incorporating a method for tracking facts can be an especially helpful practice when you encounter a long fact pattern because it can reduce the amount of time you spend re-reading the fact pattern in search of facts relevant to your analysis.
- Ordering issues. The order in which issues appear in a fact pattern probably will not be the same order that makes the most sense, structurally, for your analyses to appear in your answer. Consequently, your plan of attack should include a step for ordering the issues you spot. Keep in mind any directives your professor gives you in the prompt about ordering (e.g., beginning with the strongest claim, analyze all claims Tanya can bring against Sasha). If you have a broad prompt that contains no directives about how to order issues (e.g., analyze all potential claims), then the relationships among or chronology of concepts can help with ordering.
- Creating an answer skeleton. Once you have ordered the issues you spotted, create headings in your exam answer. Consider whether each issue you spotted will need its own heading and whether particular issues should be grouped together under one heading. Keep your headings short and sweet—they need not be in complete sentences.
- Fleshing out your answer skeleton. Before you begin adding the meat to the bones of your answer skeleton, consider the order in which you fill in your answer skeleton. Though it may make sense, structurally, for issues to be arranged in a particular order, you should analyze issues from biggest/most obvious to smallest/least obvious. The more obvious an issue is, the higher the likelihood that your professor expects most—if not all—students to spot and analyze the issue in their answers. The more time and space in the fact pattern your professor devotes to articulating facts relevant to the analysis of a specific issue, the more there is to unpack and analyze (and thus, the bigger the issue).
While there is no magic blueprint for creating an effective plan of attack, these points will help students develop an effective exam strategy. The more a student practices and works to refine their attack plan, the more poised and prepared that student will be when exam day arrives. Students may not have control over what facts they will encounter on the exam or how their professors will test them on a given issue, but they do have control over their overall preparedness for the exam and development of a system for taking the exam. So, to students I say: focus on that which is within your control.
(Victoria McCoy Dunkley)
Thursday, November 19, 2020
The best way to prepare for final exams is to work through lots of practice exams. That's because we learn best through the experience of testing ourselves. So here's a link, organized by subject matter, with past bar exam essays (and answer materials). You can freely share these exam materials free-of-charge with your students as they prepare for their final exams, courtesy of the Colorado Supreme Court: https://www.law.du.edu/academics/academic-achievement-program#colorado-exam-essays
Questions Arranged by Subject
- Administrative Law
- Civil Procedure
- Constitutional Law
- Criminal Law
- Criminal Procedure
- Family Law
- UCC Article 2 – Sales
- UCC Article 3 – Commercial Paper
- UCC Article 9 – Secured Transactions
- Wills and Trusts
Monday, November 16, 2020
When preparing for law school exams, law students frequently develop a framework for analyzing particular issues they may encounter on an exam. They also often plan how to use their outline, casebook, and rulebook during open-book exams. Indeed, many of us have seen (perhaps even created): elaborate flowcharts for a choice of law or battle of the forms analysis; lists of steps/questions to guide a personal jurisdiction, equal protection, or parol evidence rule analysis; and mind maps for navigating negligence claims. We have seen casebooks and rulebooks bursting with colorful tabbies, and outlines complete with tabs, indices, and tables of contents.
An oft-overlooked aspect of exam preparation, however, is a broader strategy for how to take the exam. Where is the checklist of steps students will take to navigate the exam? What will they do when the exam clock begins ticking? Students who develop, practice, and refine a plan of attack build exam-taking muscle memory that pays dividends on exam day. Quieted is the sense of panic and amplified is the sense of know-how.
As with so many things, there is no singular “right” way to develop a successful plan of attack and a student’s plan may not look the same for each class. However, students looking to develop and/or refine their exam-taking plan of attack may want to consider the following:
- Types of Assessments. Consider whether your exam will consist of multiple-choice questions, essay questions, short answer questions, etc. If you have a mix of multiple-choice questions and essays, consider working through the multiple-choice section first. There is typically more unpacking to do in a law school essay than you can possibly complete in the time allotted. If you begin with the essays (rather than the multiple-choice), you run the risk of exceeding your allotted time for essays and, as a result, not having adequate time to carefully work through the multiple-choice questions. If you do not know what types of assessments will be included on each of your exams, ASK. ASK NOW. If your final exam has a multiple-choice component, also ask about the number of questions and suggested time allocation in order to get a sense of the pacing. Work toward achieving that pace via practice and be sure that your plan of attack accounts for the multiple-choice component of your exam.
- Choosing where to begin. It is likely that you will encounter more than one fact pattern on your final exam, so your plan of attack should include a strategy for choosing where to begin. You should not plan to automatically proceed in the order the questions are presented. In other words, you do not have to do Question 1 first simply because it’s labeled “Question 1.” Instead, get a sense of the exam by skimming each fact pattern, noting the time suggested for each question. In choosing where to begin, adopt a methodology that best suits you. For students who prefer to ease into the exam and build confidence as they go, an uphill approach might be a good fit. Those students would begin by answering the question they perceive to be “easier” and work their way up to the harder questions. For students who prefer to get the “hard” stuff out of the way first, when they are most alert and there is a low likelihood of exam fatigue, then a downhill approach might be a better fit. Those students would begin by answering the question they perceive to be “harder” (or, perhaps, more rigorous because it has the longest suggested time) and work their way down to the easier questions.
(Victoria McCoy Dunkley)