Tuesday, June 4, 2019
Yesterday, the quiz show Jeopardy! enjoyed its highest ratings in more than 14 years, <spoiler> on the day that 32-game winner James Holzhauer lost to librarian Emma Boettcher and fell just short of breaking the all-time record for most money won during regular play. (Sadly, James walked away with only $2,464,216.) My friends in the trivia community have been watching James's exploits with various mixtures of admiration, envy, bemusement, and exasperation. The latter two emotions have been prompted not by James himself, but by the sense-making reactions of casual viewers and the media to his success, and then to his defeat.
James racked up an intimidating number of high-scoring games -- including all of the top-ten highest-scoring games of all time -- and he sometimes won by six-figure margins. To a lot of pundits, these overwhelming victories suggested a new and singular player: either someone with unmatched, superhuman genius, or someone who had come up with a novel strategy that had "broken" the game forever. From the perspective of a lot of fans at home, this made sense. How else could someone achieve such never-before-seen results without some sort of mystical secret ingredient?
But to a coterie of former players and dedicated aficionados, there was nothing mysterious or unduplicable about James's style of play. He is a tremendous player, to be sure, certainly among the best. But the skills he brought to the game are pretty much the same skills other great players have exhibited before. He knows a lot of trivia; he is very adept at using the signaling device to snatch the opportunity to answer first; he understands the optimal strategies for choosing clues and making bets. His historically high scores are due mainly to a gutsy willingness to risk losing all or most of his pot by making big bets that, when successful, have left him with insurmountable leads. In the past, even the strongest players played more conservatively, hedging their bets so a wrong answer wouldn't take them out of the running. But James is a professional gambler, and he decided to maximize his return by maximizing his risk. This was a choice, not an aptitude, and anyone playing against him would have the capacity to make the same choice.
In fact, in yesterday's game, Emma did just that, making her own big bets to take a lead that James could not overcome. When the game hinged on one final question -- one that all three contestants would have the chance to answer, and on which each would have to make a wager -- Emma, in the lead, bet most of her accumulated winnings. James, close behind in second place, did something the audience had never seen him do before -- he bet only a tiny fraction of his pot, not even enough to catch up to Emma's pre-final score. Across the country, Twitterers and newspaper columnists alike responded incredulously. He wasn't even trying! they wrote. He's throwing the game on purpose! Commentators tried to make sense of the motivation behind such uncharacteristically tame behavior as James's desire to go home to be with his young daughter or his unwillingness to destroy the previous all-time record, out of respect to the record-holder, Ken Jennings.
But, again, to those who have played the game, there was nothing inconsistent or irrational about James's small bet. If you're in second place going into the final question, and you have more than half of the leader's score, then the leader is virtually always going to bet enough so that, if she answers correctly, her score will be more than twice your pre-final score. Even if you bet everything you have from second place, if the leader gets the final question right, you cannot catch her. There's nothing you can do to win if the leader gets the final question right -- so you need to think about how to maximize your chances of winning if she gets it wrong. And if she gets it wrong, she loses the amount that she bet -- often, an amount that is big enough to drop her score below your pre-final score. In such a case, if you want to make sure that you will win if the leader answers incorrectly -- whether or not you answer correctly yourself -- then you want to make a bet small enough to stay ahead of the leader's final score if she gets the last question wrong. And that is why James bet small at the end. He was still playing to win.
I'm saying all of this not to minimize the accomplishments of a truly great Jeopardy! player, and not even primarily to teach people sound game strategies. What I'm hoping I've done is illustrate how the natural human inclination towards sense-making can easily lead to misjudgments and misinterpretations, especially when people know something well enough for it to seem familiar, but not truly intimately. Sense-making is the act of coming up with plausible rationalizations for why things are the way they are. It is not necessarily a bad tendency -- it is, after all, how scientific inquiry begins. But "plausible rationalizations", while comforting, are often inaccurate, and relying on them uncritically can be dangerous.
Our students and recent graduates preparing for the bar exam are just now in that space where they've seen enough of the structure and content of the bar exam for them to seem familiar, but not enough of them to really intimately how to do well on it. As they take practice tests and observe their fellow preparers and hear stories about people who performed well or poorly in the past, they might run into some of the same issues with sense-making that I described in everyday Jeopardy! viewers:
- Misjudging the ratio of cause to effect -- People are naturally impressed by outcomes, and when causes are not well understood, there is sometimes an assumption that big differences in outcomes can only be explained by big differences in causes. Many viewers saw James's high scores, nearly twice as high as previous records, and assumed that he was twice as smart or twice as quick as anyone who had played before him. In reality, he was probably only slightly more skillful than most of the folks he played against, but the nature of the game is such that, once a player gains a small advantage in scoring, he can exploit and multiply that advantage enormously. In a similar way, bar studiers who see big differences between themselves and their classmates, or who see only small improvements in their own performance over time, might not be familiar enough with the task of bar preparation to recognize the true magnitude of the causes of those differences. They might assume that small improvements (or plateaus) indicate that they have not learned much, when in fact they've made a great deal of progress and are nearing a tipping point of improvement. They might assume that they could never get scores as high as some classmates', because they are just not smart enough or don't have time to study as much as they'd need to, when in fact in absolute terms they might only need to improve, say, recall by ten percent. (Or the mistake could be in the other direction -- for example, assuming that adding fifteen minutes of flash card study every day will double their MBE score.) Over time and with practice and feedback, they should get better at making these judgments, but this early in the summer, we should be generous with lending some perspective to their rationalizations.
- Tendency to search for a single overarching cause -- Systems are complicated, and humans like simplicity. There is something comforting and manageable about identifying one thing -- like a super big brain or a revolutionary game strategy -- that totally explains how to achieve a particular outcome. Thus, we see graduates who insist that the key to doing well on the bar is religiously answering a certain number of MBE questions each night, or memorizing the contents of a particular outline (especially one that someone who passed the bar before them has endorsed). The truth is that the bar exam is multimodal and designed to test multiple skills and multiple dimensions of understanding. There is no single overarching cause of success on the bar, no matter how comforting that would be, and helping students to recognize early on the rich multiple approaches to success will help them proceed more realistically towards their goals.
- Tendency to attribute unexpected observations to new causes -- At a primal level, there is something unsettling about the unexpected, and one sense-making reflex is to assume that anything we haven't seen before must be a manifestation of some new element. James's unexpectedly small bet was completely explainable within the schema he used to make his earlier large bets, as applied to a new set of conditions, but viewers unfamiliar with that schema assumed that the small bet indicated a complete change in goals and strategies. In the same way, a student who sees an unexpected drop in practice test scores one week might tell themselves that it's because the testing room has changed or the weather is hotter or the lecturer that week is not as good. But the reality might simply be that the method of study the student had been using for the previous few weeks, which was fine when they had only covered three or four subjects, is now just not able to help the student handle the burden of six or seven subject's worth of materials.
Of course, it is sometimes true that new observations are attributable to new causes. The reason sense-making can be dangerous for students is not because every plausible rationalization is wrong, but because, without support, students may not be able to tell the difference between sound and unsound rationalizations. The students most likely to succeed on the bar, just like the contestants most likely to win on a game show, are those who learn enough before the big day about the challenge they face to be able to actually make good sense of what they are doing.
Tuesday, November 13, 2018
I have learned probably hundreds of tips, tricks, and techniques to improve one's performance on examinations. But there is only one that I learned with ten million people watching.
In 2005, I took the Florida Bar Exam -- my second bar exam, after passing the DC Bar Exam seven years earlier. When I returned to my car, the lone message waiting for me on my cell phone was not the expected call from my family. Instead, it was Glenn, from Culver City, California, calling to inform me that I had been selected to be a contestant on Jeopardy! -- the fast-paced quiz show in which contestants vie to answer 61 questions in 22 minutes.
The taping was to be in a month, and so I went right from cramming for the bar to cramming for trivial warfare. I knew there was no way I could study every possible subject that might come up on the show. At the same time, I felt like I ought to be "training". Today, there are websites that archive years of Jeopardy! clues, and old episodes on demand on Netflix, but these weren't available in 2005, so my main source of practice was watching the daily broadcast of the show at 7:30 p.m. And, perhaps because I felt that it was a rather precious resource, I decided that I wasn't just going to casually sit on the couch and shout out responses with the contestants. I decided that I was going to act like a contestant. Each contestant stands behind a podium and holds in one hand a pen-sized electronic button, and the first person to press that button after host Alex Trebek finishes reading the clue gets the chance to give the response -- famously, in the form of a question (e.g., "Who is George Washington?"). So, for a month, I tried to simulate their actions. I watched the show standing up, behind a living room chair. I held a clickable ballpoint pen, and practiced pressing the top button after Trebek finished reading each clue, and only then did I allow myself to call out a response in the form of a question. From time to time, I would feel a little goofy doing this, thinking, Isn't the show really about what you know? But I kept at it, because it seemed like the only way to really practice.
Finally, I arrived in California for the taping. Jeopardy! tapes five episodes in one day, a couple days every few weeks, so on the day on which I was scheduled to tape, I was herded into the studio with about a dozen other contestants. We spent a few hours signing documents and having make-up applied and learning all the rules and, most important and exciting, playing a few practice rounds on the set to familiarize ourselves with the equipment. I noticed some of the other contestants -- all clearly bright and as delighted as I was to be there -- seemed slightly awkward behind the podium. We all knew intellectually what to do, of course; we had all been fans watching the show for years, and we had just received a thorough briefing on what was expected of us. Even so, some contestants struggled to push their electronic button at the right time -- pushing it before Trebek was done talking would lock you out so that you could not answer, but if you waited too long, someone else would get in before you. Others got the hang of the button, with concentration, but then could not remember the responses they were trying to give. And there were times when contestants would press the button correctly, and give the right response, but forget to give it in the form of a question.
But when I went up on stage to practice, it was like I was standing back in my living room. I had practiced the timing of pushing my pen button so many times that, when it came time to press the real thing, I did not even have to think about it. I rang in quickly, focused entirely on recalling the information needed, and then gave the answer automatically in the form of a question. It worked in practice, and it worked in the actual taping. Yes, the show is about what you know, but it's important that nothing hinder you from demonstrating what you know. I won four games, and eventually came back to be a finalist in the Tournament of Champions.
In the years since, I have learned that what I had stumbled onto is known as "simulation training". It is a kind of practice that is not unlike the physical training that athletes do to develop muscle memory and automatic responses. In the context of quiz shows and law examinations, though, what makes simulation training particularly useful is not just the physical skills that it develops. What makes it useful is that it frees up mental space and focus for more complex thought. Not having to think about when to push the button and how to phrase my answer enabled me to devote full attention to reading the clue and retrieving the correct response.
Practicing to take examinations -- whether final exams or Bar exams -- can provide the same kind of simulation training, under the right conditions. Of course, students should write practice exams for other very good reasons, like improving legal analysis and uncovering weaknesses in subject matter knowledge, because law examinations should also be about what you know. But there is an added benefit when practice exams are done under conditions that imitate expected exam conditions. There are dozens of details and stimuli that students encounter consistently during an actual exam that, if unfamiliar, can demand valuable thought or cause detrimental distraction: dressing comfortably, locating a seat, timing bathroom use, logging into ExamSoft, calculating timing targets, contending with silence or noise, reading and following directions, cutting and pasting text, properly submitting responses, etc. Encouraging students to incorporate attention to these elements during their practice work, even when they are not really necessary, can help them improve performance, not because performance depends on finding a proper seat, but because being able to do so with almost no thought allows them to devote their mental energies to the tasks that really need them. Exam performance is about what you know, but it is important that nothing hinder you from demonstrating what you know.
Tuesday, July 17, 2018
This past week, I uncharacteristically watched a lot of reality television show competitions—mostly, Big Brother and Project Runway. Somewhere around hour six of my binge, I had a revelation. Bar exam studiers could learn a few things from the contestants on reality TV game shows. Both reality TV competitions and the bar exam studiers cram a lot of learning and formative assessment opportunities into a very short period of time. Those who learn and adjust succeed.
- Figure out which character you are.
Many reality TV contestants fit one of a few well-defined molds. For example, there is:
- The Leader – This person believes in themselves, even when others do not. They possess a confidence that is objectively justified. In Big Brother terms, this is called “The Rachel.” Everyone loves (and loves to hate) Rachel. This person will go quite far in the game.
- The Crier – This person cries, a lot. But have no fear. They will make it to the final found. They possess the substantive skills to succeed, and will succeed so long as they can focus on the task at hand.
- The Floater – This person fails to commit to any particular side. When presented with a hypothetical, they waffle. But, as Rachel Reilly of Big Brother’s Season 12 famously said “Floaters, you better grab a life vest.” If these folks pick a horse, then they undoubtedly survive another week.
- The Fainter – This person doesn’t take care of themselves. This person fails to get good sleep, eat well, or manage their stress. They will eventually faint due to exhaustion. This person can be successful if they regroup and care for themselves, properly.
- The Middle - This person is typically forgettable on reality TV. They don't win challenges, and they don't come in last place either. They don't cause drama; instead they just put their head down and play the game. This person will do just fine--even if no one is watching.
- The Weak Link – This person fails to win any challenges. This person is constantly placed “on the chopping block” because of their sub-par performances. This person is legitimately at-risk.
Bar exam studiers are no different. The key to success is to recognize the role you are playing and adjust accordingly. Just like on Big Brother, leaders, criers, floaters, middlers, and even fainters can succeed with the proper planning. Simply be self-aware and thoughtful about how you want the season to progress.
- Learn to cut off the outside world.
Everyone on reality competitions is isolated from the outside world. The competitors do not have access to social media or the internet. They rarely speak to loved ones. They live in a bubble. While I do not recommend such an existence for most people, most days. For law students studying for the bar exam, it is a potentially glorious plan. For optimal success, most studiers should stay singularly focused on their task – the bar exam. Forget about Facebook, Google, and Big Brother. I promise you, the internet will still exist in August. So, until then, just put up an “out of office” message and get studying!
- Develop a “showmance.”
On reality competitions, “showmances” and “bromances” are common. Showmances are formed when two contestants bond together—sometimes romantically—during the show’s short production. Two challengers lend support to one another for the purposes of mutual success in the competition. While showmances are sometimes mocked by the viewing audience, they do offer numerous strategical benefits to the competitors. Similarly, when studying for the bar exam, forming a deep, mutually beneficial relationship with another bar studier is advisable. The two studiers can help keep each other on task, and offer a sounding board for test-taking ideas and substantive rules. In short, look for a friend or significant other with which to commiserate and cerebrate.
Best of luck competitors! (Kirsha Trychta)
Tuesday, February 6, 2018
Grit: a noun, meaning courage and resolve; strength of character.
Numerous law review articles and research studies have discussed the importance of "grit" in law school success. But grit isn't unique to academia; rather grit is essential for success in virtually any intense, high-stakes environment, including the Master Sommelier's exam and the Olympics. Don't believe me? Watch SOMM and WINNING to see just what I mean. These two documentary movies (both currently available on Netflix) highlight the importance of grit, and help remind law students that:
1. You typically learn more from your failures than you do from your successes.
2. Getting back up and trying again, especially when you're exhausted, is essential.
3. You should strive for perfection, so that if you fall a bit short, you'll still be successful.
4. You should want to succeed for yourself, not to please someone else; internal motivation is key.
SOMM "takes the viewer on a humorous, emotional and illuminating look into a mysterious world—the Court of Master Sommeliers and the massively intimidating Master Sommelier Exam. The Court of Master Sommeliers is one of the world's most prestigious, secretive, and exclusive organizations. Since its inception almost 40 years ago, less than 200 candidates have reached the exalted Master level. The exam covers literally every nuance of the world of wine, spirits and cigars. Those who have passed have put at risk their personal lives, their well-being, and often their sanity to pull it off. Shrouded in secrecy, access to the Court Of Master Sommeliers has always been strictly regulated, and cameras have never been allowed anywhere near the exam, until now."
SOMM puts the effort needed to pass the bar exam into crisp perspective. Law students will undoubtedly identify with one, or several, of the study strategies employed by the sommelier hopefuls. Students may also appreciate the various outsiders' viewpoints offered by each test-taker's significant other.
WINNING is one film about "five legendary athletes. The compelling and inspiring story of the journeys of tennis champion Martina Navratilova, golf great Jack Nicklaus, Olympic gymnast Nadia Comaneci, track and field star Edwin Moses, and Dutch Paralympian Esther Vergeer. Through candid interviews and footage of their most exciting championship moments, WINNING reveals their dreams, challenges and triumphs and explores why some athletes achieve greatness."
WINNING highlights how impactful external pressures to succeed can be on one's psyche. Those who succeeded in the athletic arena did so because they personally wanted to win. Viewers takeaway a real appreciation for the concept that a genuine desire to prove to yourself that you can achieve your own goals will motivate you to wake-up early and stay late each day. In addition, WINNING teaches the importance of striving for perfection while also maintaining realistic goals and expectations. Students of the law, just like Olympians, are benefitted when they remain vigilant about identifying their personal weaknesses and looking for ways to improve upon those skills. (Kirsha Trychta)
Tuesday, October 17, 2017
During the first week of class I asked my students if they had any lingering questions that weren't resolved during Orientation. Several students inquired, "Where is the student lounge?" Admittedly our student lounge is somewhat difficult to find, with the entrance tucked between two vending machine on the second floor. I gave them directions and then jokingly described the student lounge as a place that only appears to those law students who already know of its whereabouts—which incidentally helps keep the room secreted from non-law students looking for a cool new spot to relax. Students aptly pointed out that I had also inadvertently described a key aspect of the Room of Requirement, a magical all-purpose space that featured prominently in the latter-half of the Harry Potter series.
[Sidenote: For those non-magical folk who aren’t familiar with Harry Potter, the Room of Requirement “only appears when a person has real need of it – and always comes equipped for the seeker's purpose. Any purpose.” For example, the Room of Requirement took the form of a bathroom for the headmaster when he was most in need, a training facility for Harry and the other members of his Army, and a storage room for many other students wishing to hide certain nefarious objects.]
The Potterheads were right, but if I had to pick the real Room of Requirement within the law school, it would undoubtedly be the Academic Excellence Center, especially in October. We never know who is going to walk through our door or what issue, question, or request they might bring with them. Just last week we fielded questions about academic advising, studying for midterm exams, debriefing after midterm exams, outlining, time management, moot court, legal writing, seminar papers, mental health resources, financial aid, new attorney swearing-in ceremonies, and summer employment, just to name a few.
I believe that my colleagues, while supportive of the Center, really don’t comprehend the varied roles that academic support professors play in the law school at any one time. To better capture the ever evolving list of activities within the Center, we recently installed a Survey Kiosk. The kiosk is actually an i-pad mounted on a chest-high stand near the door to the Center. The i-pad is locked using Apple’s Guided Access feature so that visitors can only access one webpage, namely a survey link.
We then created a 15-second survey that heavily relies on the use of skip logic. We now ask everyone to complete the survey following their visit to the Center. We also posted the survey link to our Facebook page, just in case someone forgets to complete the questionnaire before leaving the Center. The survey allows us to quickly capture the following information about each visit:
- Visitor’s class year (prospective student, 1L, 2L, 3L, or graduate)
- Who they visited within the Center
- Whether the meeting was a walk-in or by appointment
- Nature of the visit, i.e. the topic that was discussed
- Overall usefulness of the meeting, rated on a Likert Scale; and
- Any additional comments
In just two months, we have received roughly 200 real-time responses. This data has already allowed us to track which days of the week and weeks within the semester generate increased foot traffic, how well the Dean’s Fellows and Peer Writing Consultants are connecting with their classmates, and the types of services being most utilized. Unsurprisingly, 1Ls continue to make-up the bulk of our client base. But, we anticipate a sharp increase in 3L foot traffic in the spring semester, when the 3Ls turn their attention to applying for and sitting for the bar exam.
This real-time kiosk system will replace our end-of-the-semester evaluation, which historically has suffered from low response rates. The data should also be immensely helpful when we are tasked with completing annual Faculty Activity Reports and Performance Reviews next summer. Previously, we relied on a much less empirical system, consisting primarily of fuzzy memories, email inbox search results, and painstaking calendar reviews.
All-in-all, the Survey Kiosk has been a successful experiment, thus far. If you’re interested in doing something similar at your institution, you can purchase a basic i-pad and stand for under $1,000.00—making this an ideal project to submit for a technology grant, especially in light of its relatively low cost and easy implementation. Finally, we are also happy to share our survey setup with you; just ask. Unfortunately, we can't post the survey link here for you to view, because all of your curiosity clicks will create false responses in the data. (Kirsha Trychta)
Tuesday, July 4, 2017
Do you remember the game show “Supermarket Sweep”? (If not, click here for a short video.) The concept was simple: contestants had a short period of time to spend as much money as possible in a grocery store. The team with the highest total at the cash register won. Although the contestants were permitted to navigate the aisles in any manner they wished, the winning teams all tended to follow a predictable pattern to success. To win, contestants had to (1) pick-up numerous big ticket items, (2) grab a few specialty items, and (3) use their time efficiently.
Bar studiers can follow a similar strategy for success in July.
- Big Ticket Items
Whole hams, check. Super-sized tubs of condiments, check. Putting the “heavy lifting” items in the game show cart typically resulted in a good total at the cash register—but maybe not enough to win. Rather the contestants needed these big ticket items just to stay competitive. Likewise, bar studiers have to master certain subtopics, if they want to be competitive on game day. What’s the bar exam equivalent of a half-dozen fifty-pound bags of dog food? Answer: civil procedure jurisdiction, individual constitutional rights, contract formation, criminal homicide, evidentiary relevance, hearsay, and tortious negligence. According to the NCBE’s Subject Matter Outline, these big ticket items account for roughly one-quarter of the 200 MBE questions. Therefore, bar studiers are wise to pick-up these topics before moving on to the more nuanced specialty items.
- Specialty Items
If you watched the video link, you might have asked yourself: “Why is everyone buying bottles of Lysol cleaner?” After all, Lysol retails for less than $10.00. Well, on that particular episode, Lysol was a specialty bonus item—an item that could result in a larger than expected payoff at the cash register (provided the contestant could find the small bottle in the vast store). The bar exam is full of specialty items too. Bar studiers must identify select, discrete subrules and exceptions that they can master very quickly, resulting in a positive return-on-investment. A bar studier is not going to be able to acquire every unique subrule or exception. Rather the bar studier must be strategic about spotting those particular rules that they can “put in their shopping cart” with just a small amount of targeted effort. For example, spending an hour to learn the difference between spousal immunity and confidential marital communication is time well spent. Conversely, spending three days spinning your wheels on that same question is not. After all, the clock is ticking.
- Time Management
With only a few minutes to shop, contestants must use their time wisely. Similarly, bar studiers have to maximize their study time in July. Developing a detailed plan of attack for July tends to be the step most bar studiers skip; bar studiers mistakenly believe that all study hours are created equal. To ensure maximum time efficiency, bar studiers should use the results of their midterm exam to develop a day-by-day plan of attack for July. Bar studiers should start by deciding which topics will be tackled each day by identifying which topics will result in the biggest gains. A solid plan will surely include sufficient time to acquire any big tickets items that are not already in the bar studier’s cart and also include time to not only identify, but also learn select specialty rules. The plan should also incorporate a mix of non-negotiable language (“No matter what, I am going to …”) and some contingencies (“If time permits, I will…” or “I need X or Y, but would love both.”). I’ve attached a Download Calendar - Bar Exam July 2017 to help the bar studier start crafting his/her individualized plan for success. Once the plan is complete, the only thing left to do is execute with game-show-winning precision.
P.S. Happy Fourth of July! (Kirsha Trychta)