Stop Making Sense

Yesterday, the quiz show Jeopardy! enjoyed its highest ratings in more than 14 years, <spoiler> on the day that 32-game winner James Holzhauer lost to librarian Emma Boettcher and fell just short of breaking the all-time record for most money won during regular play.  (Sadly, James walked away with only $2,464,216.)  My friends in the trivia community have been watching James's exploits with various mixtures of admiration, envy, bemusement, and exasperation.  The latter two emotions have been prompted not by James himself, but by the sense-making reactions of casual viewers and the media to his success, and then to his defeat.

James racked up an intimidating number of high-scoring games -- including all of the top-ten highest-scoring games of all time -- and he sometimes won by six-figure margins.  To a lot of pundits, these overwhelming victories suggested a new and singular player: either someone with unmatched, superhuman genius, or someone who had come up with a novel strategy that had "broken" the game forever.  From the perspective of a lot of fans at home, this made sense.  How else could someone achieve such never-before-seen results without some sort of mystical secret ingredient?

But to a coterie of former players and dedicated aficionados, there was nothing mysterious or unduplicable about James's style of play.  He is a tremendous player, to be sure, certainly among the best.  But the skills he brought to the game are pretty much the same skills other great players have exhibited before.  He knows a lot of trivia; he is very adept at using the signaling device to snatch the opportunity to answer first; he understands the optimal strategies for choosing clues and making bets.  His historically high scores are due mainly to a gutsy willingness to risk losing all or most of his pot by making big bets that, when successful, have left him with insurmountable leads.  In the past, even the strongest players played more conservatively, hedging their bets so a wrong answer wouldn't take them out of the running.  But James is a professional gambler, and he decided to maximize his return by maximizing his risk.  This was a choice, not an aptitude, and anyone playing against him would have the capacity to make the same choice. 

In fact, in yesterday's game, Emma did just that, making her own big bets to take a lead that James could not overcome.  When the game hinged on one final question -- one that all three contestants would have the chance to answer, and on which each would have to make a wager -- Emma, in the lead, bet most of her accumulated winnings.  James, close behind in second place, did something the audience had never seen him do before -- he bet only a tiny fraction of his pot, not even enough to catch up to Emma's pre-final score.  Across the country, Twitterers and newspaper columnists alike responded incredulously.  He wasn't even trying! they wrote.  He's throwing the game on purpose!  Commentators tried to make sense of the motivation behind such uncharacteristically tame behavior as James's desire to go home to be with his young daughter or his unwillingness to destroy the previous all-time record, out of respect to the record-holder, Ken Jennings.

But, again, to those who have played the game, there was nothing inconsistent or irrational about James's small bet.  If you're in second place going into the final question, and you have more than half of the leader's score, then the leader is virtually always going to bet enough so that, if she answers correctly, her score will be more than twice your pre-final score.  Even if you bet everything you have from second place, if the leader gets the final question right, you cannot catch her.  There's nothing you can do to win if the leader gets the final question right -- so you need to think about how to maximize your chances of winning if she gets it wrong.  And if she gets it wrong, she loses the amount that she bet -- often, an amount that is big enough to drop her score below your pre-final score.  In such a case, if you want to make sure that you will win if the leader answers incorrectly -- whether or not you answer correctly yourself -- then you want to make a bet small enough to stay ahead of the leader's final score if she gets the last question wrong.  And that is why James bet small at the end.  He was still playing to win.

I'm saying all of this not to minimize the accomplishments of a truly great Jeopardy! player, and not even primarily to teach people sound game strategies.  What I'm hoping I've done is illustrate how the natural human inclination towards sense-making can easily lead to misjudgments and misinterpretations, especially when people know something well enough for it to seem familiar, but not truly intimately.  Sense-making is the act of coming up with plausible rationalizations for why things are the way they are.  It is not necessarily a bad tendency -- it is, after all, how scientific inquiry begins.  But "plausible rationalizations", while comforting, are often inaccurate, and relying on them uncritically can be dangerous.

Our students and recent graduates preparing for the bar exam are just now in that space where they've seen enough of the structure and content of the bar exam for them to seem familiar, but not enough of them to really intimately how to do well on it.  As they take practice tests and observe their fellow preparers and hear stories about people who performed well or poorly in the past, they might run into some of the same issues with sense-making that I described in everyday Jeopardy! viewers:

• Misjudging the ratio of cause to effect -- People are naturally impressed by outcomes, and when causes are not well understood, there is sometimes an assumption that big differences in outcomes can only be explained by big differences in causes.  Many viewers saw James's high scores, nearly twice as high as previous records, and assumed that he was twice as smart or twice as quick as anyone who had played before him.  In reality, he was probably only slightly more skillful than most of the folks he played against, but the nature of the game is such that, once a player gains a small advantage in scoring, he can exploit and multiply that advantage enormously.  In a similar way, bar studiers who see big differences between themselves and their classmates, or who see only small improvements in their own performance over time, might not be familiar enough with the task of bar preparation to recognize the true magnitude of the causes of those differences.  They might assume that small improvements (or plateaus) indicate that they have not learned much, when in fact they've made a great deal of progress and are nearing a tipping point of improvement.  They might assume that they could never get scores as high as some classmates', because they are just not smart enough or don't have time to study as much as they'd need to, when in fact in absolute terms they might only need to improve, say, recall by ten percent.  (Or the mistake could be in the other direction -- for example, assuming that adding fifteen minutes of flash card study every day will double their MBE score.)  Over time and with practice and feedback, they should get better at making these judgments, but this early in the summer, we should be generous with lending some perspective to their rationalizations.
• Tendency to search for a single overarching cause -- Systems are complicated, and humans like simplicity.  There is something comforting and manageable about identifying one thing -- like a super big brain or a revolutionary game strategy -- that totally explains how to achieve a particular outcome.  Thus, we see graduates who insist that the key to doing well on the bar is religiously answering a certain number of MBE questions each night, or memorizing the contents of a particular outline (especially one that someone who passed the bar before them has endorsed).  The truth is that the bar exam is multimodal and designed to test multiple skills and multiple dimensions of understanding.  There is no single overarching cause of success on the bar, no matter how comforting that would be, and helping students to recognize early on the rich multiple approaches to success will help them proceed more realistically towards their goals.
• Tendency to attribute unexpected observations to new causes -- At a primal level, there is something unsettling about the unexpected, and one sense-making reflex is to assume that anything we haven't seen before must be a manifestation of some new element.  James's unexpectedly small bet was completely explainable within the schema he used to make his earlier large bets, as applied to a new set of conditions, but viewers unfamiliar with that schema assumed that the small bet indicated a complete change in goals and strategies.  In the same way, a student who sees an unexpected drop in practice test scores one week might tell themselves that it's because the testing room has changed or the weather is hotter or the lecturer that week is not as good.  But the reality might simply be that the method of study the student had been using for the previous few weeks, which was fine when they had only covered three or four subjects, is now just not able to help the student handle the burden of six or seven subject's worth of materials.  

Of course, it is sometimes true that new observations are attributable to new causes.  The reason sense-making can be dangerous for students is not because every plausible rationalization is wrong, but because, without support, students may not be able to tell the difference between sound and unsound rationalizations.  The students most likely to succeed on the bar, just like the contestants most likely to win on a game show, are those who learn enough before the big day about the challenge they face to be able to actually make good sense of what they are doing.

[Bill MacDonald]