The Charisma of Numbers

Today's Washington Post has a fascinating and disturbing article about the company HireVue and its signature product, an artificial intelligence hiring system through which employers can set up automated "interviews" with prospective employees.  The system "uses candidates’ computer or cellphone cameras to analyze their facial movements, word choice and speaking voice before ranking them against other applicants based on an automatically generated 'employability' score."  Based on these scores, HireVue's clients -- which include large organizations like Unilever and Goldman Sachs -- can choose which candidates they would like to bring in for actual human interaction.

The growing reliance of employers on HireVue and its competitors suggests several issues of interest to law students.  Can we expect that someday soon, they too will be forced to welcome their new computer overlords by developing another set of skills -- namely, the art of using just the right expressions and intonations to appeal to the interviewing algorithm?  How do we even know what appeals to that algorithm, and whether the appealing features actually bear any relationship to job performance, if HireVue releases no information about what it is measuring, what it assigns value to, or, indeed, even what a candidate did wrong?  (The mystery and validity issues echo some complaints about the UBE, but at least bar examinees are told their scores.)  Like it or not, this Pandora's boxing ring is now open, and it's only a matter of time until young attorneys are sent in to altercate.

To get some perspective on the rigor of the HireVue system, the Post reporter spoke to researchers in applicable fields, including Luke Stark, an AI researcher who was

skeptical of HireVue’s ability to predict a worker’s personality from their intonations and turns of phrase. . . . Systems like HireVue, he said, have become quite skilled at spitting out data points that seem convincing, even when they’re not backed by science. And he finds this “charisma of numbers” really troubling because of the overconfidence employers might lend them while seeking to decide the path of applicants’ careers.

The charisma of numbers is something I feel I run up against over and over again.  And I say this as a person who values data and statistics!  I believe it is difficult to make consistently effective decisions or to take wise action without obtaining and evaluating relevant numerical information.  And, true, in a field in which our success is largely measured numerically (GPAs, retention rates, bar passage rates), numbers can possess either star power or infamy.

But, notwithstanding their dazzle and clout, numbers should only be powerful if they are attached to something meaningful.  If they are being misused or misunderstood, that can mean mistaking the sizzle for the steak.   Figures can be seductive when they seem rounded, or extravagant, or provocative, or revealing.  It's easy to jump on the conspicuously appealing numbers -- the highest GPA, the apparently significant pattern in MBE scores, the increase in median starting salaries -- just as it's easy to be attracted to the confident, well-spoken cutie who walks into the party.  But the GPA might be based on a disproportionate number of generously graded courses; the MBE pattern might be statistically insignificant; the median salary increase might represent slippage, not advancement, if similar schools are seeing an even larger increase.  Causes, reliability, and context all matter.

The danger of the charisma of numbers is that sometimes, even when a person is only looking at the surface, they don't feel like they are being shallow, because numbers are supposed to be scientific and rational.  We need to remember, and teach our students and colleagues, that, even with the most alluring numbers, you should really spend some time with them first, get to know their flaws and idiosyncrasies, before you commit to them.

[Bill MacDonald]