The Legal Whiteboard

A Reply to the Empiricists at NWU Law

My blog post from last week, "Too Good for BigLaw: The Statistician Edition" has resulted in a minor kerfuffle with some of the distinguished empiricists at Northwestern Law.  See Dan Rodriguez, Law School Sorting and the Partnership Track:  Northwestern Empiricists Weigh In, Word on the Streeterville [The Blog of NWU Law Dean].  NWU Law folks were not impressed with my analysis.  Dan Rodriguez was gracious enough to send me the link at the same time his post, quoting the views of his colleagues, went live.  He has also encouraged me to reply publicly. 

I am happy to do that.  Let me start with big picture issues.  Then, for those folks with the curiousity and stamina to wade through arcane details--and experience tells me this is a small group--I will directly address, point by point, the the issues raised by Kate Litvak and Max Schanzenbach.  But at the outset, I will say that I am not conceding any ground.

Big Picture

First off, it is important to understand why this topic generated so much traffic.  See Above the Law, ABA Daily Journal, TaxProf, Instapundit,  Adam Smith. 

It all started with a provocative blog post by Vivia Chen, the columnist for The Careerist.  Vivia reviewed hiring and promotion data from the NLJ 250 Law School Hiring Survey and noted that elite law school graduates were becoming partner in very low numbers when compared to the hiring pipeline. Vivia editorialized on the numbers in a way that played into readers' fragile egos and insecurities.  Of course, that is her job, which she does very well.

In a nutshell, here is why people care -- or more precisely, get anxious -- about this topic:  it is conventional wisdom that graduation from elite law schools produces better career outcomes.  When that expectation is countered by actual marketplace data, people are surprised.  See, e.g., Bruce MacEwen,  "The Best & The Brightest" at  Adam Smith Esq. (leading blog on law firm economics).  Surprise is the first reason this issue got so much play.  Emotion is the second.

Emotion matters because very few lawyers and law professors are dispassionate on this topic.  When it comes to conventional wisdom on law school pedigree, we all have horses in the race.  Because we are human beings, we lawyers and law professors don't wait for balanced market data to develop our own entrenched worldviews.  When the conventional wisdom favors us, we go with it -- albeit we aren't really conscious this is happening. So when data upset the apple cart and potentially make us look complacent, our passions get aroused. 

The folks at Above the Law have built a entire business model around such predictable lawyer foibles.  The more chum thrown in the water, the higher the ad revenues.  It's just that simple.

Vivia's primary point, stated through metaphor, is that regional schools (such as Chicago Loyola) seem to be making partner at higher rates than the elite schools (such as Chicago).  This is a reasonable inference because the ratio of associates hired to partners promoted appears to be consistently high for elite law schools and very low for a large number of regional law schools.  This very point was made independently by Bruce MacEwen, who is a very sophisticated guy who advises law firms on strategy.

That said, there was ample opportunity for readers to draw spurious inferences from Vivia's metaphor-driven blog post.  Thus, to avoid any school-specific claims (a 1-year crossectional sample is not suitable for such a purpose), I pooled the schools by U.S. News ranking, drawing a line between elite and non-elite at the T14 mark.  Why T14?  Because these schools have played a closed loop of musical chairs for 20 years in the U.S. News rankings.  These schools would be viewed by most employers as "national" law schools.  

Here is what the data showed:

• Pipeline in:  53.7% T14, 46.3% non-T14
• Partners Promoted: 29.4% T14. 70.6% non-T14.

That is, well, an enormous skew. In 2011, for every 5.43 elite grads hired, a senior associate from an elite school makes partner.  For non-elite schools, that corresponding statistic is 1.95.  Vivia found these numbers surprising and somewhat counter-intuitive.  So did Bruce MacEwen, Above the Law, ABA Journal, etc.

There are ways to break down these numbers to gain additional insights, but the key point here is one of magnitude.  Elite law graduates are supposed to be smarter and more capable -- no one expects these folks to be on short side of any race, tournament or desired outcome.  The magnitude of hiring/promotion gap is the surprising fact that needs to be explained.

I had observed roughly the same skew several years ago (pooled 2007 and 2008 data) and alluded to it in this article, "Why is the Job Market Changing," Nat'l Jurist (Nov. 2010).  I also follow other relevant studies, such as The After the JD, which have noted differences between elite and non-elite graduates.   So I had a head start in thinking through possible explanations. I thus offered five theories, all of which could work in concert, to explain the large skew in the data:

1. Selection effects
2. Differences in first jobs
3. Intergeneraional privilege
4. Influence of admission criteria on the associate pipeline
5. "A Better Plan B" for elite grads

So, to be very clear, I am not using the NLJ 250 data to support the above theories.  It is the reverse: I am offering the above theories as a likely explanation for the very large skew between elite and nonelite grads.  Framed as a open-ended research question, it might be written, "why are elite grads not becoming BigLaw partners in numbers commensurate with hiring patterns and general presumptions of their higher ability?"  That is a mystery and a puzzle.

Statistics Minutiae  [After the jump ...]

Each of my five theories could be its own independent research study.  But that's a side show.  Testing the theories will not change the skew.  Don't forget the big picture -- we have a  counterinituitive data point that needs to be explained. 

A better approach is to triangulate what we know from historical patterns, published studies, the data itself, theory and commonsense and apply Occam's Razor to find a simple set of explanations that explain all the data points.  My blog post never made any strong claims on relative importance or validity of my theories.  I was just trying to formulate a reasonably plausible set of explanations.  When something is highly contestable and important, then we can go to the trouble of formal hypothesis testing.  But we have five theories here.  It is unlikely all five are wrong.  If they were, we'd need a sixth killer theory, heretofore unknown, to explain the skew.  Again, Occam's Razor.

To my mind, this is the source of disagreement between myself, Kate and Max:  they are focused on poking holes in my theories, but I want a satisfactory explanation for this very large skew.  Max doesn't offer any explanation, and Kate's explanations quickly wilt when Occam's Razor is applied.

From this point forward, this post may be of interest to readers with a large appetite for the inner workings of inferential statistics.

Here is Kate's macro level critique:

In short, the averages he is discussing cannot tell us anything useful. The main problem is selection effects combined with the level of data aggregation: (1) no individual-level data on hired/promoted lawyers; (2) all top-250 law firms are grouped together; (3) all non-T14 law schools are grouped together.

Unfortunately, I don't know what "useful" means here.  My analysis unearths some data points that are surprising and counter-intuitive -- so says the blogosphere.  I think the averages, viewed in light of my five theories, are potentially useful if I were a law firm partner and wanted to understand unwanted attrition in my law firm-- a potential multimillion dollar problem. If I am law professor, I might also find it useful to adjust my worldview so that fits surprising and counter-intuitive data points.  After all, we are occasionally called upon to provide career advice. 

Regarding Kate's three points on data limitations, it is possible break out the data as Kate suggests.  And when broken down, it boosters rather than detracts from my initial analysis. Let me be clear, however, on the precise sample I am working with:  

• It is called the NLJ 250 Law School Hiring Survey, but the sample is not complete.  Here is what it includes:  155 firms supplied data.  127 are AmLaw 200 (AmLaw 200 is based on revenues), and 27 firms not in the Am Law 200, but in the NLJ 250 (NLJ 250 is based on lawyer counts).
• Of the 127 AmLaw firms in the sample (63.5% response rate), median profits per partner is $970,000 ($1.2 million average) versus $807,000 ($1.1 million average) for the 73 that did not supply data.  So the sample is not comprised of the less profitable firms; they are, if anything, more profitable. The difference in means, however, is not statistically significant.
• The sample firms are larger: 567 median, 819 average for sample; 401 median, 472 average for the missing firms.

Based on the above, the sample appears to be a reasonable representation of the BigLaw universe.  Further, we know the directionality of the missing data, which is useful for interpreting any results. 

[One more sampling point: Kate objects to grouping all non-T14 schools together.  I disagee.  There is no point in breaking down the law schools further by U.S. News tier -- Andy Morriss and I documented that during the halycon days of the mid-2000s, outside the Top 25 or so, large law firm jobs were limited to between 0 and 15% of NLJ 250 entry level jobs.  In contrast, T14 was typically well north of 50%, even with large numbers of judicial clerkships.  There is one relevant divide here: regional versus national.  A few regional schools, such GW, Fordham, Emory, UCLA, USC and Texas, have a strong foothold for students in the top 30% or 50%.  After that, the grade cutoffs turn all other law schools to near decimal dust in terms of market pull.  I have parsed the data in previous years--these in-between schools are, well, in-between the regional and nationals in terms of outcomes.  Breaking them out won't tell us anything.]

As I noted in the original post, and pointed out by Kate, we almost certainly have massive selection effects at work here.  We know from other research that T14 grads gravitate to the most prestigious firms where the washout rates are very high.  See Zaring & Henderson.  The value of the partnership (higher profits) in combination with better outplacement prospects may make this tradeoff rational.  Sure enough, there are large selection effects: 61% of the T14 graduates in the sample joined firms in 2011 in the Top 50 as measured by Profits per Partner (PPP).  This supports the First Jobs theory. 

That said, even in the high prestige firms, non-elite grades are getting more than their pipeline's share of the partnership promotions (or, stated differently, T14s are leaving in larger proportions): 

• Among the 25 most profitable firm in the Am Law 200, T14 accounted for 76.0% of associates hired, compared to 24.0% for non-elites  graduates.  But for partnership, the numbers fall to 60% versus 40%.
• Same pattern if we broaden to 50 most profitable firms:  71.4% T14 hired, 28.6% non-T14;  for partners, 55.6% versus 44.4%.
• The pattern continues for AmLaw 200 firms outside the Top 50 PPP bracket:  40.5% T14 hired, 59.5% non-elite; for partners, 24.0% T14, 76.0% non elite. 

So, using the underlying raw numbers, let's translate these figures into ratios of associates hired to partners promoted:

• Top 25 PPP:    T14 = 10.1, non-T14 =3.9
• Top 50 PPP:    T14 = 4.8, non-T14 = 1.9
• Non Top 50:    T14 = 3.9, non-T14 = 1.8
• Non-AmLaw:   T14 = 2.0, non-T14 =0.8

In every bracket in the NLJ 250/AmLaw200, the T14 get superior hiring access but much lower rates of partnership.  Again, this is Ted Seto's research in another form. 

We can get more granular.  Among the 79 firms in the 2011 sample that (a) hired T14 and non-T14 associates, and (b) promoted at least one T14 lawyer to partner, there were 51 firms (65%) where the non-14 were getting promoted at rates that exceeded their pipeline percentages. Further, compared to the remaining 28 firms, these 51 firms had higher average Revenues per Lawyer ($772,000 versus $711,000) and higher average Profits per Partner ($1.2M versus $1.0M).  Perhaps non-Top 14 partners are good for business!

Had Kate had access to these data, she may not have replied so boldly. After hiring, T14 does not fare well in any bracket.  Again, why is that?

I am not saying the above data supports my five theories; rather, I am saying some combination of my five theories likely explain a good portion of the large, counter-intuitive gaps between elite and non-elite law school graduates. Here is the story in five bullet points:

• Selection Effects. If you are at a T14 school, BigLaw jobs are relatively plentiful. Interview well and you get the big paycheck.  In contrast, the Non-T14 crowd are comprised of a larger proportion of people who had to work very hard to get their foot in the BigLaw door.  These types of behaviors generally don't end after one gets hired. They become habits.
• First Jobs.  The T14 are going to the most prestigious firms where the washout rates are high. We showed this above. But the T14s washout in greater proportion than their Non-T14 peers.  So, by itself, this can't explain the overall NLJ 250/AmLaw 200 gap.
• Intergenerational Privilege.  We know from the After the JD study that elite law schools grads come from more affluence families.  See Dinovitizer & Garth, Lawyer Satisfaction in the Process of Structuring Legal Careers.  This is also the story line of Thomas Mann's Buddenbrooks novel -- affluence can produce less disciplined behavior among our progeny.
• Ratings-Driven Admissions Criteria.  We now throw away personal statements, resumes and letters of references when making admission decisions.  20 or 30 years ago we consulted commonsense when making admissions decisions.  So the pipeline to elite law schools, and hence to the NLJ 250, has likely been affected in deleterious ways.
• "A Better Plan B."  Elite grads have more options, presumably in gov't and business.  This is obvious.  But it is too speculative a category to explain the whole elite/non-elite gap, which is very large.

Kate's critiques in order:

• Selection effects:  Kate says I can't rule out that that the non-T14 crowd is getting hired and promoted because they have, say, important client connections.  Why would I try to rule this out?  It is a de minimis effect at best.  It fails Occam's Razor.
• First Jobs.  Kate says "probably true."  We can stop right there.  We are looking for the most plausible explanations.  Nonetheless, the data above are also responsive to rest of Kate's criticism--the T14/non T14 pattern holds up in every part of the BigLaw hierarchy.
• Intergenerational Privilege.  Kate thinks I don't have the data to document T14's higher family wealth.  But The After the JD research does have these data.  Further, Garth and Dinovitzer make an empirical case that it affects satisfaction in the large firm environment.  In short, we have a literature on this topic -- I am merely pointing to it.
• Rankings effect.  Kate thinks I lack any concrete data to make this claim.  She is right.  But we have all watched law schools make insane trade-offs in admissions to goose their rankings.  I am making the leap that this affects the labor pool in unhealthy ways.  Supply chain analytics are important in other industries.  Law firms are waking up to this.
• "A Better Plan B."  We both agree that we need better data to make sweeping claims on this one.  We also agree that elites will have the edge.  It is just a matter of measuring the size and direction (gov't, business, public interest, academia) of the edge.

What I find puzzling about Kate's comments is that she never addresses the large gap between T14 hiring and promotions.  To my mind, that is the storyline -- the puzzle that needs to be unraveled.  If she doesn't like my theories, then she should formulate something simpler and more plausible.  But let's focus on explaining the gap.  Where is the sixth killer theory?

Max makes an entirely different point.  Max writes:

What to make of Bill’s piece?  I don’t think it tells us much. Bill is looking at the conditional probabilities. ...   If firms are willing to take fewer risks on Loyola students (e.g., we hire the number one at Loyola and that’s it), then it’s not surprising that the conditional probability is higher. In my view, this result likely suggests a bias in favor of T14 students (perhaps for rational reasons regarding search costs), and an even stronger reason to attend an elite school if one’s goal is to be a top 250 law firm partner.

Max is certainly right.  If a student's goal is to be a BigLaw partner, he/she you will have an easier time getting hired if he/she attends an elite school.  You are the same person after all, so your behavior, motivation and ability can be treated as a constant -- getting your foot in the door is key. 

But Max overlooks the perspective of a law firm, which might look at the data and say, "My god, we have been calculating these conditional probabilities all wrong.  Clients aren't paying for training any more.  We can't afford this revolving door of T14s, who don't seem as engaged as the less elite grads."  So I think the data say quite a lot.  The huge presumption favoring elite grads created the Bi-Modal distribution; this propensity was arguably the beginning of the end of the BigLaw model where firms just passed off higher salaries onto clients.  No more.

Conclusion

Just like Phil Corboy, who could not get a job on LaSalle Street because he was Catholic, or Sandra Day O'Connor, who could not break into the Arizona corporate bar because she was a woman, or Joe Flom who was shut out of Wall Street because he was a Jew, there is always a contemporaneous narrative that justifies the status quo.  But eventually the prejudice becomes too expensive to bear.  And we change. 

The corporate law firm sector is now competing over market share for the first time ever.  Whether elite grads are better or worse hiring bets is now becoming a matter of strategic importance.  Unlike in earlier years, firms have to think twice before paying a snobbery tax for hiring criteria disconnected from, or even negatively correlated with, performance.

People's self image are all bound up with this topic. It has very little to do with rationality, and much more to do with identity and emotion.  But the takeaway is very simple: Pedigree does not reflect a natural ordering; it is socially constructed, initially based on substance and merit, but eventually perpetuated by inertia and vested interest.  Especially during times of transition and stress, it too can crumble.  This forebodes the regeneration of substance, and that is something that all of us should welcome.

Counter-Intuition in the Legal Academy (and Life)

Posted by Jeff Lipshaw

Larry Kramer, Stanford's dean, was in Boston yesterday, and we had lunch.  (Full disclosure:  I am an unabashed Larry Kramer fan.)   I've previously lauded the efforts of Larry and the Stanford faculty to reform the second and third year curriculum, primarily to provide more interdisciplinary and joint degree opportunities as between the law school and other arms of the university, like the business school, med school, engineering school, and so on.   Larry reported that the program was moving ahead nicely, but that students were still tending to select more traditional classes, presumably in reaction to the contracting job market over the last couple years, and on the theory that a more traditional transcript would be more attractive to employers in a tight market.

We agree this was a natural flight to safety, but that the more appropriate reaction, perhaps counter-intuitive, would be to double-down on the interdisciplinary opportunities.   In our conversation, I suggested to Larry the following as an illustration of his point.

A free-lance reporter called me the other day wanting to understand a contract lawsuit recently filed by the operator of the Park City ski resort in Utah.  In a nutshell, the operator as lesseee had a fifty year lease, and apparently the right to renew it for another fifty years.  I don't have the underlying contract, only the complaint, but it's clear the party were negotiating amended terms of a renewal, and in the process, the operator created some ambiguity over whether it had exercised the renewal option precisely and literally according to the terms of the agreement.  It also appears that the lessor said something in the negotiations to the effect that the lessee hadn't properly renewed, and was subject to being evicted.  Nevertheless, the lessor has not yet sought to terminate the lease or the occupancy, and is still collecting rent.  Moreover, the case is a declaratory judgment action filed by the lessee, seeking a determination that it indeed renewed the lease.

Now I think this may well be a lovely case for demonstrating the interplay of express and implied conditions as a matter of doctrine, but it seems to me a perfect example of how little you know if all you know is contract doctrine or even litigation strategy.   In the past, I've used the metaphor of a ricocheting bullet for the unintended consequences of a hardball action, particularly the filing of a lawsuit.  Deciding to escalate, and to escalate publicly, invokes not just legal doctrine (i.e., a prediction how a third party decision maker might rule), and not just trial strategy (i.e., how do we put ourselves in the best position to win the litigation), but business strategy, public policy, public relations, psychology, finance theory, and other disciplines, all mediated by a well-grounded view of human nature.

I don't think a law student gets more "practice-ready" or more attractive to an employer by taking more old-style doctrinal classes that do things like focus solely on the kernel of legal doctrine that may lie at the heart of a problem.  I can think of a bunch of metaphorical terms for this reaction to change, crisis and uncertainty:  bunker mentality, cocooning, circling the wagons, etc.  Of course, I'm already in hot water with the contract professors union for suggesting that the universe as we have come to know it will not end with a reduction from six credit hours of contract law doctrine in the first year.  That too strikes me as circling the wagons.

Flights to safety and control are the typically intuitive (and, in my view, wrong) reaction to change and uncertainty (e.g., banning laptops or cutting off wireless access as the means by which teachers seek to control the attention of students in a classroom).   We already know or what we've previously done feels safe and controlled.  Being interdisciplinary is, by definition, unsafe and uncontrolled, by virtue of the fact that there are no intra-disciplinary authorities to tell you that what you are doing is okay.  (If there's an interdisciplinary authority, then the interdiscipline has really become its own discipline.)  But is the intuitive flight to safety and control the right one?

[Cross-posted at Legal Profession Blog.]

The Eye Contact Revolution

Several years ago, a student wrote in my teaching evaluations that I had a hard time with eye contact; the context was office hours.  I had not really thought about it,  but upon reflection, I realized that he/she was right.  More recently, in organizing peer evaluations of student presentations in my Legal Professions class, the topic of eye contact kept coming up.  And I agreed.  Students who maintained audience eye contact seemed more confident, competent and memorable.

This morning, via the Big Think, I read that we are -- or could be, if we got our act together -- in the midst of a eye contact revolution.  Here is an excerpt:

Michael Ellsberg, author of  The Power of Eye Contact: Your Secret For Success in Business, Love and Life, and The Education of Millionaires: It’s Not What You Think, and It’s Not Too Late thinks most of us are getting it wrong, eye-contact wise. Good eye contact, he argues, is as important to connecting meaningfully with others (at play or at work) as is, say, bathing. What’s needed, says Ellsberg, is a soft, open gaze – one that neither intimidates nor communicates anxiety on the part of the gazer.

[Why does this matter?]

Ellsberg’s “eye contact revolution” is aimed not only at careerists, but at the social and spiritual heart of our glowing screen-obsessed world. Looking up from our smartphones and into each other’s eyes, he believes, will increase the quality of every aspect of our lives.

Ellsberg suggests trying out his techniques in low stakes interactions, such as sales clerks and waiters and waitresses. 

This reminds me of Warren Buffett's experimentation with Dale Carnegie's techniques for making friends and influencing people.  In his teens, Buffett practiced the Carnegie methods for several weeks and tracked the results.  Then he reverted to his former self and agained tracked the data.  At the end of the process, he compared the statistical output and realized that the variation could not be explained [reasonably] by chance.  So thereafter, he committed himself changed.  See Alice Schroeder, The Snowball: Warren Buffett and the Business of Life (2008).

Below is a Big Think video of Ellsberg discussing his ideas.

[Posted by Bill Henderson]

Considering the Med School Analogy

Posted by Jeff Lipshaw

Bill is as usual too kind, but indeed we have been friends since that fateful lunch in Bloomington six years ago this month.  I'm immensely flattered that Bill and Andy have seen fit to hand me the dry erase marker, and the first thing I did with it was draw a far more accurate self-portrait than the picture Bill managed to dig out of some P.R. file somewhere.

I flew out to San Francisco from Boston yesterday.  About an hour into the flight, the flight attendant asked over the PA whether there was a medical professional on board, and I saw somebody up near the front of the plane hit the call button.  I had been discussing this very situation with my third-year medical student son a couple weeks ago, reacting to a post from my friend Howard Wasserman at PrawfsBlawg.  Howard told the story of a newly-graduated doctor flying in fear of being called on in such a situation, believing that she didn't have the practice skills to intervene.  Howard's point, if I can restate it, was that pointing to medical education as the model for "practice-ready" lawyers was a mistake, and that newly-minted doctors were just as "unpractice-ready" as newly-minted lawyers.

I wouldn't want a newly-minted doctor to perform neurosurgery on anybody I liked, but I thought Howard had overplayed the meaning of the anecdote.  My son advises me that invariably what you need on an airplane is an EMT, not a doctor.  (He tells me all med students are CPR-trained, but he'd be far more competent to diagnose a cough or a rash than to apply the CPR.)  If the only person on the plane to step up was a third year med student, I'd still think that person was more qualified to act as a GP doctor, relatively speaking, than the typical doctrinally-trained law grad would be to act as a GP lawyer.  Others may feel differently.

Having said that, and having observed my own education, my son's, and the education I'm now employed to help provide, I am still convinced that medical education, apart from internship and residency, effectively creates "doctors" by the time med students are in their third year. The great bulk of medical education takes place in three years - the first two in the classroom (culminating in the USMLE Step 1 exam), and the fourth being largely devoted to rotations in specialties and the residency placement process. The third year is the one that is brutally intense. On a outpatient service, the students spend something 10.5 hours a day, five days a week in the clinic, then go home and study for 2-3 hours more. On inpatient rotations, they go from 6:30 am to 5:00 pm in the hospital, and again go home to study for 2-3 hours. They have an exam at the end of each rotation.

There is an important institutional difference in medical education.  There's no doubt that med students get the benefit of public and private funding of the health care systems, but they are also care providers - they do things that if they didn't others would have to do.  Finding legal institutions capable of replicating this kind of intensity for law students is a problem (particularly for non-litigating lawyers), but the point is that no law student is going to approach the kind of practice readiness I perceive in a med student of equivalent tenure by doing fifteen credit hours (three hours a day of class time) a semester of even the most pragmatic skills oriented course work. Are students, faculties, administrators, law firms, courts, corporations prepared to create the kind of full-time (I mean thirteen hours a day) experience for the final two years of law school that might well leave one thinking "that's somebody who's a lawyer"? Because to do both the theory and the practice (as the med students do) would take that kind of commitment from all concerned.

(Cross-posted at Legal Profession Blog)