Saturday, March 17, 2012

A Reply to the Empiricists at NWU Law

My blog post from last week, "Too Good for BigLaw: The Statistician Edition" has resulted in a minor kerfuffle with some of the distinguished empiricists at Northwestern Law.  See Dan Rodriguez, Law School Sorting and the Partnership Track:  Northwestern Empiricists Weigh In, Word on the Streeterville [The Blog of NWU Law Dean].  NWU Law folks were not impressed with my analysis.  Dan Rodriguez was gracious enough to send me the link at the same time his post, quoting the views of his colleagues, went live.  He has also encouraged me to reply publicly. 

I am happy to do that.  Let me start with big picture issues.  Then, for those folks with the curiousity and stamina to wade through arcane details--and experience tells me this is a small group--I will directly address, point by point, the the issues raised by Kate Litvak and Max Schanzenbach.  But at the outset, I will say that I am not conceding any ground.

Big Picture

First off, it is important to understand why this topic generated so much traffic.  See Above the Law, ABA Daily Journal, TaxProf, InstapunditAdam Smith

It all started with a provocative blog post by Vivia Chen, the columnist for The Careerist.  Vivia reviewed hiring and promotion data from the NLJ 250 Law School Hiring Survey and noted that elite law school graduates were becoming partner in very low numbers when compared to the hiring pipeline. Vivia editorialized on the numbers in a way that played into readers' fragile egos and insecurities.  Of course, that is her job, which she does very well.

In a nutshell, here is why people care -- or more precisely, get anxious -- about this topic:  it is conventional wisdom that graduation from elite law schools produces better career outcomes.  When that expectation is countered by actual marketplace data, people are surprised.  See, e.g., Bruce MacEwen,  "The Best & The Brightest" at  Adam Smith Esq. (leading blog on law firm economics).  Surprise is the first reason this issue got so much play.  Emotion is the second.

Emotion matters because very few lawyers and law professors are dispassionate on this topic.  When it comes to conventional wisdom on law school pedigree, we all have horses in the race.  Because we are human beings, we lawyers and law professors don't wait for balanced market data to develop our own entrenched worldviews.  When the conventional wisdom favors us, we go with it -- albeit we aren't really conscious this is happening. So when data upset the apple cart and potentially make us look complacent, our passions get aroused. 

The folks at Above the Law have built a entire business model around such predictable lawyer foibles.  The more chum thrown in the water, the higher the ad revenues.  It's just that simple.

Vivia's primary point, stated through metaphor, is that regional schools (such as Chicago Loyola) seem to be making partner at higher rates than the elite schools (such as Chicago).  This is a reasonable inference because the ratio of associates hired to partners promoted appears to be consistently high for elite law schools and very low for a large number of regional law schools.  This very point was made independently by Bruce MacEwen, who is a very sophisticated guy who advises law firms on strategy.

That said, there was ample opportunity for readers to draw spurious inferences from Vivia's metaphor-driven blog post.  Thus, to avoid any school-specific claims (a 1-year crossectional sample is not suitable for such a purpose), I pooled the schools by U.S. News ranking, drawing a line between elite and non-elite at the T14 mark.  Why T14?  Because these schools have played a closed loop of musical chairs for 20 years in the U.S. News rankings.  These schools would be viewed by most employers as "national" law schools.  

Here is what the data showed:

  • Pipeline in:  53.7% T14, 46.3% non-T14
  • Partners Promoted: 29.4% T14. 70.6% non-T14.

That is, well, an enormous skew. In 2011, for every 5.43 elite grads hired, a senior associate from an elite school makes partner.  For non-elite schools, that corresponding statistic is 1.95.  Vivia found these numbers surprising and somewhat counter-intuitive.  So did Bruce MacEwen, Above the Law, ABA Journal, etc.

There are ways to break down these numbers to gain additional insights, but the key point here is one of magnitude.  Elite law graduates are supposed to be smarter and more capable -- no one expects these folks to be on short side of any race, tournament or desired outcome.  The magnitude of hiring/promotion gap is the surprising fact that needs to be explained.

I had observed roughly the same skew several years ago (pooled 2007 and 2008 data) and alluded to it in this article, "Why is the Job Market Changing," Nat'l Jurist (Nov. 2010).  I also follow other relevant studies, such as The After the JD, which have noted differences between elite and non-elite graduates.   So I had a head start in thinking through possible explanations. I thus offered five theories, all of which could work in concert, to explain the large skew in the data:

  1. Selection effects
  2. Differences in first jobs
  3. Intergeneraional privilege
  4. Influence of admission criteria on the associate pipeline
  5. "A Better Plan B" for elite grads

So, to be very clear, I am not using the NLJ 250 data to support the above theories.  It is the reverse: I am offering the above theories as a likely explanation for the very large skew between elite and nonelite grads.  Framed as a open-ended research question, it might be written, "why are elite grads not becoming BigLaw partners in numbers commensurate with hiring patterns and general presumptions of their higher ability?"  That is a mystery and a puzzle.

Statistics Minutiae  [After the jump ...]

Each of my five theories could be its own independent research study.  But that's a side show.  Testing the theories will not change the skew.  Don't forget the big picture -- we have a  counterinituitive data point that needs to be explained. 

A better approach is to triangulate what we know from historical patterns, published studies, the data itself, theory and commonsense and apply Occam's Razor to find a simple set of explanations that explain all the data points.  My blog post never made any strong claims on relative importance or validity of my theories.  I was just trying to formulate a reasonably plausible set of explanations.  When something is highly contestable and important, then we can go to the trouble of formal hypothesis testing.  But we have five theories here.  It is unlikely all five are wrong.  If they were, we'd need a sixth killer theory, heretofore unknown, to explain the skew.  Again, Occam's Razor.

To my mind, this is the source of disagreement between myself, Kate and Max:  they are focused on poking holes in my theories, but I want a satisfactory explanation for this very large skew.  Max doesn't offer any explanation, and Kate's explanations quickly wilt when Occam's Razor is applied.

From this point forward, this post may be of interest to readers with a large appetite for the inner workings of inferential statistics.

Here is Kate's macro level critique:

In short, the averages he is discussing cannot tell us anything useful. The main problem is selection effects combined with the level of data aggregation: (1) no individual-level data on hired/promoted lawyers; (2) all top-250 law firms are grouped together; (3) all non-T14 law schools are grouped together.

Unfortunately, I don't know what "useful" means here.  My analysis unearths some data points that are surprising and counter-intuitive -- so says the blogosphere.  I think the averages, viewed in light of my five theories, are potentially useful if I were a law firm partner and wanted to understand unwanted attrition in my law firm-- a potential multimillion dollar problem. If I am law professor, I might also find it useful to adjust my worldview so that fits surprising and counter-intuitive data points.  After all, we are occasionally called upon to provide career advice. 

Regarding Kate's three points on data limitations, it is possible break out the data as Kate suggests.  And when broken down, it boosters rather than detracts from my initial analysis. Let me be clear, however, on the precise sample I am working with:  

  • It is called the NLJ 250 Law School Hiring Survey, but the sample is not complete.  Here is what it includes:  155 firms supplied data.  127 are AmLaw 200 (AmLaw 200 is based on revenues), and 27 firms not in the Am Law 200, but in the NLJ 250 (NLJ 250 is based on lawyer counts).
  • Of the 127 AmLaw firms in the sample (63.5% response rate), median profits per partner is $970,000 ($1.2 million average) versus $807,000 ($1.1 million average) for the 73 that did not supply data.  So the sample is not comprised of the less profitable firms; they are, if anything, more profitable. The difference in means, however, is not statistically significant.
  • The sample firms are larger: 567 median, 819 average for sample; 401 median, 472 average for the missing firms.

Based on the above, the sample appears to be a reasonable representation of the BigLaw universe.  Further, we know the directionality of the missing data, which is useful for interpreting any results. 

[One more sampling point: Kate objects to grouping all non-T14 schools together.  I disagee.  There is no point in breaking down the law schools further by U.S. News tier -- Andy Morriss and I documented that during the halycon days of the mid-2000s, outside the Top 25 or so, large law firm jobs were limited to between 0 and 15% of NLJ 250 entry level jobs.  In contrast, T14 was typically well north of 50%, even with large numbers of judicial clerkships.  There is one relevant divide here: regional versus national.  A few regional schools, such GW, Fordham, Emory, UCLA, USC and Texas, have a strong foothold for students in the top 30% or 50%.  After that, the grade cutoffs turn all other law schools to near decimal dust in terms of market pull.  I have parsed the data in previous years--these in-between schools are, well, in-between the regional and nationals in terms of outcomes.  Breaking them out won't tell us anything.]

As I noted in the original post, and pointed out by Kate, we almost certainly have massive selection effects at work here.  We know from other research that T14 grads gravitate to the most prestigious firms where the washout rates are very high.  See Zaring & Henderson.  The value of the partnership (higher profits) in combination with better outplacement prospects may make this tradeoff rational.  Sure enough, there are large selection effects: 61% of the T14 graduates in the sample joined firms in 2011 in the Top 50 as measured by Profits per Partner (PPP).  This supports the First Jobs theory. 

That said, even in the high prestige firms, non-elite grades are getting more than their pipeline's share of the partnership promotions (or, stated differently, T14s are leaving in larger proportions): 

  • Among the 25 most profitable firm in the Am Law 200, T14 accounted for 76.0% of associates hired, compared to 24.0% for non-elites  graduates.  But for partnership, the numbers fall to 60% versus 40%.
  • Same pattern if we broaden to 50 most profitable firms:  71.4% T14 hired, 28.6% non-T14;  for partners, 55.6% versus 44.4%.
  • The pattern continues for AmLaw 200 firms outside the Top 50 PPP bracket:  40.5% T14 hired, 59.5% non-elite; for partners, 24.0% T14, 76.0% non elite. 

So, using the underlying raw numbers, let's translate these figures into ratios of associates hired to partners promoted:

  • Top 25 PPP:    T14 = 10.1, non-T14 =3.9
  • Top 50 PPP:    T14 = 4.8, non-T14 = 1.9
  • Non Top 50:    T14 = 3.9, non-T14 = 1.8
  • Non-AmLaw:   T14 = 2.0, non-T14 =0.8

In every bracket in the NLJ 250/AmLaw200, the T14 get superior hiring access but much lower rates of partnership.  Again, this is Ted Seto's research in another form. 

We can get more granular.  Among the 79 firms in the 2011 sample that (a) hired T14 and non-T14 associates, and (b) promoted at least one T14 lawyer to partner, there were 51 firms (65%) where the non-14 were getting promoted at rates that exceeded their pipeline percentages. Further, compared to the remaining 28 firms, these 51 firms had higher average Revenues per Lawyer ($772,000 versus $711,000) and higher average Profits per Partner ($1.2M versus $1.0M).  Perhaps non-Top 14 partners are good for business!

Had Kate had access to these data, she may not have replied so boldly. After hiring, T14 does not fare well in any bracket.  Again, why is that?

I am not saying the above data supports my five theories; rather, I am saying some combination of my five theories likely explain a good portion of the large, counter-intuitive gaps between elite and non-elite law school graduates. Here is the story in five bullet points:

  • Selection Effects. If you are at a T14 school, BigLaw jobs are relatively plentiful. Interview well and you get the big paycheck.  In contrast, the Non-T14 crowd are comprised of a larger proportion of people who had to work very hard to get their foot in the BigLaw door.  These types of behaviors generally don't end after one gets hired. They become habits.
  • First Jobs.  The T14 are going to the most prestigious firms where the washout rates are high. We showed this above. But the T14s washout in greater proportion than their Non-T14 peers.  So, by itself, this can't explain the overall NLJ 250/AmLaw 200 gap.
  • Intergenerational Privilege.  We know from the After the JD study that elite law schools grads come from more affluence families.  See Dinovitizer & Garth, Lawyer Satisfaction in the Process of Structuring Legal Careers.  This is also the story line of Thomas Mann's Buddenbrooks novel -- affluence can produce less disciplined behavior among our progeny.
  • Ratings-Driven Admissions Criteria.  We now throw away personal statements, resumes and letters of references when making admission decisions.  20 or 30 years ago we consulted commonsense when making admissions decisions.  So the pipeline to elite law schools, and hence to the NLJ 250, has likely been affected in deleterious ways.
  • "A Better Plan B."  Elite grads have more options, presumably in gov't and business.  This is obvious.  But it is too speculative a category to explain the whole elite/non-elite gap, which is very large.

Kate's critiques in order:

  • Selection effects:  Kate says I can't rule out that that the non-T14 crowd is getting hired and promoted because they have, say, important client connections.  Why would I try to rule this out?  It is a de minimis effect at best.  It fails Occam's Razor.
  • First Jobs.  Kate says "probably true."  We can stop right there.  We are looking for the most plausible explanations.  Nonetheless, the data above are also responsive to rest of Kate's criticism--the T14/non T14 pattern holds up in every part of the BigLaw hierarchy.
  • Intergenerational Privilege.  Kate thinks I don't have the data to document T14's higher family wealth.  But The After the JD research does have these data.  Further, Garth and Dinovitzer make an empirical case that it affects satisfaction in the large firm environment.  In short, we have a literature on this topic -- I am merely pointing to it.
  • Rankings effect.  Kate thinks I lack any concrete data to make this claim.  She is right.  But we have all watched law schools make insane trade-offs in admissions to goose their rankings.  I am making the leap that this affects the labor pool in unhealthy ways.  Supply chain analytics are important in other industries.  Law firms are waking up to this.
  • "A Better Plan B."  We both agree that we need better data to make sweeping claims on this one.  We also agree that elites will have the edge.  It is just a matter of measuring the size and direction (gov't, business, public interest, academia) of the edge.

What I find puzzling about Kate's comments is that she never addresses the large gap between T14 hiring and promotions.  To my mind, that is the storyline -- the puzzle that needs to be unraveled.  If she doesn't like my theories, then she should formulate something simpler and more plausible.  But let's focus on explaining the gap.  Where is the sixth killer theory?

Max makes an entirely different point.  Max writes:

What to make of Bill’s piece?  I don’t think it tells us much. Bill is looking at the conditional probabilities. ...   If firms are willing to take fewer risks on Loyola students (e.g., we hire the number one at Loyola and that’s it), then it’s not surprising that the conditional probability is higher. In my view, this result likely suggests a bias in favor of T14 students (perhaps for rational reasons regarding search costs), and an even stronger reason to attend an elite school if one’s goal is to be a top 250 law firm partner.

Max is certainly right.  If a student's goal is to be a BigLaw partner, he/she you will have an easier time getting hired if he/she attends an elite school.  You are the same person after all, so your behavior, motivation and ability can be treated as a constant -- getting your foot in the door is key. 

But Max overlooks the perspective of a law firm, which might look at the data and say, "My god, we have been calculating these conditional probabilities all wrong.  Clients aren't paying for training any more.  We can't afford this revolving door of T14s, who don't seem as engaged as the less elite grads."  So I think the data say quite a lot.  The huge presumption favoring elite grads created the Bi-Modal distribution; this propensity was arguably the beginning of the end of the BigLaw model where firms just passed off higher salaries onto clients.  No more.


Just like Phil Corboy, who could not get a job on LaSalle Street because he was Catholic, or Sandra Day O'Connor, who could not break into the Arizona corporate bar because she was a woman, or Joe Flom who was shut out of Wall Street because he was a Jew, there is always a contemporaneous narrative that justifies the status quo.  But eventually the prejudice becomes too expensive to bear.  And we change. 

The corporate law firm sector is now competing over market share for the first time ever.  Whether elite grads are better or worse hiring bets is now becoming a matter of strategic importance.  Unlike in earlier years, firms have to think twice before paying a snobbery tax for hiring criteria disconnected from, or even negatively correlated with, performance.

People's self image are all bound up with this topic. It has very little to do with rationality, and much more to do with identity and emotion.  But the takeaway is very simple: Pedigree does not reflect a natural ordering; it is socially constructed, initially based on substance and merit, but eventually perpetuated by inertia and vested interest.  Especially during times of transition and stress, it too can crumble.  This forebodes the regeneration of substance, and that is something that all of us should welcome.

Blog posts worth reading, Data on legal education, Data on the profession, Law Firms, New and Noteworthy, Scholarship on legal education, Structural change | Permalink


So, if you were advising firms, you would tell them to draw more of their associates from the ranks of regional schools than from elite schools-- more Loyola, less Harvard.

Posted by: Anon | Mar 20, 2012 4:44:41 PM

Re advice to firm, I would not be quite that categorical.

I have worked with many law firms on statistical analysis of lawyer performance, and law school attended has never shown up as a strong success factor, but lots of others do. There are several ways to identify high quality candidates in any law school pool. It is the use of the tools that matter, not the law school attended. The issue with elite schools is that they are over-fished using very bad screening tools. So, use the tools and target schools where you can be competitive with the candidates you want.

Law firms need to "select" first and "recruit" second. Instead, the recruit using flawed heuristics. There are great candidates at every law school. But right now, it is near random how they get hired. bh.

Posted by: Bill Henderson | Mar 20, 2012 8:32:29 PM

Brilliant, Bill.

Posted by: Craig M. Boise | Mar 26, 2012 6:07:25 PM

Post a comment