Monday, March 16, 2015

Do We Know What Happened to the JD Class of 2010?


Deborah Jones Merritt ("DJM"), a law professor at Ohio State, has posted a new paper analyzing employment outcomes for recent law graduates.  DJM does a fantastic amount of legwork, collecting the late-2014 employment status for more than 1,000 lawyers admitted to the Ohio bar around 2010.  From these data, she makes some claims (of increasingly problematic nature) about the path of employment progress for those lawyers, for lawyers around the country, and for legal education.  

This would be an excellent paper if DJM had been content to offer us a snapshot of employment outcomes for a cross-section of new lawyers in a second-tier legal market.  It's a sobering picture, with about one-sixth of the pool in "solo practice," which she convincingly shows often equates to a struggle to make ends meet.   And the methodology -- hand-gathering results for every lawyer -- is impressive.

My problem is that instead DJM wants to offer us a dynamic analysis, comparing 2014 to 2011, and arguing that the resulting differential tells us that there has been a "structural shift" in the market for lawyers.  It might be that the data exist somewhere to conduct that kind of analysis, but if so they aren't in the paper.  Nearly all the analysis in the paper is built on the tend line between DJM's 2014 Ohio results and national-average survey results from NALP.  

Let me say that again.  Almost everything DJM says is built on a mathematical comparison between two different pools whose data were constructed using different methods.  I would not blame you if now stopped reading.  But below the jump, some elaboration.

DJM is a careful researcher and she gives a long defense of her use of the NALP as a reasonable baseline for the 2014 Ohio pool.  For example, she points out that Ohio has many features that are common to the national market (10) and (11-12) that 2010 Ohio had a distribution of jobs between firms, government, and business  that was similar to the national pool. Yes, helpful, but what about the Ohio 2010 employment rates?  Solo practitioners?  These are the key variables for her analysis, not the firm/government split among employed lawyers.  If NALP isn't really representative of 2011 Ohio employment rates, the trend line is spurious.  

Even if DJM convinced us that the 2011 national averages are exactly the same as the 2011 Ohio averages, we would still have an apples-to-oranges problem, because NALP is a survey and DJM's results aren't.  As DJM herself discusses (6), lawyer survey response rates are often low.  Many researchers worry that surveys overstate employment rates, as they believe that unemployed attorneys are harder to find and less inclined to respond.  

<Update: As Deborah explains in her comment, her 2011 NALP data in fact do reflect methods similar to those she uses to generate the 2014 information.  >

This is a big problem for DJM's methodology, which is built on, in essence, taking 2011 and subtracting it from 2014.  If 2011 is biased upwards, DJM's measures of improvement over the intervening 4 years will be biased downwards, presenting an unduly negative picture of employment progress.   

Those are the basic problems.  Now just a little deeper into the weeds on whether NALP's results are a good baseline for Ohio, and even if so whether Ohio tells us something meaningful about the rest of the country.  The problem, in essence, is that there may be selection effects at work.  

It's true that Ohio has some excellent lawyers and is home to the headquarters of at least one national firm.  But New York, DC, San Francisco, and LA are home to dozens, if not hundreds.  Lawyers who choose to take the Ohio bar may be systematically different from those who take the bar in these other places: they might intentionally be selecting a less-competitive legal environment.  This might represent a view of themselves as being less able.  If so, perhaps it is less surprising that their employment results are not as dynamic.  Maybe not!  But we'd like to be able to rule out that story before we claim that as Ohio goes, so goes the nation.  

Lastly, the conclusion urges us to read the paper's results as supporting some of DJM's other policy preferences, especially her recommendation that law school be trimmed to two years.  Even if I believed every other word of the paper, I wouldn't understand why that would be the implication. There aren't enough law jobs, so we should lower the barriers to entry, which most labor-supply analyses would tell you will produce more lawyers and a lower premium for the degree?  

DJM should be proud of what she's accomplished, but there is a lot of work still to be done in this area before we really know what the employment prospects for lawyers will be.

| Permalink


Thanks very much for the thoughtful comments. I agree that the comparison data are less than ideal, but I think it’s important for us to work with what we have while we continue research in this under-examined field. Note that I don’t conduct any statistical tests when comparing different data sets, precisely because of the “apples to oranges” question. I make those comparisons descriptively, asking what do we know so far about the progress of the Class of 2010 from February 2011 through December 2014, and what do we know about this class’s progress compared to the only other class (2000) that has been studied extensively.
The two comparisons actually have different limits. The NALP data from Feb 2011 are not a survey as social scientists use that term. Schools begin by surveying their recent graduates, but that input is just the starting point. Schools follow up by searching for grad employment in the same way I did, as well as by sending repeated emails to the nonresponsive students and beseeching faculty for any leads they have. In the end, NALP received employment data for 93.0% of the Class of 2010 nine months after graduation. Most of the “missing” are, as in my population, likely unemployed—and observers treat them that way.
So the NALP method is actually quite similar to mine. The comparison issue here is national versus Ohio. I lay that out as clearly as I can in the article. I don’t think Ohio represents New York or North Dakota but, then again, the national averages probably don’t represent those states either. My educated-by-looking-at-previous-research guess is that Ohio is a pretty good representative of the national average. But markets in other states may not follow the national average. Someone who wants to know about the market in New York (or North Dakota) probably has to do a state-specific study. Neither my study nor the NALP national figures will be as well fitted to those states as the researcher might want.
More in a minute about the other comparisons—not sure what the limit on comment length is here.

Posted by: Deborah Jones Merritt | Mar 17, 2015 9:37:13 AM

The second set of comparisons, I think, is more important than the first. These comparisons are between the Class of 2000 in 2003 (as tracked by After the JD) and the Class of 2010 in 2014 (as studied by me). Here we have both problems that you identified: The AJD sample is national, while my population is limited to one state. In addition, AJD relied upon surveys while I used my internet-public-source method. You could say that this is comparing whole apples to orange slices.
It is, so once again I don’t use statistical tests for these comparisons. The more descriptive comparisons are still useful, I think, for these reasons. First, we all know something about apples, oranges, and the fruit market generally. When thinking about these figures, we can weigh the context and make some reasoned judgments about whether the differences are large enough that they tell us something about the fruit market.
Second, AJD was a very extensive (and, from all appearances, expensive) study. It seems like it should be useful at least as a reference point. And, if not, that’s important to know too. Some reasonable conclusions from my comparison might be (a) hmm, we don’t really know if Ohio represents other states or national averages, and (b) the AJD study may have been overly optimistic (through response bias) on how good job outcomes were for the Class of 2000.
Finally, the 2003 results for the Class of 2000 seem like a good comparison in other respects. Comparing the Class of 2010 to the Class of 2005 might overstate differences because 2005 graduated into the start of the boom. Three or four years into their careers, they probably were riding high. The Class of 2000 graduated at a good time, but then faced a recession. That was a mild one as recessions go, but the jobless recovery was significant. We can’t simulate the Great Recession in earlier periods, but the comparison at least relies upon a recent time with some economic shadows.
Bottom line for me is that the comparison in law firm employment (62.1% for the Class of 2000 three years after graduation, 40.5% for the lawyers in my population) seems too stark to stem solely from different populations or different methods—particularly because other data show a more modest decline in law firm employment over time. But this is definitely an area in which we need much, much more research. Your thoughtful comments will contribute to that, so thanks again.

Posted by: Deborah Jones Merritt | Mar 18, 2015 10:20:02 AM

Post a comment