Monday, March 16, 2015
Deborah Jones Merritt ("DJM"), a law professor at Ohio State, has posted a new paper analyzing employment outcomes for recent law graduates. DJM does a fantastic amount of legwork, collecting the late-2014 employment status for more than 1,000 lawyers admitted to the Ohio bar around 2010. From these data, she makes some claims (of increasingly problematic nature) about the path of employment progress for those lawyers, for lawyers around the country, and for legal education.
This would be an excellent paper if DJM had been content to offer us a snapshot of employment outcomes for a cross-section of new lawyers in a second-tier legal market. It's a sobering picture, with about one-sixth of the pool in "solo practice," which she convincingly shows often equates to a struggle to make ends meet. And the methodology -- hand-gathering results for every lawyer -- is impressive.
My problem is that instead DJM wants to offer us a dynamic analysis, comparing 2014 to 2011, and arguing that the resulting differential tells us that there has been a "structural shift" in the market for lawyers. It might be that the data exist somewhere to conduct that kind of analysis, but if so they aren't in the paper. Nearly all the analysis in the paper is built on the tend line between DJM's 2014 Ohio results and national-average survey results from NALP.
Let me say that again. Almost everything DJM says is built on a mathematical comparison between two different pools whose data were constructed using different methods. I would not blame you if now stopped reading. But below the jump, some elaboration.
DJM is a careful researcher and she gives a long defense of her use of the NALP as a reasonable baseline for the 2014 Ohio pool. For example, she points out that Ohio has many features that are common to the national market (10) and (11-12) that 2010 Ohio had a distribution of jobs between firms, government, and business that was similar to the national pool. Yes, helpful, but what about the Ohio 2010 employment rates? Solo practitioners? These are the key variables for her analysis, not the firm/government split among employed lawyers. If NALP isn't really representative of 2011 Ohio employment rates, the trend line is spurious.
Even if DJM convinced us that the 2011 national averages are exactly the same as the 2011 Ohio averages, we would still have an apples-to-oranges problem, because NALP is a survey and DJM's results aren't. As DJM herself discusses (6), lawyer survey response rates are often low. Many researchers worry that surveys overstate employment rates, as they believe that unemployed attorneys are harder to find and less inclined to respond.
<Update: As Deborah explains in her comment, her 2011 NALP data in fact do reflect methods similar to those she uses to generate the 2014 information. >
This is a big problem for DJM's methodology, which is built on, in essence, taking 2011 and subtracting it from 2014. If 2011 is biased upwards, DJM's measures of improvement over the intervening 4 years will be biased downwards, presenting an unduly negative picture of employment progress.
Those are the basic problems. Now just a little deeper into the weeds on whether NALP's results are a good baseline for Ohio, and even if so whether Ohio tells us something meaningful about the rest of the country. The problem, in essence, is that there may be selection effects at work.
It's true that Ohio has some excellent lawyers and is home to the headquarters of at least one national firm. But New York, DC, San Francisco, and LA are home to dozens, if not hundreds. Lawyers who choose to take the Ohio bar may be systematically different from those who take the bar in these other places: they might intentionally be selecting a less-competitive legal environment. This might represent a view of themselves as being less able. If so, perhaps it is less surprising that their employment results are not as dynamic. Maybe not! But we'd like to be able to rule out that story before we claim that as Ohio goes, so goes the nation.
Lastly, the conclusion urges us to read the paper's results as supporting some of DJM's other policy preferences, especially her recommendation that law school be trimmed to two years. Even if I believed every other word of the paper, I wouldn't understand why that would be the implication. There aren't enough law jobs, so we should lower the barriers to entry, which most labor-supply analyses would tell you will produce more lawyers and a lower premium for the degree?
DJM should be proud of what she's accomplished, but there is a lot of work still to be done in this area before we really know what the employment prospects for lawyers will be.