The Legal Whiteboard

Jim Moliterno Answers Questions on W&L's 3L Program; Supplies Additional Data on W&L

My previous post on Washington & Lee's 3L Program stirred a lot of interest and commentary, including some disbeleiving critics.  Fortunately, Professor Jim Moliterno agreed to write a reply essay, below, that completes the cycle. [Bill Henderson]

  Jim Moliterno Replies  [This is a long reply, so a PDF version online here]

A number of comments to Bill’s January 28 post and posts regarding it on other blogs cause me to enter this conversation.

Are students really coming to W&L because of the new curriculum?  Yes, to a significant extent. How do we know?  Because the entering students say so. As do many law schools, we administer a questionnaire to our enrolling students. Among the questions asked is the obvious one: why are you here?  

In the most recent such survey the students were asked to rank the strengths of the law school. Here are the top ten, in order, according to the entering students:

1. Third Year Curriculum
2. Ranking / Prestige
3. Quality of Life
4. National Reputation
5. Job Placement
6. General Cirriculum
7. Clinical Program
8. Faculty
9. Financial Aid Award
10. Size of Lexington

The curriculum reform was first.  Financial aid awards were 9^th, just ahead of the “size of Lexington.” The data does not support the unsubstantiated claims of some bloggers that students are choosing W&L because of the generosity of financial aid awards. 

The curriculum reform has steadily moved higher on the “strength” rankings given by enrolled students since 2009. The 2011 and 2012 surveys are nearly identical, and the written comments of students about their reasons for coming to W&L (none reprinted here), are more striking than the numbers themselves.

I don’t know of any better data on this proposition but the statements of those whose reasons are under study. If that data is unsatisfying to some, then they will continue to be unsatisfied.

Are there other reasons students come to W&L? Of course. W&L has a highly productive, highly visible  faculty engaged in scholarship and projects at the highest levels. Some students undoubtedly value W&L’s faculty prowess.  W&L is highly ranked. Some students undoubtedly are affected by a top 25 ranking. It has an excellent reputation as a small, closely-knit academic community.  Some students select for the sense of community and size. No reason will ever be the only reason for prospective students to choose a law school.  Changes made by law schools will affect student choices for or against a particular law school. The W&L curriculum reform is positively affecting a significant number of students’ calculus about choosing W&L.   

And some do come because of the financial aid package they were offered.  But the financial aid reason is unlikely to explain the increase in applications since 2008. Some students, the recipients of aid, undoubtedly come in part because of the aid.  That is no different than the students who choose [insert name of any school] because of the financial aid they were awarded.   In 2012, about the same number of offers of admission were made as in previous years, but instead of the usual 130 or 135 admittees choosing to attend, more than 260 made deposits. Some were asked to defer their attendance until 2013 and once the dust settled we had a class of 187 instead of the usual 130 to 135. This same class entering in 2012 listed the curriculum reform first and financial aid ninth as strengths of the law school.

What else was happening in 2008 and 09 when the applications increased by nearly 33% per year? 

In 2009 and 10, while W&L applications were on the rise, the US News ranking fell from 25-34 (while its reputation rank among academics stayed steady).   It has now recovered to 24.  If anything, that should have led to a drop in applications during 2008-2011 rather than the sharp increases that actually occurred.

Can we exclude all other possible explanations than those previously mentioned?  Of course not.  It could be that being in a small, beautiful mountain town is all the rage among young adults and 33% more students want that now than wanted it in 2007. I know of no data to prove or disprove that proposition, so it remains one that could be true. The reality is that the students who have come in recent years rate the curriculum reform among the top reasons (often the most important reason) for their attendance at W&L. That matters.

There is empirical evidence that the W&L curriculum reform is engaging students more than in the traditional “no plan” third year curriculum. Is it perfect evidence? Of course not.  Is it definitive evidence that has no flaw? Of course not. Is anything ever supported by perfect, definite evidence that has no flaw? Not to my knowledge.  We make all of our most important decisions in life based on the best available evidence. As long as the evidence is empirically sound and statistically significant, it is worthy of respect. The evidence of W&L 3L engagement increases is sound and statistically significant and marks a path toward further research and verification.

One commenter suggested that the data is suspect because the peer schools have not been identified. Their data belongs to them, not W&L.  LSSSE does not make specific school data available to other schools.  So W&L has only a composite score for those peer schools. And it would be unseemly for W&L to reveal the specific schools.  I will not do so here. But to be sure, W&L asked LSSSE to calculate the data from a list of schools because they are the schools with whom W&L competes for students and competes in the rankings. It would not have served W&L’s research interests to learn how it compares with a list of schools that it does not compete with in the marketplace. No one at W&L has the data for any specific school. 

Nonetheless, do not be mistaken, the schools with whom W&L is compared in LSSSE data are the schools anyone would expect them to be: schools that by their geography, rank and quality compete with W&L in the relevant markets for students and placement.

One observation: in the legal profession and legal education in particular, the status quo never seems to need empirical justification.  Only change is suspect and wrong until proven definitively to be otherwise. Is there any empirical evidence that the status quo third year is the best possible third year except that it has been done that way for a long time? None that I know of. The old adage, “if it ain’t broke don’t fix it” does not apply here.  The third year of legal education is “broke”.

Amid calls for its abandonment by some, dating back at least to the early 1970s report by Paul Carrington, the third year is widely acknowledged to be of the least value among the three years. (See below on W&L’s largely unchanged  approach to years 1 and 2.) The Roman Legions (and more than a few other military powers) have found out that the mere fact that something has been successfully done before is not sufficient evidence that it will prevail in the present or future. Arguing in favor of the status quo based on no empirical evidence, . . . based only on instinct and the argument that it is the way things are currently done, is an approach doomed to failure.  Just ask Kodak. (And see my forthcoming book: “The American Legal Profession In Crisis,” Oxford, March 2013.)

How about the claim that “[W&L’s LSAT has] gone down every year since [the new curriculum was announced], while its GPA rank has, after a plunge, more or less returned to where it was.” The blogger made that claim, once again without any data, let alone empirically credible data.  Actually the W&L median LSAT was steady at 166 from 2005-2010, dropped 2 points to 164 in 2011 and stayed at 164 for 2012. It has not “gone down every year since [the new curriculum was announced in 2008].” Meanwhile, the GPA of entering classes, which was in the 3.5 and 3.4 range in 2008-2010, has gone up to the 3.6 range (3.65 and 3.62) in 2011 and 2012. The two modest changes in LSAT and GPA have essentially off-set one another in US News points. Hardly the reason for pause suggested by the blogger.

It seems that as long as someone is arguing against change, no rules apply to the arguments’ underpinnings.

Here is what the empirical evidence from the LSSSE surveys shows and what it does not show: students are more engaged in their work and their work includes more writing, more collaboration and more problem solving. Here are a few charts even more striking than those Bill used in his post. Together they say that significantly more than their peers or their predecessors at W&L, current third year students are working more, writing more, collaborating more, applying law to real world problems more, and preparing for class more often. Overall, they describe a harder-working, more engaged student body. And they are working harder at acquire the skills that matter to success as a lawyer.

Together with the entering student survey numbers, here is what the application increases at W&L show and do not show: prospective students are choosing to apply and then enroll in a law school with a third year curriculum that engages them more in the work of lawyers. The data do not show what employers think or do not think about the curriculum reform.

It is too early for employment data. One full class has graduated from the new curriculum, in May 2012, and that in a time of such incredibly reduced employment of new lawyers.  No innovation, no matter how much it might improve graduates’ abilities to perform, will change employment data until employers become convinced. That time is not yet come. It is highly unlikely that employers will break out of established patterns in times like the present when they are hiring a scant few new lawyers.

We live in a world of enormous pedigree influence. So no matter how successful our curriculum is for students, I do not expect that we will make employment gains vis a vis the top five or ten schools in the rankings. Instead, I do expect that over a five to ten year period, we will make gains vis a vis schools that are in our peer group, defined roughly as everyone between 20 and 40. Likewise, if other schools institute well-designed and substantively effective  experiential education curriculum reforms, I would not expect that they will suddenly make dramatic gains against much more highly ranked schools.  For example, if a school ranked number 150 institutes excellent reforms, I would not expect them to make gains versus schools in the top 50. But if they do the reform well AND if they make it known AND if they preserve what is already effective in the rest of their curriculum AND if they have an effective career services shop, they will make gains against the schools within their peer group.

Asking for data to show that employers have broken down the doors of a law school to hire graduates after one year is actually a bit silly. No one who seriously respects data and the market would expect such data to exist. That is a five to ten year project and is limited even then by the pedigree factor mentioned in this paragraph.

Curriculum merits. The curriculum relies on clinics, externships and practicum courses.  Readers of this blog know about the first two and less about the third. The new “practicum” courses are not skills courses.  Instead, they are courses about the lawyers’ work in various practice settings. Rather than rely exclusively on courses in trial ad or negotiation or interviewing, the W&L curricular reform relies primarily on courses like The Lawyer for Failing Businesses and Mergers and Acquisitions Practicum and Corporate Counsel and Poverty Law Litigation and The Litigation Department Lawyer. In these courses, students learn the relevant substantive law, but they learn it the way lawyers do rather than the way students do.  They learn law to solve a client’s problem. This alone is an activity that adjusts students’ mental pathways from student to lawyer.

At W&L, students are purely students in years 1 and 2. They read cases from casebooks and attend class and take final exams. None of that focus of lost. But, they spend their third year learning law as lawyers do, with a client’s service at the center rather than an exam at the center. For example, in a course called The Lawyer For Failing Businesses, the students are placed in the role of a lawyer representing a failing business.  They counsel the client about bankruptcy options. They draft the documents necessary to start a bankruptcy proceeding. They negotiate with creditors. They draft financing documents. They deal with ancillary litigation. In doing so, they learn bankruptcy law, but not for the mere transient purpose of passing a three hour exam. They learn it in the context of its immediate use for a client.

Is it quite as wide as a traditional course in Bankruptcy? No. But it does convey to students the theory of bankruptcy law and its use. In practice, lawyers do not answer clients’ questions by saying, “Yes, I learned about that on the Tuesday of the third week of my Torts course.” They use the essence of a topic to research and discover the best answer to the client’s very specific set of circumstances. 

Nothing of consequence is lost by missing a topic in a course.  Studies show that students retain about 10% of what we tell them. Coverage-need is passé.  It is what faculty members argued (sometimes disingenuously) when their Property course was being reduced from 6 to 4 credits. It is old news. No one can claim that students can be exposed to every law topic that might be beneficial to them. Students need the essence of a topic for their use in practice, not the detail, likely forgotten in any event. We unduly glorify ourselves to think that students remember everything we say in class or assign them to read. They retain the core. We hope.

Many of us have had students say, “I never learned anything about Contracts [insert whatever course you like] until I used it in my clinic [my summer job, my externship, my practice].” Of course students are wrong to say this. They acquired cognitive knowledge during their course; they realized the gain when they used the knowledge. But their comments do have meaning: to fully grasp and understand, students must not only acquire knowledge but they must also use it.

First and second year, a three year curriculum not a third year curriculum. The reformed third year curriculum follows from the first and second years. It does not stand alone and is no rejection of the good that exists in traditional legal education.

The first year has long served a valuable purpose.  In it, students’ thinking is transformed to that of a legal analyst and the skill-peak of academics is on display most prominently. The comparative advantage in teaching by academics is most pronounced in the first year. We shine and are perhaps irreplaceable here.  Students must have the critical thinking skills that we provide to them through our first year teaching techniques.  W&L instruction in the first year is largely unchanged from the past. We have added courses in international law and administrative law to the usual stable of 1L courses, and we now teach professional responsibility to second semester, first year students. (Some schools have done the same, but we are in the minority in including these three courses in the 1L year.)  But the mission of the first year is the same as it ever was. It succeeds.

The second year is also largely unchanged from the past.  Our second year students predominantly enroll in the core subjects that are not covered in the first year: evidence, corporations, basic tax, constitutional law (a first year subject at many schools), criminal procedure, trusts and estates (though to a lesser extent than in the past), etc. Many students engage in the law journal activities, the moot court competitions and the newer negotiation, mediation, client counseling and transactional skills competitions. 

I won’t repeat here everything about how the third year works, but it requires a full credit load (24 credits) of experiential education, including clinics, externships, immersion courses (litigation and transactional), practicum courses (elaborate simulations of practice settings), and a service requirement.  But within the student’s third year there is space for a traditional course in each semester if the student chooses.  So the student who lands a clerkship and has not yet taken Fed Courts can do so in the third year, for example, without being in an overload.

Ours remains a three year curriculum, with the first two years attending well to the traditional missions of the law school experience. The third year is being made more valuable; the first two are not being slighted or cast aside.

Bar exam? So far we have not seen statistically significant bar exam results.  In one year, the pass rate was up and the next year down, but neither to statistically significant levels.  We are paying attention to this possible issue and so far see no cause for concern.  We will continue to monitor. I would say that the current, traditional bar exam is itself an impediment to legal education reform.  With some states testing 28 subjects and students typically taking fewer than that number in the entire three years, room for courses that include among their teaching goals problem solving, team work, writing, business sense, etc., are a luxury that insecure students and law schools cannot afford. All schools have some students who are bar-exam-at-risk.  Some schools have a majority of such students. In general, the more insecure the students and law school, the less able they are to reform their curriculum to reflect the actual needs of students to succeed as lawyers. The bar exam has always been touted as a “gatekeeper.”  But as the subjects tested have proliferated and the practice has become more sophisticated and less reliant on rote memorization of knowledge, the gatekeeper bears less and less relationship to what is on the other side of the gate. A macramé test would also keep the gate secure, but it would say nothing about the qualities of the passing takers to excel on the other side of the gate.  The traditional bar exam becomes less and less relevant to the practice of law every year.

There is nothing anti-academic about studying the work of lawyers. To say so betrays a false elitism more likely borne of insecurity than of truth. Many legal academics could not do what lawyers do: solve real clients’ problems that involve extra-legal attributes. The work of lawyers is sophisticated. It partakes of some of the rigor of law school teaching and scholarship, but it also relies on sophisticated problem-solving  and  a multiplicity of other  talents.  Some who claim that lawyer work is mundane and uninteresting fail to understand the nature of that work in the first instance. Some who make the claim seek cover from their own lack of capacity to do such work. Describing it as uninteresting allows the speaker to hide his or her inadequacy. The study of effective lawyers is a sophisticated inquiry. The work of excellent lawyers is not mundane. And the mundane tasks undertaken by beginning lawyers in the past are becoming commoditized and outsourced.

The current system of legal education fails to account for a simple truth: the skill-set of legal academics is not a perfect overlap with that of the role to which the vast majority of our students aspire. The 19^th Century redesign of legal education was based on the premise that law school’s primary mission was not to create lawyers but rather to create law professors. (This conclusion is documented in the correspondence of the main contemporary actors involved in the reform.)  Many adjustments have been made over the subsequent century and a quarter, but the remnants of those 19^th Century decisions persist today.

Generally speaking, legal academics are excellent law analyzers and theorists. We are critical thinkers and precise analysts of law and its theoretical underpinnings. Students need this same talent and we are best at conveying it, especially in the traditional first-year courses and teaching modes. But to be successful lawyers, students need more than that foundational thinking skill.  They need to learn how to problem solve when some of the factors are not strictly law-related; they need to learn to work in teams and to manage projects; they need to acquire a measure of business sense whether they serve as business counsel or manage their own law shop; they need to learn how to manage risk and assess the risk adversity level of clients; they need to communicate the law and its constraints to non-lawyers; they need to acquire bedside manner.  In short, there is a multitude of talents and skills and attributes that students need to acquire that are not the skill-domain of academics (with many academics being an exception to this rule).

One blogger said that the 3L curriculum at W&L “focuses on practical lawyer skills.” This sort of statement sells the new curriculum far short of its reality. It actually focuses on the attributes, skills and mental habits of successful lawyers, all while providing students with substantive law and theoretical learning as well. A broad view of lawyer skills would include the mental development fostered in the first year as well.  It is time to stop pretending that legal analysis is not a practical lawyer skill.  It is—and it is both critical and fundamental—but it is not the only skill/attribute/talent that lawyers need to be successful.

Successful lawyers can and should be our partners in providing this education.  They know better than we do about many of these skills and attributes.  Indeed, some of this learning will inevitably continue to take place after law school.  But the economic realities of today’s legal market dictate that less teaching is being done after law school. Law firms teach less than they once did and more graduates are having to find their own way as solos. Demands from all quarters are that legal education provide at least a head start on the development of students in these realms. Prospective students, the practicing branch, and paying clients are all making such demands, and we ignore them at our peril. Some law schools that ignore this market demand will fail.

Legal education and the legal profession are at a crossroads. Applications are strikingly down for a reason. Schools can stand pat if they choose, and some have the market power to do so for a significant time after change would be prudent and effective. All others do so at their peril. Change is not good merely for change’s sake. But it is not prudent to stay the same when the world has changed. The practicing branch has changed; client needs and demands have changed; the society that the legal profession claims to serve has changed. Only legal education (and the organized bar) now remain stubbornly tied to anachronistic ways. The legal profession itself and legal education in particular, live as if they had eyes on the back of their head, but none on their face. Only what is past seems to be valued-- Even when what has past has no empirical basis and the conditions in which it exists have dramatically changed.

Bill Henderson based his opinion on good data.  Not perfect data but good data. Data sufficient to guide decisions in most realms of life and work.  The responses to Bill’s post to date have been based on virtualy no data, but rather on surmise and rumor and vague impressions of W&L’s. My fondest hope would be that many thoughtful, careful innovators pursue their projects and produce as much data as the legal education project allows.  This is not a one-size-fits-all enterprise. But the W&L reform is one that preserves the best of a traditional legal education while enhancing what can be improved about traditional legal education. It does not deny the value of academic work. It does not deny the value of traditional teaching methods. It adds to them third year experiences that the best data available shows are having positive effects. 

[posted by Jim Moliterno]

Washington & Lee is Biggest Legal Education Story of 2013

Here it is in a nutshell. There is empirical evidence that Washington & Lee’s experiential 3L curriculum is delivering a significantly better education to 3L students—significantly better than prior graduating classes at W&L, and significantly better than W&L’s primary competitors. Moreover, at a time when total law school applicants are on the decline, W&L’s getting more than its historical share of applicants and getting a much higher yield. When many schools are worried about revenues to survive next year and the year after, W&L is worried about creating the bandwidth needed to educate the surplus of students who enrolled in the fall of 2012, and the backlog of applicants that the school deferred to the fall of 2013.

[This is a long essay.  If you want it in PDF format, click here.]

Alas, now we know:  There is a market for high quality legal education. It consists of college graduates who don’t want to cast their lot with law schools who cannot guarantee students entree to meaningful practical training.  Some might argue that W&L is not objectively better-- that the 3L curriculum is a marketing ploy where the reality falls well short of promotional materials and that, regardless, prospective students can't judge quality. 

Well, in fact there is substantial evidence that the W&L 3L program delivers comparative value.   The evidence is based on several years' worth of data from the Law School Survey of Student Engagement (LSSSE). I received permission from Professor James Moliterno, someone who took a leadership role in building W&L’s third year program, to share some of the key results (each school controls access to its LSSSE data.) They are below.

But before getting into empirical evidence, I want to put squarely on the table the most sobering finding that likely applies to virtually all of legal education. It is this: On several key LSSSE metrics, W&L has made impressive gains vis-à-vis its own historical benchmarks and its primary rival schools. But even for this leader, there remains enormous room for improvement. More on that below.

Here is the bottom line:  Traditional legal education, when it is measured, does not fare very well. Yet, as W&L shows, substantial improvement is clearly possible. We law professors can respond to this information in one of two ways:

1. Don’t measure, as it may disconfirm our belief that we are delivering a great education.
2. Measure—even when it hurts—and improve. 

I am in the second camp. Indeed, I don’t know if improvement is possible without measurement.  Are we judging art work or the acquisition of key professional skills needed for the benefit of clients and the advancement of the public good? 

Moving the Market

I doubt I will ever forget Jim Moliterno’s September 2012 presentation at the Educating Tomorrow’s Lawyers (ETL) conference at the University of Denver. He presented a single graph (chart below) showing W&L actual applicant volumes since 2008 versus what would have happened at W&L if its applicant volume had followed national trends.

While law school applicants crested a few years ago, W&L enjoyed a large run-up in volume of applicants, presumably due to the launching of their new 3L program.  This larger applicant pool effectively served as a buffer when applicant declines began in 2011 and 2012.  Since 2008, overall law school applicants are down -19%, yet W&L is up overall +33%.   

But much more significantly, after their experiential 3L year was up and running and the overall legal job market continued to stagnate, W&L yields spiked. Ordinarily they would enroll 135 students. But for the fall of 2012, they received enrollment commitments from well over 260 students. Indeed, at the ETL conference Jim Moliterno said the school had to offer financially attractive deferments to get the class to approximately 185 incoming students -- a 50 student bulge.

When Jim Moliterno showed the above graph and explained the corresponding changes in yield, my good friend Gillian Hadfield, a skeptical, toughminded, evidence-demanding economist who teaches at USC Law, leaned over and said to me, “that is the single most important takeaway from this entire conference.” I agreed.  The market for a legal education with practical training is, apparently, much more inelastic than the market for traditional JD programs.

Yet, what is perhaps most remarkable is that a large proportion of incoming students at W&L were enrolling based on little more than faith.  Nobody knew for sure if W&L  had the ability to pull off their ambitious 3L curriculum. The program relies on a large cadre of adjunct professors, after all, and W&L is located in remote Lexington, Virginia.  Many law faculty outside of W&L, and perhaps some inside, thought (or perhaps think) that the program could not live up to the hype.  Well, as shown below, the program appears to have produced meaningful gains.

The only data-driven critique anyone can muster is that the gains remain significantly short of perfection. But that critique bites harder on the rest of us.  To use a simple metaphor, W&L is tooling around in a Model-T while the rest of us rely on horse and buggy. What ought to be plain to all of us, however, is that, just like automobile industry circa 1910, we are entering a period of staggering transformation that will last decades. And transformation will be roughly equal parts creation and destruction. See Schumpeter.

W&L Data, Internal Historical Benchmark

LSSSE is a phenomenally rich dataset – nearly 100 questions per year on a wide variety of topics related to student classroom experience, faculty interaction, type and quantity of assessments, time allocation, and perceived gains on a variety of dimensions related to personal and professional development. The survey instrument is online here.  

Aside from a host of questions related to demographics, career goals, and debt, major sections in the LSSSE include:

• Section 1, Intellectual Experience (20 questions)
• Section 2, Examinations (1 question)
• Section 3, Mental Activities (5 questions)
• Section 4, Writing (3 questions)
• Section 5, Enriching Educational Experiences (9 questions)
• Section 6, Student Satisfaction (7 questions)
• Section 7, Time Usage (11 questions)
• Section 8, Law School Environment (10 questions)
• Section 9, Quality of Relationships (3 questions)
• Section 10, Educational and Personal Growth (16 questions)

W&L deserves to be a detailed case study. But frankly, legal education can’t wait. So I will do the best I can to cover the landscape in a blog post. I hope every law faculty member who reads this post makes a strong plea to their dean to enroll in LSSSE. Why? So your school can benchmark itself against the detailed LSSSE case studies that are bound to flow out of W&L and other innovative law schools.  Though they don’t get much press, there are, in fact, other innovative law schools.

The dataset I have for W&L covers 2004, 2008, and 2012.  This is the same data that Jim Moliterno briefly shared at the ETL conference.  I have put them into bar charts so that readers can see the scores on several questions at once.  Two important interpretative notes:

1. LSSSE is especially useful when an entire class (1L, 2L, or 3L cohort) experiences a curricular change.  This happened with Indiana Law's 1L Legal Professions class.  It is also happening here, as all W&L 3L students had the benefit of the experiential 3L curriculum.  Assuming nothing else signficant has changed (a safe assumption when it comes to legal education), the classwide change enables a simple "event study" analysis.
2. W&L LSSSE scores for 2004 and 2008 are much more alike than they are different. The big differences appear between 2008 and 2012. So that is what I discuss below.

Section 1 differences are displayed below (3L students only).  Click on the chart to enlarge.

 
The big takeaway here is that W&L gained in 17 out of 20 categories.  Because Section 1 is put on a 4 point scale, just like a traditional academic grading system, we can analyze the data using something akin to a LSSSE Section GPA .  W&L's Section 1 GPA for 2008 was 2.52, which is essentially on the C+/B- cut point.  Only one factor -- communicated with faculty via email--was meaningfully above a 3.0. 

We can contrast that with a 2.85 GPA for 2012, which is in the B-/B territory.  W&L's overall average increased by .33 points, and six measure are above 3.0.  It experienced the biggest gains on the following:

• +.77, Put together ideas or concepts from different courses when completing assignments or during class discussions.
• +.75, Participated in a clinical or pro bono project as part of a course or for academic credit.
• +.53, Put together ideas or concepts from different courses when completing assignments or during class discussions.
• +.51, Worked with classmates outside of class to prepare class assignments.
• +.49, Prepared two or more drafts of a paper or assignment before turning it in.
• +.47, Discussed assignments with a faculty member.
• +.44, Used email to communicate with a faculty member (now a 3.65).
•  +.43, Talked about career plans or job search activities with a faculty member or advisor
• +.41, Worked with other students on project during class

There is still enormous room for improvement, but W&L's 3L experiential program appears to have really moved the needle on factors related to the Section 1 Intellectual Experiences factors.

 W&L fares even better on Section 3, which covers the mental activities that ostensibly comprise "thinking like a lawyer." [Click on chart to enlarge]

As shown above, W&L 3Ls drop in only one category -- rote memorization for repeating on an exam.  Surely, that pleases the W&L faculty.  These are 3Ls after all.   The overall Section 3 GPA, which excludes 3a, moves from 3.07 (B) to 3.41 (B+).  Question 3c to 3e are true higher order lawyering skills. W&L ought to wheel out these data the next time some bar association claims that legal education is not accomplishing anything.  At some places, maybe.  But good things appear to be happening at W&L.

Washington & Lee shows similar gains in the other key LSSSE sections.  If you are curious, you'll have to wait for the detailed W&L case study, which I hope will get written someday by someone at W&L.  What is no doubt of greater interest to the broader legal education community, however, is how well W&L is doing against other law schools--i.e., like us.

W&L Data, External Peer Benchmarks

 LSSSE data are the property of law school who pay for the survey.  The survey is designed to improve the education programming rather than create an industrywide ranking.  Roughly 50% of law schools participate each year.  Since its inception in 2003, 179 law schools have participate for at least one year.

Although the data are reported at the individual school-level, comparative benchmarks are a key part of the LSSSE value proposition.  Comparative benchmarks include size, public/private, the total LSSSE sample, and a peer group specified by the school.  For example, at Indiana, we might want to look at other Big 10 public law schools.  We don't get to see our rivals' scores, individually, but we can get a group average for five or more schools we select that are also participating in that specific year.

I am told that schools typically pick their peer groups based on similar rank, geography, and applicant pool, etc. I thought W&L's peer comparison would be the most relevant to show here.

Below are the 11 (out of 20) factors in LSSSE Section 1 in which W&L is higher than its peer benchmark at statistically significant levels.  Again, only 3Ls in the sample I am using here. [Click on to enlarge]

On these 11 benchmarks, W&L posts a "GPA" of 3.02 (B) versus 2.45 for the peers (C+).  Again, W&L has plenty of room to grow, but relatively speaking, it is dramatically outperforming its competition.

What about those critical Section 3 Mental Activities that comprise "thinking like a lawyer"?  Again, W&L is outdistancing the competition.  [Click to enlarge]

The Section 3 GPA, excluding 3a memorization, is 3.41 for W&L (B+) versus 3.12 for its peers (B).

Section 4 pertains to writing.  Ask any professional development coordinator in a law firm about the biggest weakenesses of incoming associates, and you'll get a near unanimous reply: "writing."  Well, the best way to become a better legal writer is to write.  How did to W&L 3Ls do on that front?  3L students at W&L write a ton. [See chart below, click on to enlarge.]
W&L 3Ls write roughly the same number of 20-page papers as those at peer schools, but in the 1-4 and 5-19 page category, W&L 3Ls surge ahead of the competition at statistically signficant levels. In the above chart, the 3.27 score for papers in the 5-19 page range corresponds to 6-7 medium length papers during the 3L year.  Peers, in contrast, are roughly at 3 medium length papers. The 3.68 score in the 1-4 page category also equals roughly 7 short papers during the 3L year; peers write roughly half that number, roughly 3-4 short numbers.

Section 7 covers time usage.  Not surprisingly, W&L 3Ls spend more time prepping for classes beyond just reading assigned text -- roughly 7 hours more per week. [See chart below, click on to enlarge.]

Imagine if you did anything seven hours more per week for two full semesters -- you'd get better at it.  Time on task really matters to professional development.

Section 9 focuses on the quality of relationships within the school.  In terms of 3L student relationships with faculty and administration, they are quite high -- indeed, higher at statistically significant levels than W&Ls peer schools. [See chart below, click on to enlarge.]

Finally, Section 10 asks a series of questions related to how well the law school experience has contributed to the student's knowledge, skill and personal development.  [See chart below, click on to enlarge.]

On 10 of 15 questions, W&L is posting higher scores than its competition -- all at statistically significant level.  But as I noted above, there remains room for improvement.  W&L Section 10 "GPA" is 2.99 (B).  Its competitor's GPA is 2.7 (B-).

Conclusion

There are three takeaways from this blog posts:

1. A sizeable number of prospective students really do care about practical skills training and are voting with their feet.  W&L has therefore become a big winner in the race for applicants. 
2. W&L's 3L experiential curriculum is substantial improvement over the curriculum W&L offered in 2004 and 2008; moreover, there is room for even more improvement.
3. There is substantial evidence that W&L, with some modest focused energy on the curriculum, is now offering a better educational experience than its peer schools -- albeit, the current grade is a "B" at best for W&L and likely lower for the rest of us.  We all, therefore, have a lot of work to do.

The example of the Washington & Lee 3L experiential year ought to be a watershed for legal education. We can no longer afford to ignore data.  Through LSSSE, high quality comparative data are cheap and comprehensive.  And that information, as we have seen, can significantly improve the value of a legal education.

[Posted by Bill Henderson]

A Blueprint for Change

Several months ago I wrote a book review of Brian Tamanaha's important book, Failing Law Schools.  In the last paragraph, I wrote:

Brian discusses the bleak employment prospects of law schools, but (through no fault of his own) understates the nature of the structural change that is occurring in the U.S. and global market for legal services.  In Part II, I will write about some logical next steps for law schools looking to get ahead of the coming tsunami.

I tried to write Part II, but a blog post just was not up to the task.  Further, I sensed that my colleagues were in no mood for half-baked solutions. There has been enormous criticism of legal education on the blogs and in the media, but very little in the way of detailed prescriptions to improve the situation.  I felt an obligation to back off on the criticism and focus on solutions.  So, in essence, Part II of my Tamanaha review became an article.

I just posted to SSRN an article entitled "A Blueprint for Change" forthcoming in the Pepperdine Law Review.  It is both a diagnosis and a proposed solution -- a solution I am actively pursuing. Here is the abstract:

This Article discusses the financial viability of law schools in the face of massive structural changes now occurring within the legal industry. It then offers a blueprint for change – a realistic way for law schools to retool themselves in an attempt to provide our students with high quality professional employment in a rapidly changing world. Because no institution can instantaneously reinvent itself, a key element of my proposal is the “12% solution.” Approximately 12% of faculty members take the lead on building a competency-based curriculum that is designed to accelerate the development of valuable skills and behaviors prized by both legal and nonlegal employers. For a variety of practical reasons, successful implementation of the blueprint requires law schools to band together in consortia. The goal of these initiatives needs to be the creation and implementation of a world-class professional education in which our graduates consistently and measurably outperform graduates from traditional J.D. programs.

I have a large backlog of shorter articles and analyses that I have not posted because I wanted my own detailed solution in the public domain.   I hope to tie all of these ideas together over the coming weeks.

Thank you, Brian Tamanaha, for writing an book that required me to think in terms of solutions.

[posted by Bill Henderson]

How to Increase Your Law School's Academic Reputation

Law schools care deeply about their academic reputation.  If this were not true, my Indiana Law mailbox would not be stuffed full with glossy brochures sharing the news of faculty publications, impressive new hires, areas of concentration, and sundry distinguished speaker series, etc. 

Because of the timing of these mailings – I got nearly 100 in Sept and October—I am guessing that the senders hoped to influence the annual U.S. News & World Report Academic Reputation survey.  Cf. Michael Sauder & Wendy Espeland, Fear of Falling: The Effects of U.S. News & World Report Rankings on U.S. Law Schools 1 (Oct 2007) (reporting "increases in marketing expenditures aimed toward raising reputation scores in the USN survey"). But does it work?  A recent study by Larry Cunningham (St. Johns Law) suggests that the effect is, at best, decimal dust.  

Glossy brochures may not reliably affect Academic Reputation, but I have uncovered four factors that are associated with statistically significant increases and decreases of USN Academic Reputation.  To illustrate, consider the scatterplot below, which plots the 1993 ordinal rank of USN Academic Reputation against the 2012 ordinal rank [click on to enlarge].

Four sets of dot (Red, Blue, Orange, and Green), each representing distinctive shared features of law schools, tend to be above or below the regression line.  These patterns suggest that changes in USN Academic Reputation over time are probably not the result of random chance. But we will get to the significance of the Red, Blue, Orange, and Green dots soon enough.  

The primary takeaway from the above scatterplot is that 2012 USN Academic Reputation is overwhelmingly a function of 1993 USN Academic Reputation.   Over 88% of the variation is explained by a school's starting point 20 years earlier.  Part of this lock-in effect may be lateral mobility.  That is, there are perks at higher ranked schools: they tend to pay more; the teaching loads are lighter; and the prestige is greater, etc. So school-level reputations rarely change, just the work addresses of the most productive scholars.  This is, perhaps, the most charitable way to explain the enormous stickiness of USN Academic Reputation.

That said, the scatterplot does not show a perfect correlation; slightly less than 12% of the variation is still in play to be explained by influences other than starting position.  A small handful of schools have made progress over these 20 years (these are the schools above the regression line), and a handful have fallen backwards (those below the line).

The Red circles, Blue rectangles, Orange diamonds, and Green circles represent four law school-level attributes.  The Reds have been big gainers in reputation, and so have the Blues. In contrast, the Oranges have all experienced big declines; and as as a group, so have the Greens.  When the attributes of the Red, Blue, Orange, and Green Schools are factored into the regression, all four are statistically signficant (Red, p =.000; Blue, p = .001; Orange, p = .012; Green, p = .000) and the explained variation increases 4% to 92.3%.  As far as linear models goes, this is quite an impressive result. 

Before you look below the fold for answers, any guesses on what is driving the Red and Blue successes and Orange and Green setbacks?

Changed the Law School Name (average gain: +39.0)

Red circles are the five law schools that, over the last 20 years, have changed university affiliations and thereby changed their names.  These include:

• Michigan State University.  In 2004, the Detroit College of Law became Michigan State University College of Law.  DCL was ranked 155 in Academic Reputation in 1993;  in 2012, MSU Law was ranked 96, reflecting a 59 point jump, which is the largest in the dataset.

• Quinnipiac Law.  In the mid-1990s, the University of Bridgeport Law School became Quinnipiac University School of Law.  This switch in university affiliations came about as the result of law faculty and students wanting to distance themselves from the financial support given to Bridgeport from the Reverend Sun Myung Moon.  Since 1993, Quinnipiac's academic reputation has climbed from 166 to 123 (+43 spots).

• Seattle University.  In 1994, the University of Puget Sound transferred the sponsorship of its law school to Seattle University, leading to the renaming and relocation of the law school.  The reincarnated law school has fared well in USN Academic Reputation, increasing from 113 to 71 (+42).

• University of New Hampshire.  In the spring of 2010, Franklin Pierce Law Center signed an affiliation agreement with the University of New Hampshire, the state's flagship public university, and in turn changed its name.  UNH Law has fared well in the USN Academic Reputation survey, climbing from 151 to 123 (+28).

• Penn State Law.  In 2000, the independent Dickinson School of Law, one of the oldest law schools in the nation, merged with Big 10 powerhouse Penn State University.  The merger has been good for USN Academic Reputation, which has increased from 107 in 1993 to 84 in 2012 (+23).

So, USN Academic Reputation is likely influenced by the halo of a stronger university brand.  But this strategy is only open to a handful of independent law schools and those affiliated with a weak, financially struggling central universities.  So it is not generalizable as a strategy for increasing Academic Reputation.  Sorry to get your hopes up.

Well, what about the Blue retangles?

Conservatives Welcome (average gain: +28.3)

This one is a little counterintuitive.  I identifed three research-oriented law schools where, compared to the rest of the legal academy, conservatives have fared well during faculty hiring: George Mason, San Diego, and Pepperdine. Why these three? (If there are other law schools that have tried to build a strong conservative faculty brand, they have escaped my attention.)

1. George Mason's Law & Economics emphasis.
2. San Diego Law is a conservative Catholic law school that hosts The Right Coast blog.
3. Pepperdine Law is a Christian-centered law school that hired Kenneth Starr to serve as dean as dean after he rapped up this tenure as Independent Counsel of the Clinton Whitewater investigation.

As show in the scatterplot above, all three law schools have fared very well in Academic Reputation:  GMU (#76 to #51, +25), San Diego (#69 to #51, +18), and Pepperdine (#107 to #65, +42). 

But wait, fellow academics vote in the USN Academic Reputation survey, and supposedly we are an overwhelmingly liberal.  So why did these three conservative school fare so well?  This could be combination of three factors:

• Discounts on productive scholars.  Because GMU and San Diego are not put off by conservative credentials, they have gotten highly productive scholars at a discount.  Among law schools on SSRN, GMU Law ranks #18 in all-time downloads and San Diego ranks #21 -- both are significantly higher than these schools' USN Academic Reputation and overall USN rank. But this does not explain Pepperdine, which ranks #117.

• USN "echo chamber" effect.  My colleague, Jeff Stake, has documented that a school's USN Academic Reputation is influenced by changes in its overall USN ranking.  So, if a school manages to increase its overall rank, USN Academic Reputation then rises.  See Stake, The Interplay between Law School Rankings, Reputations, and Resource Allocation, 81 Ind. L. J. 229 (2006).  A strong conservative brand probably helps a law school attract more than its share of highly credentialed conservative students.  Until 2001, GMU Law was perennially a T2 law school; but in 2012, it was ranked #39.  Likewise, until 2004, Pepperdine was perennially a T3/T4 (note their used to be five USN tiers); but in 2012, it was ranked #49.  In contrast, USD Law (ranked #69 in the USN Overall in 2012) has increased its Academic Reputation significantly but moved sideways in the rankings (query: did USD understand the optimal tradeoffs between LSAT and UGPA?)

• USN Voters.  The Survey voters are supposedly deans, associates deans, and newly tenured faculty.  It is at least conceivable that administrators are, as a group, less liberal than their faculty.  After all, they have to balance the law school budget each year.  Similarly, law school administrators, who are accountable to central universities, and younger faculty, who just cleared the tenure gauntlet, are probably quite in tune with law schools comprised of highly productive scholars.  And San Diego and GMU Law excel on that metric.  This might be a non-factor.  It is hard to tell.

If moving on USN Academic Reputation is really important to a faculty, the lesson here is, "make a hard, high-profile right turn, and wait a decade."  That said, there are probably not enough spoils to go around for more than a handful of conservative law schools to use this strategy.

Name changes and conversativism are the factors associated with an increases in USN Academic Reputation.  What are negative factors?

Scandals and negative NY Times coverage (average drop: -24.3)

The three orange triangles are three schools that gained unprecedented notoriety based on either a rankings scandal or extensive negative treatment in the New York Times.

• Scandals.  Illinois and Villanova both voluntarily disclosed that they submitted false admissions credentials to both the ABA and U.S. News.  And both have taken a huge hit:  within the incredibly stickly Tier 1, Illinois's Academic Reputation rank was #22 in 1993, #22 in 2011, and then #39 in 2012 (-17); similarly, Villanova's went from #69 in 1993, to #62 in 2011, to #106 in 2012 (-37).  Quite a severe pummeling by USN voters!

• New York Times coverage.  In his year long focus on law schools, David Segal of the New York Times signaled out New York Law School as a particularly egregious example of the excesses of law school.  See Segal, Law School Economics: Ka-Ching!, N.Y. Times, July 16, 2011.  If you were teaching in legal education in 2011, you read that article.  New York Law School's Academic Reputation went from #95 in 1993, to #94 in 2011, to #114 in 2012 (-19).  Ironically, New York Law School was embarking on real innovation in the years prior to the story, but negative press in the NY Times, regardless of accuracy or fairness, is a bell that can't be unrung.

The last factor is perhaps the most troubling. 

Schools in the Rust Belt (average drop: -13.0)

There are 31 schools in the so-called Rust Belt, which I define as western PA and NY, Ohio, Michigan, Indiana, Illinois and Wisconsin. These 31 law schools experienced an average USN Academic Reputation decline of 13 spots.  And note, this includes the MSU Law +59 miracle jump and several elite law schools such as Northwestern (-2), Chicago (-3), and Michigan (-3) that are in the highly sticky T14 range.  So, to get a -13 average, we need some really big negative numbers from many law schools.  

Here, I will not name names.  Instead, let me share the ten biggest drops by Rust Belt schools: -17, -20, -27, -30, -34, -34, -42, -43, -46, -49.  Eight of the ten biggest 20-year drops were Rust Belt schools (and one of the remaining two was Villanova, which earned its spot through scandal).

Why is this happening?  Well, the economic center of gravity of the US economy has been moving to the south and west for several decades now.  Although this affects the Northeast just like it does the Midwest, the Northeast has become an agglomeration of "advanced producer services", which includes bankers, consultants, accountants, and lawyers.  See Henderson & Alderson, The Changing Economic Geography of Law U.S. Law Firms (2008) (documenting a large increase in corporate law lawyers in the Northeastern Mid-Atlantic region and the relative hollowing out of corporate lawyers in the Rust Belt, with the exception of Chicago). 

The implication is that regional law schools in the Rust Belt are more likely to be serving a stagnant regional economy.  This is not particularly attractive to prospective law students.  See Henderson & Morriss, Student Quality as Measures by LSAT Scores: Migration Patterns in the U.S. News Rankings Era, 81 Ind. L. J. 163 (2006) (documenting that students will trade down in USN ranking to attend a school in large and growing corporate legal market).  So this is likely the "echo chamber" effect playing itself out in conformity with larger systemic trends affecting the legal market.  See Stake, supra.

Some might argue that the declines are the result of academic snobbery against the flyover states.  If so, this prejudice must have arisen with avengence during the last 20 years.  Or, less plausibly, some might argue that these schools have had a harder time recruiting or retaining sufficiently talented, productive faculty.  Remember, this is the same survey that boosted Detroit College of Law a record +59 jump when it made the 90-mile move to East Lansing ... which is very much in the Rust Belt.  That +59 point jump probably had a lot more to do with a Big Ten brand than the production of high quality faculty scholarship.  

After we consider starting position, the Jeff Stake "echo chamber" effect, scandals, name changes, conservative branding, and basic measure error inherent in any survey work, how much unexplained variation can we really assign to the true changes in the academic quality of law schools?  To my mind, virtually nothing.   

Below is a scatterplot that places Predicted 2012 Academic Reputation (based on starting position, name changes, conservativism, scandals, and Rust Belt status) against Actual 2012 Academic Reputation. [click on to enlarge.]  

The top three outperformers in the new model are Alabama, Georgia State and Stetson.  Was their secret sauce a better faculty, or the echo chamber aided by sunny weather, a growing southern economy, and/or cheap in-state tuition in an era of rising costs?  Regardless, congrats!

Conclusion

Here is a very big puzzle.  Law faculty are comprised of very smart people, yet we organize virtually all of our hiring, strategic plans, and marketing efforts in an effort to make gains in a reputational game that cannot be won.  Why?  That is a very big topic and, alas, the basis for a future post.

[posted by Bill Henderson]

Review of Failing Law Schools, by Brian Tamanaha (Part I)

[Update: I edited the review below to remove three paragraphs from my analysis.  It was a metaphor that was not key to my review of Brian's book yet could be fairly viewed as insulting to readers I both respect and hoped to persuade.  I am sorry about that. It was a substantial change, so I am acknowledging it here. wdh.]

Many legal academics are going to dismiss Brian Tamanaha's book, Failing Law Schools, without ever reading a page.  A larger number may simply ignore it.  That is ironic, because this is the response one would expect if Tamanaha's account of a corrupt, self-indulgence academic culture were true. 

I have lived inside this culture since I joined the academy in 2002. And I can attest that very few people inside the academy believe that we are living the high life on the backs of our students. But in the year 2012, that perception does not matter very much.  Rather, the perception that matters is the one from the outside looking in. 

Over the last eighteen months or so, The New York Times, The Wall Street Journal, The Washington Post, The Atlantic, the legal press and countless blogs (many written by unhappy students) have relentlessly hammered away at law schools.

The  lay public, including most practicing lawyers, are looking for a definitive account that can explain the legal education's maelstrom.  Tamanaha's account is a veritable Brandeis Brief on what went wrong, chocked full of facts and history and persuasive analysis. 

It begins with a deal between the ABA and AALS to join forces to persuade the state bars to restrict entry to ABA accredited law schools (the ABA's goal) and thereby to elevate the stature of the legal professoriate (the AALS's goal).  Once this deal was struck -- in the early 20th century -- pretty much every change accrued to the benefit of the law faculties: higher salaries, lower teaching loads, the advent of administrators to lighten the burden of governance, and more freedom to pursue scholarly interests.  When U.S. News & World Report ranking appear in the early 1990s, the law schools are forced to make choices.  And our collective behavior suggests that vanity and prestige are all-too-likely to trump important principles like student diversity or honesty in reporting data.

For us law professors, here is our conundrum. From the outside looking in, things look bad, even corrupt.  Yet we don't feel we have done anything wrong.  We are certain that we lack the intent to cheat or defraud.  But that, unfortunately, is error #1.  As we all know, establishing intent is always a matter of circumstantial evidence.  So let's review that evidence from the perspective of the neutral fact finder.

Life is objectively good for us:  We have high salaries, social prestige, lots of travel, job security, and near absolute freedom to organize our time outside the three to six hours a week we teach, 30 weeks a years.  Against this backdrop, there is consensus among legal employers that we are not very good at practical training including, in the eyes of many, basic legal writing.  Moreover, the overproduction of lawyers creates problems for the legal profession as a whole.  Similarly, our students  are saddled with enormous debt and nothing we are doing curricularly seems geared to solving their burgeoning unemployment  or underemployment problem.  The federal government finances this "system."  And through Income-Based Repayment programs, the U.S. taxpayers are backstopping our high costs.

Because law faculty seems to be getting the long end of the bargain here, our subjective feelings of honesty and rectitude are unlikely to be viewed by many students, practicing lawyers, or the broader public as credible.  In fact, they may be viewed as insincere or out of touch.  How did things get so badly out of kilter? 

But for Tamanaha, some pesky journalists, angry students, and the ticking time-bomb of law students debt, I am confident that we law professors could coast along on our present track for another several decades.  As an insider, I can honestly testify that we believe--sincerely beheve--that we care about our students, the quality of their education, their debt loads, and their future job prospects.  But  looking at the same set of facts, history will draw its own conclusions. And Tamanaha, akin to a lawyer building a case, offers up a very compelling narrative that the dispassionate observer is likely to find convincing. 

Other bloggers and news outlets have commented on Tamanaha's book, often drawing very different conclusions. Compare Brian Leiter's Law School Updates and Orin Kerr at Volokh Conspiracy (Tamanaha's argument has merit, particularly when he suggests that lower ranked law schools should consider changing their models), with Scott Greenfield at Simple Justice (here and here) (Tamanaha describes an insular, out-of-touch professoriate from the top down that distains the input of practicing lawyers) and the Chronicle of Higher Education (subscription req'd)  (describing Tamanaha's thesis, "Law schools are bloated with too many underworked, overpaid professors whose salaries are supported by tuition increases that are making law school a losing bet for many students").

What are the proper inferences to draw? 

In late 2011, I reviewed a copy of Tamanaha's book as part of the peer-review process for University of Chicago Press.  My primary advice to Brian, communicated directly to him as well as his editors, was "to condemn the sin, not the sinner."  Legal academics may seem culpable for privileging their interests ahead of students, I said, but these are the same folks who need to be relied upon to fix the problem. (The alternative is that nearly all of U.S. legal education will collapse under the weight of high costs and fewer entry level legal jobs; and on many days, I think the latter is just as likely as the former.)  

Frankly, I don't know if my "condemn the sin, not the sinner" recommendation was good advice.  In order to change, the legal academy may need more pressure brought to bear from outside forces. This may happen if the legal academy is painted as more selfish, insular, elitist and out of touch than we already look now.  Congress and the Department of Education hold the ultimate trump card, and Tamanaha's book provides the essential supporting evidence for radical action.  If and when this happens, law faculties will be forced to pick sides.  

History is now playing out right before our eyes.  I believe there is a good chance that Brian Tamanaha's book will be viewed--by history at least--as a great act of courage.  The implication, of course, is that the rest of us will look foolish.

Brian discusses the bleak employment prospects of law schools, but (through no fault of his own) understates the nature of the structural change that is occuring in the U.S. and global market for legal services.  In Part II, I will write about some logical next steps for law schools looking to get ahead of the coming tsunami. 

[posted by Bill Henderson]

A Reply to the Empiricists at NWU Law

My blog post from last week, "Too Good for BigLaw: The Statistician Edition" has resulted in a minor kerfuffle with some of the distinguished empiricists at Northwestern Law.  See Dan Rodriguez, Law School Sorting and the Partnership Track:  Northwestern Empiricists Weigh In, Word on the Streeterville [The Blog of NWU Law Dean].  NWU Law folks were not impressed with my analysis.  Dan Rodriguez was gracious enough to send me the link at the same time his post, quoting the views of his colleagues, went live.  He has also encouraged me to reply publicly. 

I am happy to do that.  Let me start with big picture issues.  Then, for those folks with the curiousity and stamina to wade through arcane details--and experience tells me this is a small group--I will directly address, point by point, the the issues raised by Kate Litvak and Max Schanzenbach.  But at the outset, I will say that I am not conceding any ground.

Big Picture

First off, it is important to understand why this topic generated so much traffic.  See Above the Law, ABA Daily Journal, TaxProf, Instapundit,  Adam Smith. 

It all started with a provocative blog post by Vivia Chen, the columnist for The Careerist.  Vivia reviewed hiring and promotion data from the NLJ 250 Law School Hiring Survey and noted that elite law school graduates were becoming partner in very low numbers when compared to the hiring pipeline. Vivia editorialized on the numbers in a way that played into readers' fragile egos and insecurities.  Of course, that is her job, which she does very well.

In a nutshell, here is why people care -- or more precisely, get anxious -- about this topic:  it is conventional wisdom that graduation from elite law schools produces better career outcomes.  When that expectation is countered by actual marketplace data, people are surprised.  See, e.g., Bruce MacEwen,  "The Best & The Brightest" at  Adam Smith Esq. (leading blog on law firm economics).  Surprise is the first reason this issue got so much play.  Emotion is the second.

Emotion matters because very few lawyers and law professors are dispassionate on this topic.  When it comes to conventional wisdom on law school pedigree, we all have horses in the race.  Because we are human beings, we lawyers and law professors don't wait for balanced market data to develop our own entrenched worldviews.  When the conventional wisdom favors us, we go with it -- albeit we aren't really conscious this is happening. So when data upset the apple cart and potentially make us look complacent, our passions get aroused. 

The folks at Above the Law have built a entire business model around such predictable lawyer foibles.  The more chum thrown in the water, the higher the ad revenues.  It's just that simple.

Vivia's primary point, stated through metaphor, is that regional schools (such as Chicago Loyola) seem to be making partner at higher rates than the elite schools (such as Chicago).  This is a reasonable inference because the ratio of associates hired to partners promoted appears to be consistently high for elite law schools and very low for a large number of regional law schools.  This very point was made independently by Bruce MacEwen, who is a very sophisticated guy who advises law firms on strategy.

That said, there was ample opportunity for readers to draw spurious inferences from Vivia's metaphor-driven blog post.  Thus, to avoid any school-specific claims (a 1-year crossectional sample is not suitable for such a purpose), I pooled the schools by U.S. News ranking, drawing a line between elite and non-elite at the T14 mark.  Why T14?  Because these schools have played a closed loop of musical chairs for 20 years in the U.S. News rankings.  These schools would be viewed by most employers as "national" law schools.  

Here is what the data showed:

• Pipeline in:  53.7% T14, 46.3% non-T14
• Partners Promoted: 29.4% T14. 70.6% non-T14.

That is, well, an enormous skew. In 2011, for every 5.43 elite grads hired, a senior associate from an elite school makes partner.  For non-elite schools, that corresponding statistic is 1.95.  Vivia found these numbers surprising and somewhat counter-intuitive.  So did Bruce MacEwen, Above the Law, ABA Journal, etc.

There are ways to break down these numbers to gain additional insights, but the key point here is one of magnitude.  Elite law graduates are supposed to be smarter and more capable -- no one expects these folks to be on short side of any race, tournament or desired outcome.  The magnitude of hiring/promotion gap is the surprising fact that needs to be explained.

I had observed roughly the same skew several years ago (pooled 2007 and 2008 data) and alluded to it in this article, "Why is the Job Market Changing," Nat'l Jurist (Nov. 2010).  I also follow other relevant studies, such as The After the JD, which have noted differences between elite and non-elite graduates.   So I had a head start in thinking through possible explanations. I thus offered five theories, all of which could work in concert, to explain the large skew in the data:

1. Selection effects
2. Differences in first jobs
3. Intergeneraional privilege
4. Influence of admission criteria on the associate pipeline
5. "A Better Plan B" for elite grads

So, to be very clear, I am not using the NLJ 250 data to support the above theories.  It is the reverse: I am offering the above theories as a likely explanation for the very large skew between elite and nonelite grads.  Framed as a open-ended research question, it might be written, "why are elite grads not becoming BigLaw partners in numbers commensurate with hiring patterns and general presumptions of their higher ability?"  That is a mystery and a puzzle.

Statistics Minutiae  [After the jump ...]

Each of my five theories could be its own independent research study.  But that's a side show.  Testing the theories will not change the skew.  Don't forget the big picture -- we have a  counterinituitive data point that needs to be explained. 

A better approach is to triangulate what we know from historical patterns, published studies, the data itself, theory and commonsense and apply Occam's Razor to find a simple set of explanations that explain all the data points.  My blog post never made any strong claims on relative importance or validity of my theories.  I was just trying to formulate a reasonably plausible set of explanations.  When something is highly contestable and important, then we can go to the trouble of formal hypothesis testing.  But we have five theories here.  It is unlikely all five are wrong.  If they were, we'd need a sixth killer theory, heretofore unknown, to explain the skew.  Again, Occam's Razor.

To my mind, this is the source of disagreement between myself, Kate and Max:  they are focused on poking holes in my theories, but I want a satisfactory explanation for this very large skew.  Max doesn't offer any explanation, and Kate's explanations quickly wilt when Occam's Razor is applied.

From this point forward, this post may be of interest to readers with a large appetite for the inner workings of inferential statistics.

Here is Kate's macro level critique:

In short, the averages he is discussing cannot tell us anything useful. The main problem is selection effects combined with the level of data aggregation: (1) no individual-level data on hired/promoted lawyers; (2) all top-250 law firms are grouped together; (3) all non-T14 law schools are grouped together.

Unfortunately, I don't know what "useful" means here.  My analysis unearths some data points that are surprising and counter-intuitive -- so says the blogosphere.  I think the averages, viewed in light of my five theories, are potentially useful if I were a law firm partner and wanted to understand unwanted attrition in my law firm-- a potential multimillion dollar problem. If I am law professor, I might also find it useful to adjust my worldview so that fits surprising and counter-intuitive data points.  After all, we are occasionally called upon to provide career advice. 

Regarding Kate's three points on data limitations, it is possible break out the data as Kate suggests.  And when broken down, it boosters rather than detracts from my initial analysis. Let me be clear, however, on the precise sample I am working with:  

• It is called the NLJ 250 Law School Hiring Survey, but the sample is not complete.  Here is what it includes:  155 firms supplied data.  127 are AmLaw 200 (AmLaw 200 is based on revenues), and 27 firms not in the Am Law 200, but in the NLJ 250 (NLJ 250 is based on lawyer counts).
• Of the 127 AmLaw firms in the sample (63.5% response rate), median profits per partner is $970,000 ($1.2 million average) versus $807,000 ($1.1 million average) for the 73 that did not supply data.  So the sample is not comprised of the less profitable firms; they are, if anything, more profitable. The difference in means, however, is not statistically significant.
• The sample firms are larger: 567 median, 819 average for sample; 401 median, 472 average for the missing firms.

Based on the above, the sample appears to be a reasonable representation of the BigLaw universe.  Further, we know the directionality of the missing data, which is useful for interpreting any results. 

[One more sampling point: Kate objects to grouping all non-T14 schools together.  I disagee.  There is no point in breaking down the law schools further by U.S. News tier -- Andy Morriss and I documented that during the halycon days of the mid-2000s, outside the Top 25 or so, large law firm jobs were limited to between 0 and 15% of NLJ 250 entry level jobs.  In contrast, T14 was typically well north of 50%, even with large numbers of judicial clerkships.  There is one relevant divide here: regional versus national.  A few regional schools, such GW, Fordham, Emory, UCLA, USC and Texas, have a strong foothold for students in the top 30% or 50%.  After that, the grade cutoffs turn all other law schools to near decimal dust in terms of market pull.  I have parsed the data in previous years--these in-between schools are, well, in-between the regional and nationals in terms of outcomes.  Breaking them out won't tell us anything.]

As I noted in the original post, and pointed out by Kate, we almost certainly have massive selection effects at work here.  We know from other research that T14 grads gravitate to the most prestigious firms where the washout rates are very high.  See Zaring & Henderson.  The value of the partnership (higher profits) in combination with better outplacement prospects may make this tradeoff rational.  Sure enough, there are large selection effects: 61% of the T14 graduates in the sample joined firms in 2011 in the Top 50 as measured by Profits per Partner (PPP).  This supports the First Jobs theory. 

That said, even in the high prestige firms, non-elite grades are getting more than their pipeline's share of the partnership promotions (or, stated differently, T14s are leaving in larger proportions): 

• Among the 25 most profitable firm in the Am Law 200, T14 accounted for 76.0% of associates hired, compared to 24.0% for non-elites  graduates.  But for partnership, the numbers fall to 60% versus 40%.
• Same pattern if we broaden to 50 most profitable firms:  71.4% T14 hired, 28.6% non-T14;  for partners, 55.6% versus 44.4%.
• The pattern continues for AmLaw 200 firms outside the Top 50 PPP bracket:  40.5% T14 hired, 59.5% non-elite; for partners, 24.0% T14, 76.0% non elite. 

So, using the underlying raw numbers, let's translate these figures into ratios of associates hired to partners promoted:

• Top 25 PPP:    T14 = 10.1, non-T14 =3.9
• Top 50 PPP:    T14 = 4.8, non-T14 = 1.9
• Non Top 50:    T14 = 3.9, non-T14 = 1.8
• Non-AmLaw:   T14 = 2.0, non-T14 =0.8

In every bracket in the NLJ 250/AmLaw200, the T14 get superior hiring access but much lower rates of partnership.  Again, this is Ted Seto's research in another form. 

We can get more granular.  Among the 79 firms in the 2011 sample that (a) hired T14 and non-T14 associates, and (b) promoted at least one T14 lawyer to partner, there were 51 firms (65%) where the non-14 were getting promoted at rates that exceeded their pipeline percentages. Further, compared to the remaining 28 firms, these 51 firms had higher average Revenues per Lawyer ($772,000 versus $711,000) and higher average Profits per Partner ($1.2M versus $1.0M).  Perhaps non-Top 14 partners are good for business!

Had Kate had access to these data, she may not have replied so boldly. After hiring, T14 does not fare well in any bracket.  Again, why is that?

I am not saying the above data supports my five theories; rather, I am saying some combination of my five theories likely explain a good portion of the large, counter-intuitive gaps between elite and non-elite law school graduates. Here is the story in five bullet points:

• Selection Effects. If you are at a T14 school, BigLaw jobs are relatively plentiful. Interview well and you get the big paycheck.  In contrast, the Non-T14 crowd are comprised of a larger proportion of people who had to work very hard to get their foot in the BigLaw door.  These types of behaviors generally don't end after one gets hired. They become habits.
• First Jobs.  The T14 are going to the most prestigious firms where the washout rates are high. We showed this above. But the T14s washout in greater proportion than their Non-T14 peers.  So, by itself, this can't explain the overall NLJ 250/AmLaw 200 gap.
• Intergenerational Privilege.  We know from the After the JD study that elite law schools grads come from more affluence families.  See Dinovitizer & Garth, Lawyer Satisfaction in the Process of Structuring Legal Careers.  This is also the story line of Thomas Mann's Buddenbrooks novel -- affluence can produce less disciplined behavior among our progeny.
• Ratings-Driven Admissions Criteria.  We now throw away personal statements, resumes and letters of references when making admission decisions.  20 or 30 years ago we consulted commonsense when making admissions decisions.  So the pipeline to elite law schools, and hence to the NLJ 250, has likely been affected in deleterious ways.
• "A Better Plan B."  Elite grads have more options, presumably in gov't and business.  This is obvious.  But it is too speculative a category to explain the whole elite/non-elite gap, which is very large.

Kate's critiques in order:

• Selection effects:  Kate says I can't rule out that that the non-T14 crowd is getting hired and promoted because they have, say, important client connections.  Why would I try to rule this out?  It is a de minimis effect at best.  It fails Occam's Razor.
• First Jobs.  Kate says "probably true."  We can stop right there.  We are looking for the most plausible explanations.  Nonetheless, the data above are also responsive to rest of Kate's criticism--the T14/non T14 pattern holds up in every part of the BigLaw hierarchy.
• Intergenerational Privilege.  Kate thinks I don't have the data to document T14's higher family wealth.  But The After the JD research does have these data.  Further, Garth and Dinovitzer make an empirical case that it affects satisfaction in the large firm environment.  In short, we have a literature on this topic -- I am merely pointing to it.
• Rankings effect.  Kate thinks I lack any concrete data to make this claim.  She is right.  But we have all watched law schools make insane trade-offs in admissions to goose their rankings.  I am making the leap that this affects the labor pool in unhealthy ways.  Supply chain analytics are important in other industries.  Law firms are waking up to this.
• "A Better Plan B."  We both agree that we need better data to make sweeping claims on this one.  We also agree that elites will have the edge.  It is just a matter of measuring the size and direction (gov't, business, public interest, academia) of the edge.

What I find puzzling about Kate's comments is that she never addresses the large gap between T14 hiring and promotions.  To my mind, that is the storyline -- the puzzle that needs to be unraveled.  If she doesn't like my theories, then she should formulate something simpler and more plausible.  But let's focus on explaining the gap.  Where is the sixth killer theory?

Max makes an entirely different point.  Max writes:

What to make of Bill’s piece?  I don’t think it tells us much. Bill is looking at the conditional probabilities. ...   If firms are willing to take fewer risks on Loyola students (e.g., we hire the number one at Loyola and that’s it), then it’s not surprising that the conditional probability is higher. In my view, this result likely suggests a bias in favor of T14 students (perhaps for rational reasons regarding search costs), and an even stronger reason to attend an elite school if one’s goal is to be a top 250 law firm partner.

Max is certainly right.  If a student's goal is to be a BigLaw partner, he/she you will have an easier time getting hired if he/she attends an elite school.  You are the same person after all, so your behavior, motivation and ability can be treated as a constant -- getting your foot in the door is key. 

But Max overlooks the perspective of a law firm, which might look at the data and say, "My god, we have been calculating these conditional probabilities all wrong.  Clients aren't paying for training any more.  We can't afford this revolving door of T14s, who don't seem as engaged as the less elite grads."  So I think the data say quite a lot.  The huge presumption favoring elite grads created the Bi-Modal distribution; this propensity was arguably the beginning of the end of the BigLaw model where firms just passed off higher salaries onto clients.  No more.

Conclusion

Just like Phil Corboy, who could not get a job on LaSalle Street because he was Catholic, or Sandra Day O'Connor, who could not break into the Arizona corporate bar because she was a woman, or Joe Flom who was shut out of Wall Street because he was a Jew, there is always a contemporaneous narrative that justifies the status quo.  But eventually the prejudice becomes too expensive to bear.  And we change. 

The corporate law firm sector is now competing over market share for the first time ever.  Whether elite grads are better or worse hiring bets is now becoming a matter of strategic importance.  Unlike in earlier years, firms have to think twice before paying a snobbery tax for hiring criteria disconnected from, or even negatively correlated with, performance.

People's self image are all bound up with this topic. It has very little to do with rationality, and much more to do with identity and emotion.  But the takeaway is very simple: Pedigree does not reflect a natural ordering; it is socially constructed, initially based on substance and merit, but eventually perpetuated by inertia and vested interest.  Especially during times of transition and stress, it too can crumble.  This forebodes the regeneration of substance, and that is something that all of us should welcome.

"The Coming Crash in Legal Education: How We Got Here, and Where We Go Now"

A new paper by Prof. Richard Bourne of the University of Baltimore. Bourne has been teaching for over 30 years, following five years in practice and time as a teaching fellow at Harvard. I find the reflections of long time participants in legal education to be useful inputs in evaluating how things have changed. This is an interesting paper with much to offer.

The abstract:

This paper will first track the ways in which the legal services market has grown and changed over the past forty years. It will then track the major changes that have attended legal education during the same period and the increasing dependence of the legal education industry on student debt. The paper will then explore why, at long last, the boom-times may have run their course and why, at some point, painful changes will likely occur. Though they cannot be described in detail, the author will attempt to outline the likely nature of the changes that will occur. Finally, the paper will briefly explore how the predicted reckoning may yet lead to an improvement in the marketing of legal services and an enhanced role for law schools in preparing new attorneys for the new bar they will be joining.

There's quite a bit of provocative stuff in here (on the chopping block are clinics, faculty scholarship, "law and.." courses, merit scholarships, and light course loads!) but also the traditional laments about U.S. News: 

If law schools could somehow eliminate or seriously weaken the impact of U.S. News rankings they could begin to cut back in a big way on many of the marketing costs that currently burden legal education. 

It isn't a "ruinous U.S. News sweepstakes" that drives the cost structure of legal education. Brochures and other marketing measures are a tiny fraction of even a single entry level professor's salary. The law school cost structure is largely salaries. I do agree with Bourne (especially since he cites Bill's and my work on this point) that competition for rankings has shifted financial aid to merit aid and away from need based aid, with deleterious consequences for the profession, legal education, and general social mobility.

But, as Bill and I have written elsewhere, if law schools released more useful data for students to use and facilitated such comparisons, U.S. News would be less important. It is important precisely because law schools don't make it easy to compare across schools while applicants making massive investments in education desperately want to compare their options. Fill that need with something better than the current rankings and the U.S. News problem will solve itself.

Bourne notes in his conclusion,"The time has come to stop pursuing the ephemeral benefits of prestige, simply for the sake of prestige, and to deliver more in the way of value."  That's much more important than fussing about U.S. News. But how to do that? A footnote at the end of Bourne's paper notes Alfred Z. Reed's thoughtful and provocative 1921 report, Training for the Public Profession of the Law and its argument for a wide range of legal training, producing different kinds of lawyers licensed to do different things. As Bourne notes Reed's report "was rejected as heretical by the organized bar." Maybe that's where to start the discussion.

[Posted by Andy Morriss]

Legal Education's Ninety-Five Theses

Brent E. Newton, an adjunct professor at Georgetown University Law Center, has posted a legal education reform piece on SSRN, entitled The Ninety-Five Theses: Systemic Reforms in the American Legal Education and Licensure   [Hat-tip TaxProf].  Judging by his title, Newton is hoping to spur a Reformation of legal education, akin to what Martin Luther did for Christianity in the 16th century.  If that is his agenda, I will not stand in his way.  

According to his GULC web bio, Newton is Deputy Staff Director of the U.S. Sentencing Commission; prior to that, he had a distinguished career as a public defender.  Newton is not the only adjunct-practitioner who has forcefully challenged U.S. legal education.  In 2008, Jason Dolin (solo practitioner, adjunct at Capital), published Opportunity Lost: How Law School Disappoints Law Students, the Public, and the Legal Profession.   In 2010, Steve Bennett (partner at Jones Day, adjunct at Fordham) published a law review article entitled, When Will Law Schools Change?

Law professors rarely engage with these critiques; to acknowledge these critiques, some might argue, is to give them oxygen and legitimacy.  I think this approach is a huge mistake.  Any enterprise interested in long-term success cares about the perceptions held by its stakeholders -- and adjuncts are definitely in that group.  In times of crisis, we need friends, not enemies.  Further, Newton, Dolin and Bennett are serious people and very capable lawyers.  If you leaf through these articles, you'll see that they read like Brandeis Briefs against the legal education establishment.  The authors present thoughtful, fact-based, and (albeit occasionally) trenchant arguments on why we, speaking as a legal education insider, should change. 

Simple question:  Can any of us identify a single historical example in which the establishment reformed itself because a critic effectively marshaled facts and logic to reveal the errors of its ways?  Institutional change doesn't happen that way -- facts and logic are no match for a few thousand egos and pious rationalizations for why others should change, but not me.  

The common storyline for institutional change is failure, with the rise of other institutions that better address the social, political and economic needs of stakeholders and broader society.  A less common narrative is institutional adaption, thanks in part to (1) the self-interest and survival instincts, and (2) the serendipity of timely, brilliant leadership.  (Does the legal academy have a few hundred great leaders?)

That said, Newton, Dolin and Bennett may be on the right side of history.  Because of the overproduction of law school graduates and their high levels of debt, we are now at a point when survival for a large proportion of law schools can no longer be taken for granted.  "What cannot go on forever, won't."  Herbert Stein, economist.  

Prediction:  In the next few years, some law schools will change and thrive.  Others won't and they will fail.  There will be nasty recriminations and gnashing of teeth.  A few at the very top will throw dice and decide not to change.  They will survive, but the innovations taking root in the rest of the law school hierarchy will make them look like anachronisms.  It will be a slow decay.  In the meantime, some aspects of the Post-Langdellian paradigm will look a lot like the suggestions made by Newton, Dolin and Bennett.  In twenty years, maybe sooner, the revolution will be over. Finally, Newton et al. will get a must deserved footnote.

[Posted by Bill Henderson]