Tuesday, February 4, 2014

If We Make Legal Education More Experiential, Would it Really Matter?

I think the answer is yes.  But, unfortunately, in virtually all of the debate surrounding legal education, there is a tremendous lack of clarity and precision about how we assess improvements in quality.  And equally relevant, if a gain is real, was it worth the cost?

The purpose of this essay is to chip away at this serious conceptual gap.  Until this gap is filled, experiential education will fall significantly short of its potential. 

Is Experiential Legal Education Better?  And if so, at What Cost?

Many legal educators believe that if we had more clinics, externships, and skills courses in law school, legal education would be better.  Why?  Because this more diversified curriculum would become more "experiential."  

Inside the legal education echo chamber, we often accept this claim as self-evident. The logic runs something like this.  A competent lawyer needs domain knowledge + practical skills + a fiduciary disposition (i.e., the lawyer’s needs are subservient to the needs of clients and the rule of law).  Since practical skills—and some would argue, a fiduciary disposition—cannot be effectively acquired through traditional Socratic or lecture teaching methods, the ostensible logic is that schools become better by embracing the "learning-by-doing" experiential approach.

That may be true.  I would bet on it. But the per-unit cost of legal education is also probably going up as well.  So, have we really created a viable and sustainable long-term improvement to legal education?  

In my mind, the questions we should be asking instead are the following:  (1) Among experiential teaching methods, which ones are the most effective at accelerating professional development?  And (2) among these options, how much does each cost to operate?  Quality and cost must be assessed simultaneously.  After they are evaluated, then we will be able to make choices and tradeoffs. 

Let's start with quality, which I define as moving lawyers toward their peak effectiveness potential as rapidly and cost-effectively as possible. This is an education design problem, as we are trying to find the right combination of education (building domain knowledge) and experience (acquiring and honing skills through practice).  There is also likely to be an optimal way to sequence the various educational and experiential steps. 

Creating Compelling Evidence of Educational Quality

We legal educators have many ideas on how to improve educational quality, but we make no real progress if employers and students remain unconvinced.  Can it be shown that because of a specific type of experiential curriculum at School X, its graduates are, during the first few years of practice, more capable lawyers than graduates of School Y?  

[Side bar:  If you are skeptical of this market test, it is worth noting that it was the preferences of law firm employers who gave rise to the existing national law school hierarchy.  It happened about 100 years ago when a handful of law schools adopted the case method, required undergraduate education as a prerequisite to admission, and hired scholars as teachers.  As a general matter, this was a far better education than a practitioner reading lecture notes at the local YMCA.  See William Henderson, "Successful Lawyer Skills and Behaviors," in Essential Qualities of the Professional Lawyer ch 5 (P. Haskins ed., 2013).]

If a law school can produce, on balance, a better caliber of graduates than its competitors, then we are getting somewhere.  As this information diffuses,  employers (who want lawyers who make their lives easier) will preference law schools with the better graduates, and law students (who want more and better career options) will follow suit. Until we have this level of conceptual and empirical clarity, we might as well be debating art or literature.

If students and employers are responding to particular curricula, it is reasonable to assume they are responding to perceived value (i.e., quality as a function of price).   I believe there are three steps needed to create a legal education curriculum that truly moves the market.

1. Clarity on Goals.  We need to understand the knowledge, skills, and behaviors that are highly prized by legal and non-legal employers.  Truth be told, this is tacit knowledge in most workplaces. It is hard intellectual work to translate tacit knowledge into something explicit that can be communicated and taught. But we are educators -- that is our job!  If we think employers are missing something essential, we can add in additional factors. That's our job, too.

2. Designing and Building the Program. Working backwards from our goals, let's design and build curricula that will, overall, accelerate development toward those goals.  This is harder and more rigorous than lesson planning from a casebook.

3. Communicating Value to the Market.  If our program is indeed better, employers and students need to know it.  This also requires a crisp, accurate message and a receptive audience.  This requires planning and effort.  That said, if our program truly is producing more effective lawyers, it logically follows that our graduates (i.e., the more effective lawyers) will be the most  effective way to communicate that message. 

Regarding point #3, in simple, practical terms, how would this work?  

During the 1L year, we show our law students the roadmap we have developed (step #2) and spend the next two years filling in the knowledge, skills, and behaviors needed to achieve their career goals.  This professional development process would be documented through a portfolio of work.  This would enable students to communicate specific examples of initiative, collaborative learning, problem-solving, or a fiduciary disposition, etc., developed during law school.  Students would also know their weaknesses, and have a clear plan for their future professional development. In a word, they'd stand out from other law graduates because, as a group, they would be much more intentional and self-directed (i.e., they'd know where they are going and how to get there). 

With such a curriculum in place, our law school would collaborate with employers assess the performance of our graduates.  By implication, the reference point for assessing quality would be graduates from other law schools.  When our graduates fare better, future graduates will be more heavily recruited.  Why?  Because when an employer hires from our school, they would be more likely to get a lawyer who helps peers and clients while adding immediate enterprise value.    

I suspect that many of my legal academic colleagues would argue the best law schools are not trade schools -- I 100% agree.  But I am not talking about a trade school model.  Rather, a world-class law school creates skilled problem-solvers who combine theory with practice and a fiduciary disposition. Graduates of a world-class law school would be reliably smart, competent, and trustworthy.  This is a very difficult endeavor. It takes time, planning, collaboration, creativity and hard work.  But the benefits are personal, organizational, and societal.  

At a practical level, I think few law schools have targeted this goal with a full, unbridled institutional commitment.  But the opportunity exists.

Applied Research 

When I got tenure in 2009, I decided that I was going to spend the next several years doing applied research. I am a fact guy.  Rather than argue that something is, or is not, better, I prefer to spend my time and effort gathering evidence and following the data.  I am also a practical guy.  The world is headed in this direction, thanks to the ubiquity of data in the digital age.  And, on balance, that is a good thing because it has the potential to reduce conflict. 

I have pursued applied work in two ways:  (1) building stuff (curricula, selection systems, lawyer development tools, datasets for making strategic decisions, etc.) and assessing how well it works, and (2) observing and measuring the work of others.

A Law School Curriculum Worth Measuring

A couple of years ago, a really unique applied research opportunity fell onto my lap.  I had a series of lengthy discussions on the future of legal education with Emily Spieler, who was then serving as dean of Northeastern University School of Law in Boston, a position she held for over a decade.  One of the raps on legal education is that it is more alike than it is different.  In fact, this very point was just made by the ABA Taskforce on Legal Education.  See ABA Task Force On The Future Of Legal Education, Report And Recommendations (Jan. 2014) at 2.

Emily, in contrast, said her school was unique -- that the curriculum better prepared students for practice and enabled them to make better career planning decisions.  Also, Emily stated that Northeastern students were more sensitized to the needs of clients and the privilege and burden of being a lawyer--specifically, that Northeastern grads become aware, before graduation, that their own lack of competency and diligence has real-world consequences for real-world people. And that reality weighed on students' minds.  

Tall claims.   But if Northeastern coulddeliver those outcomes more effectively than the traditional unstructured law school curriculum, I wanted to know about it.  

On a purely structural level, Northeastern Law is definitely unique.  Most law schools are organized on either quarters (University of Chicago, my alma mater) or semesters (Indiana University, where I teach). Northeastern, however, has both.  The 1L year curriculum at Northeastern is the traditional two semester model.  But after that, the school flips to quarters -- one quarter in law school, and one quarter in a cooperative placement with a legal employer, such as a judge, prosecutor’s office, a law firm, a corporate legal department, or a public interest organization.  

This classroom/coop sequence occurs four times over eight quarters.  Because the cooperative placement is not viewed as part of Northeastern's ABA-required course work -- all the contact hours are packed into two 1L semesters and four 2L/3L quarters -- students can be paid during cooperative placements.  And in any given semester, roughly 30 to 40% are getting paid. 

This system has been up and running for 45 years--over 5,000 students have become lawyers through this program.  What an amazing research opportunity! 

Now imagine the faculty meeting where the law professors get together to discuss and deliberate over whether to adopt the Northeastern model.  At Northeastern, "summer" means summer quarter, not summer vacation.  

How did this unique curricular structure come into being?  That is quite an interesting story. During the 1950s, the law school at Northeastern was shuttered.  Yet, reflecting the zeitgeist of the times, a group of Northeastern law alumni and young lawyers who were skeptical of their own legal education (at elite national law schools) petitioned Northeastern to reopen the law school and feature a more progressive, forward-looking curriculum.  The university administration agreed to reopen the law school on the condition that the school adopt the signature cooperative education model.  So this crucial decision was essentially made at the birth of the law school over four decades ago.  Once up and running, Northeastern Law implemented other innovations, such as the narrative grading policy--i.e., no letter grades and no GPA.  This was done in order to mitigate competition and encourage a focus on collaboration and skills development. 

The Outcomes Assessment Project

Back in 2011, my conversations with Emily Spieler eventually led me to make a two-day pilgrimage to Boston to talk with Northeastern Law faculty, students, administrators, and coop employers.  Suffice it to say, I was surprised by what I witnessed --a truly differentiated legal education with a substantial alumni/ae base spanning 45 years.  

That pilgrimage eventually led to my involvement in Northeastern Law's Outcomes Assessment Project (OAP), which is something akin to The After the JD Project, but limited in scope to Northeastern -- although Northeastern will provide all of the project tools and templates to other law schools interested in studying their own alumni.  From the outset, the OAP has been set up to scale to other law schools. 

There are lots of tricky methodological issues with Northeastern.  For example,

  • It has a longstanding public interest tradition; Northeastern Law is overrepresented in government service, public interest, and non-profit sectors (including a sizeable contingent of law professors and legal clinicians). See Research Bulletin No 1.
  • Its student body was over 50% female almost from the outset, nearly 20 years before legal education as a whole. 
  • Because of its progressive roots, GLBT law students have long been drawn to Northeastern Law -- again, nearly two decades before it was deemed safe to be out.

Because of this distinctive profile, we have to worry that any differences in graduates are primarily due to a selection effect (who applied and enrolled) versus a treatment effect (they got a different type of education).  That said, the admissions data show that Northeastern Law students are, like other law students, strongly influenced by the US News rankings.   If a student gets admitted to Northeastern Law and BC, BU, or Harvard Law, Northeastern seldom wins.  

Over the coming months, I am going to use OAP data to attempt to develop some analytical and empirical clarity to some of the questions surrounding experiential education.   Preliminary data from our Research Bulletin No 3 suggest that the coop program does remarkably well in developing the three apprenticeships identified by the Carnegie Report.  More on that later. 

Print version of this essay at JD Supra.


Data on legal education, Important research, Innovations in legal education, Scholarship on legal education | Permalink


If problem-solving abilities are high-end added value, why are doctrinal courses - which illustrate legal systems wrestling with socio-legal problems - not as likely as experiential education - which illustrate form processing - to provide needed education. I agree that we need more precision and, in particular, we need to remember the skills taught in nominally "doctrinal" courses.

Posted by: Robert Rosen | Feb 5, 2014 10:20:54 AM

No reform needed of the training lawyers now receive in neglecting to return phone calls from clients or in arriving late to closings without apology; they've got those skills down pat.

Posted by: Jonathan Silber | Feb 6, 2014 12:24:22 PM

Give it up. Law school can't be fixed. It can only be abolished. Law should be taught as an undergraduate major. Licensing should not depend on having a piece of paper from a school, but on character and competence demonstrated by actual supervised practice.

Posted by: Walter Sobchak | Feb 6, 2014 1:30:41 PM

My dear Professor Henderson,

I am afraid you have been led down a false primrose path with regards to the efficacy and outcomes of Northeastern Law School’s co-op program. In brief, it is rife with problems that the school in no wishes to even acknowledge, much less fix.

- The number of co-op employers only moves in one direction: up. This holds true even if there are no new co-op employers. In slightly better times, I have seen co-op employers who have not taken on an intern in five years or more – that is 20 co-op cycles – be listed as “active.” I have also seen entities that have dissolved or otherwise evaporate continue to be on the co-op employer listing for a few cycles. Harmless errors, subpar list maintenance, or venal subterfuge? I suppose that is in the eyes of the beholder.

- Co-ops themselves are of wildly variable quality, ranging from the superb to the blatantly atrocious. I will be the first to admit that some of the co-ops are of remarkably high quality, with varied, challenging, real legal work, undertaken in the presence of genuinely supportive and caring lawyers who may actually hire a student every now and again. But there are at least just as many who are small law firms looking for free labor or to tempt one to abuse their Wexis privileges, or judges looking for a free secretary, and so forth. “Work” can consist of basic proofreading of nonlegal documents, providing babysitting/daycare services to the boss’s progeny, running household errands, answering phones, and sundry other tasks more suitable to the most horrible of undergraduate unpaid internships. Some students are lucky and wind up with four great experiences; others are not and have four atrocities.

- Related to the last point, there are longstanding rumors that overly negative co-op reviews go to the trash (corroborated by trying to find reviews of co-ops by students known to have had a bad experience or at co-ops renowned for their terribleness). The implication is that the school is more worried about their relationship with co-op employers (and maintaining that ever-increasing number of participating employers) than about the quality of their students’ educational/vocational experiences. In other words, all manner of bad, and even unethical behavior on the part of the co-op employers gets tacit approval from the law school.

A word on outcomes. For many years, the school claimed 40% of its graduates were accepting job offers from one of their co-op employers. I do not know when this stat was introduced, but I have seen it extend as far back as 2003 in brief online searches. For many years, they stated “on average, nearly 40% of our graduates accept a job offer from a co-op employer.” In late 2009 or early 2010, that changed to “on average, 40% of our graduates accept a job offer from a co-op employer). Obviously, there would have needed to be a very significant uptick in co-op hiring for one year’s results to raise the average of the previous seven (or more) years’ performances. Indeed, even if the previous seven years were 37 for four years and 38 for three years – a fair reading of “nearly 40” – that most recent year’s performance would have had to jump to 58. In the height of the recession. To the extent of my knowledge, the school never revealed how it arrived at its numbers at any time in question; it was just another law school’s variant on the old warhorse “95% employment at $160k average.”

Anyways, the school continued to promote that 40% co-op stat even after their 2011 enhanced ABA disclosure revealed that only 90 of 182 graduates, or 49%, received those full-time, long-term, JD-required jobs within nine months of graduation, whether from a co-op employer or not. In my pseudonymous perambulations, I have learned that one of the law student advocacy groups questioned the veracity of that statistic – perhaps in light of its enormous incongruence with their ABA disclosures – and the stat has since disappeared for the first time in nearly a decade or more. The 2012 outcomes are even worse, with only 43% obtaining FT, LT, bar license-required jobs within nine months of graduation. According to the Faculty Lounge, this is the 161st best showing out of 197 law schools (or the 36th worst). In my quick glance at those FL rankings, I cannot find another law school with such an enormous variance between US News rank and job outcomes. To be blunt, it is only a few percentage points higher than Suffolk or New England Law Boston; both are fourth-tier law schools. This does not speak well for the prowess of the co-op program or for employers’ proclivities for (supposedly) better-trained law school graduates. As far as my experiences go, the co-op program has shot very wide of the mark with respect to all aspects of “market-moving” legal education curricula: Goals, design of program, and value to the market.

Goals and program design could be improved were there the will and aptitude to accomplish it; things that have been sorely lacking from my perspective on the law school. Value to the market, though, is an insurmountable obstacle – it will not overcome 130 years of the Cravath Method of Hiring. If anything, the prestige-as-proxy-for-intelligence-and-aptitude has become even more endemic with the recession and consequent glut of college graduates of every strip and variety. See Kellogg professor Lauren Rivera’s “Ivies, Extracurriculars, and Employment: How Elite Employers View Education Credentials” in the journal Research in Social Stratification and Mobility.

In short, then, the school is not unique, except in that it has co-ops instead of 1L and 2L summer internships, and “evaluations” (which are often no more than one sentence, regardless of one’s actual performance) instead of grades. Oh, and that one sentence almost invariably contains one of a few standard adjectives that most legal employers in the area automatically translate into a letter grade. The pedagogy is the same, the course offerings are the same, the professors are similarly bereft of practice experiences beyond 1-2 years in BigLaw many years ago, and the job outcomes are actually quite a bit worse than the average law school’s. As is so often the case in legal education, what we have here is no more than some more of the Emperor’s new clothes.

- One of Northeastern Law School’s hundreds of un/underemployed graduates

P.S. As for Northeastern being overrepresented in government service, only 13 of 215 graduates in 2012 found government work, according to the ABA disclosures. That does not much seem like overrepresentation to me. 11 out of 182 for 2011 grads.

P.P.S. “It happened about 100 years ago when a handful of law schools adopted the case method, required undergraduate education as a prerequisite to admission, and hired scholars as teachers. As a general matter, this was a far better education than a practitioner reading lecture notes at the local YMCA.” You are aware that Northeastern Law School started out of the Huntington Avenue YMCA in Boston, right?

Posted by: Unemployed Northeastern | Feb 9, 2014 9:59:05 AM

Post a comment