The Legal Whiteboard

Who Are These Companies?

Below is a photo of the exhibitor list at the LegalTech New York trade show.  [Click on to enlarge]

There was a lot of money sloshing around this trade show.  What do these companies sell?  How do they make money?  Who are their clients? Who founded these companies and who financed their growth? Are lawyer-employees a key part of their business models?  These are the questions I am asking.

If you think that the ethics rules (MR 5.4) are keeping nonlawyers out the legal industry, you need to come to LegalTech.  

[Posted by Bill Henderson]

Washington & Lee is Biggest Legal Education Story of 2013

Here it is in a nutshell. There is empirical evidence that Washington & Lee’s experiential 3L curriculum is delivering a significantly better education to 3L students—significantly better than prior graduating classes at W&L, and significantly better than W&L’s primary competitors. Moreover, at a time when total law school applicants are on the decline, W&L’s getting more than its historical share of applicants and getting a much higher yield. When many schools are worried about revenues to survive next year and the year after, W&L is worried about creating the bandwidth needed to educate the surplus of students who enrolled in the fall of 2012, and the backlog of applicants that the school deferred to the fall of 2013.

[This is a long essay.  If you want it in PDF format, click here.]

Alas, now we know:  There is a market for high quality legal education. It consists of college graduates who don’t want to cast their lot with law schools who cannot guarantee students entree to meaningful practical training.  Some might argue that W&L is not objectively better-- that the 3L curriculum is a marketing ploy where the reality falls well short of promotional materials and that, regardless, prospective students can't judge quality. 

Well, in fact there is substantial evidence that the W&L 3L program delivers comparative value.   The evidence is based on several years' worth of data from the Law School Survey of Student Engagement (LSSSE). I received permission from Professor James Moliterno, someone who took a leadership role in building W&L’s third year program, to share some of the key results (each school controls access to its LSSSE data.) They are below.

But before getting into empirical evidence, I want to put squarely on the table the most sobering finding that likely applies to virtually all of legal education. It is this: On several key LSSSE metrics, W&L has made impressive gains vis-à-vis its own historical benchmarks and its primary rival schools. But even for this leader, there remains enormous room for improvement. More on that below.

Here is the bottom line:  Traditional legal education, when it is measured, does not fare very well. Yet, as W&L shows, substantial improvement is clearly possible. We law professors can respond to this information in one of two ways:

1. Don’t measure, as it may disconfirm our belief that we are delivering a great education.
2. Measure—even when it hurts—and improve. 

I am in the second camp. Indeed, I don’t know if improvement is possible without measurement.  Are we judging art work or the acquisition of key professional skills needed for the benefit of clients and the advancement of the public good? 

Moving the Market

I doubt I will ever forget Jim Moliterno’s September 2012 presentation at the Educating Tomorrow’s Lawyers (ETL) conference at the University of Denver. He presented a single graph (chart below) showing W&L actual applicant volumes since 2008 versus what would have happened at W&L if its applicant volume had followed national trends.

While law school applicants crested a few years ago, W&L enjoyed a large run-up in volume of applicants, presumably due to the launching of their new 3L program.  This larger applicant pool effectively served as a buffer when applicant declines began in 2011 and 2012.  Since 2008, overall law school applicants are down -19%, yet W&L is up overall +33%.   

But much more significantly, after their experiential 3L year was up and running and the overall legal job market continued to stagnate, W&L yields spiked. Ordinarily they would enroll 135 students. But for the fall of 2012, they received enrollment commitments from well over 260 students. Indeed, at the ETL conference Jim Moliterno said the school had to offer financially attractive deferments to get the class to approximately 185 incoming students -- a 50 student bulge.

When Jim Moliterno showed the above graph and explained the corresponding changes in yield, my good friend Gillian Hadfield, a skeptical, toughminded, evidence-demanding economist who teaches at USC Law, leaned over and said to me, “that is the single most important takeaway from this entire conference.” I agreed.  The market for a legal education with practical training is, apparently, much more inelastic than the market for traditional JD programs.

Yet, what is perhaps most remarkable is that a large proportion of incoming students at W&L were enrolling based on little more than faith.  Nobody knew for sure if W&L  had the ability to pull off their ambitious 3L curriculum. The program relies on a large cadre of adjunct professors, after all, and W&L is located in remote Lexington, Virginia.  Many law faculty outside of W&L, and perhaps some inside, thought (or perhaps think) that the program could not live up to the hype.  Well, as shown below, the program appears to have produced meaningful gains.

The only data-driven critique anyone can muster is that the gains remain significantly short of perfection. But that critique bites harder on the rest of us.  To use a simple metaphor, W&L is tooling around in a Model-T while the rest of us rely on horse and buggy. What ought to be plain to all of us, however, is that, just like automobile industry circa 1910, we are entering a period of staggering transformation that will last decades. And transformation will be roughly equal parts creation and destruction. See Schumpeter.

W&L Data, Internal Historical Benchmark

LSSSE is a phenomenally rich dataset – nearly 100 questions per year on a wide variety of topics related to student classroom experience, faculty interaction, type and quantity of assessments, time allocation, and perceived gains on a variety of dimensions related to personal and professional development. The survey instrument is online here.  

Aside from a host of questions related to demographics, career goals, and debt, major sections in the LSSSE include:

• Section 1, Intellectual Experience (20 questions)
• Section 2, Examinations (1 question)
• Section 3, Mental Activities (5 questions)
• Section 4, Writing (3 questions)
• Section 5, Enriching Educational Experiences (9 questions)
• Section 6, Student Satisfaction (7 questions)
• Section 7, Time Usage (11 questions)
• Section 8, Law School Environment (10 questions)
• Section 9, Quality of Relationships (3 questions)
• Section 10, Educational and Personal Growth (16 questions)

W&L deserves to be a detailed case study. But frankly, legal education can’t wait. So I will do the best I can to cover the landscape in a blog post. I hope every law faculty member who reads this post makes a strong plea to their dean to enroll in LSSSE. Why? So your school can benchmark itself against the detailed LSSSE case studies that are bound to flow out of W&L and other innovative law schools.  Though they don’t get much press, there are, in fact, other innovative law schools.

The dataset I have for W&L covers 2004, 2008, and 2012.  This is the same data that Jim Moliterno briefly shared at the ETL conference.  I have put them into bar charts so that readers can see the scores on several questions at once.  Two important interpretative notes:

1. LSSSE is especially useful when an entire class (1L, 2L, or 3L cohort) experiences a curricular change.  This happened with Indiana Law's 1L Legal Professions class.  It is also happening here, as all W&L 3L students had the benefit of the experiential 3L curriculum.  Assuming nothing else signficant has changed (a safe assumption when it comes to legal education), the classwide change enables a simple "event study" analysis.
2. W&L LSSSE scores for 2004 and 2008 are much more alike than they are different. The big differences appear between 2008 and 2012. So that is what I discuss below.

Section 1 differences are displayed below (3L students only).  Click on the chart to enlarge.

 
The big takeaway here is that W&L gained in 17 out of 20 categories.  Because Section 1 is put on a 4 point scale, just like a traditional academic grading system, we can analyze the data using something akin to a LSSSE Section GPA .  W&L's Section 1 GPA for 2008 was 2.52, which is essentially on the C+/B- cut point.  Only one factor -- communicated with faculty via email--was meaningfully above a 3.0. 

We can contrast that with a 2.85 GPA for 2012, which is in the B-/B territory.  W&L's overall average increased by .33 points, and six measure are above 3.0.  It experienced the biggest gains on the following:

• +.77, Put together ideas or concepts from different courses when completing assignments or during class discussions.
• +.75, Participated in a clinical or pro bono project as part of a course or for academic credit.
• +.53, Put together ideas or concepts from different courses when completing assignments or during class discussions.
• +.51, Worked with classmates outside of class to prepare class assignments.
• +.49, Prepared two or more drafts of a paper or assignment before turning it in.
• +.47, Discussed assignments with a faculty member.
• +.44, Used email to communicate with a faculty member (now a 3.65).
•  +.43, Talked about career plans or job search activities with a faculty member or advisor
• +.41, Worked with other students on project during class

There is still enormous room for improvement, but W&L's 3L experiential program appears to have really moved the needle on factors related to the Section 1 Intellectual Experiences factors.

 W&L fares even better on Section 3, which covers the mental activities that ostensibly comprise "thinking like a lawyer." [Click on chart to enlarge]

As shown above, W&L 3Ls drop in only one category -- rote memorization for repeating on an exam.  Surely, that pleases the W&L faculty.  These are 3Ls after all.   The overall Section 3 GPA, which excludes 3a, moves from 3.07 (B) to 3.41 (B+).  Question 3c to 3e are true higher order lawyering skills. W&L ought to wheel out these data the next time some bar association claims that legal education is not accomplishing anything.  At some places, maybe.  But good things appear to be happening at W&L.

Washington & Lee shows similar gains in the other key LSSSE sections.  If you are curious, you'll have to wait for the detailed W&L case study, which I hope will get written someday by someone at W&L.  What is no doubt of greater interest to the broader legal education community, however, is how well W&L is doing against other law schools--i.e., like us.

W&L Data, External Peer Benchmarks

 LSSSE data are the property of law school who pay for the survey.  The survey is designed to improve the education programming rather than create an industrywide ranking.  Roughly 50% of law schools participate each year.  Since its inception in 2003, 179 law schools have participate for at least one year.

Although the data are reported at the individual school-level, comparative benchmarks are a key part of the LSSSE value proposition.  Comparative benchmarks include size, public/private, the total LSSSE sample, and a peer group specified by the school.  For example, at Indiana, we might want to look at other Big 10 public law schools.  We don't get to see our rivals' scores, individually, but we can get a group average for five or more schools we select that are also participating in that specific year.

I am told that schools typically pick their peer groups based on similar rank, geography, and applicant pool, etc. I thought W&L's peer comparison would be the most relevant to show here.

Below are the 11 (out of 20) factors in LSSSE Section 1 in which W&L is higher than its peer benchmark at statistically significant levels.  Again, only 3Ls in the sample I am using here. [Click on to enlarge]

On these 11 benchmarks, W&L posts a "GPA" of 3.02 (B) versus 2.45 for the peers (C+).  Again, W&L has plenty of room to grow, but relatively speaking, it is dramatically outperforming its competition.

What about those critical Section 3 Mental Activities that comprise "thinking like a lawyer"?  Again, W&L is outdistancing the competition.  [Click to enlarge]

The Section 3 GPA, excluding 3a memorization, is 3.41 for W&L (B+) versus 3.12 for its peers (B).

Section 4 pertains to writing.  Ask any professional development coordinator in a law firm about the biggest weakenesses of incoming associates, and you'll get a near unanimous reply: "writing."  Well, the best way to become a better legal writer is to write.  How did to W&L 3Ls do on that front?  3L students at W&L write a ton. [See chart below, click on to enlarge.]
W&L 3Ls write roughly the same number of 20-page papers as those at peer schools, but in the 1-4 and 5-19 page category, W&L 3Ls surge ahead of the competition at statistically signficant levels. In the above chart, the 3.27 score for papers in the 5-19 page range corresponds to 6-7 medium length papers during the 3L year.  Peers, in contrast, are roughly at 3 medium length papers. The 3.68 score in the 1-4 page category also equals roughly 7 short papers during the 3L year; peers write roughly half that number, roughly 3-4 short numbers.

Section 7 covers time usage.  Not surprisingly, W&L 3Ls spend more time prepping for classes beyond just reading assigned text -- roughly 7 hours more per week. [See chart below, click on to enlarge.]

Imagine if you did anything seven hours more per week for two full semesters -- you'd get better at it.  Time on task really matters to professional development.

Section 9 focuses on the quality of relationships within the school.  In terms of 3L student relationships with faculty and administration, they are quite high -- indeed, higher at statistically significant levels than W&Ls peer schools. [See chart below, click on to enlarge.]

Finally, Section 10 asks a series of questions related to how well the law school experience has contributed to the student's knowledge, skill and personal development.  [See chart below, click on to enlarge.]

On 10 of 15 questions, W&L is posting higher scores than its competition -- all at statistically significant level.  But as I noted above, there remains room for improvement.  W&L Section 10 "GPA" is 2.99 (B).  Its competitor's GPA is 2.7 (B-).

Conclusion

There are three takeaways from this blog posts:

1. A sizeable number of prospective students really do care about practical skills training and are voting with their feet.  W&L has therefore become a big winner in the race for applicants. 
2. W&L's 3L experiential curriculum is substantial improvement over the curriculum W&L offered in 2004 and 2008; moreover, there is room for even more improvement.
3. There is substantial evidence that W&L, with some modest focused energy on the curriculum, is now offering a better educational experience than its peer schools -- albeit, the current grade is a "B" at best for W&L and likely lower for the rest of us.  We all, therefore, have a lot of work to do.

The example of the Washington & Lee 3L experiential year ought to be a watershed for legal education. We can no longer afford to ignore data.  Through LSSSE, high quality comparative data are cheap and comprehensive.  And that information, as we have seen, can significantly improve the value of a legal education.

[Posted by Bill Henderson]