Adding to our extraordinarily popular "Teaching Assistants," "Reefer Brief," and "Friday Frivolity" features, I've decided to add a new "What's All the Fuss About?" series, in which we have a look at contracts scholarship that everyone is downloading. The first in this series, avant la lettre was last year's post on Dave Hoffman and Yonathan Arbel's smash hit, Generative Interpretation. This time, we have a look at Lawyering in the Age of AI, by Jonathan Choi (left), Amy Monahan (below, right), and Dan Schwarcz (below, left) (collectively "the Authors").
Because we are all terrified that AI will replace us, most studies of AI have tested how well various large language models (LLMs) do when asked to perform certain tasks or undertake some bit of legal analysis. The Authors are more hopeful that AI will become a tool that humans can use to assist in legal work. To that end, the Authors have conducted "the first randomized controlled trial of the effect of AI assistance on human legal analysis." (3) They randomly assigned sixty students a to complete various legal tasks, some with, some without the assistance of GPT-4. The students received a limited amount of training on using GPT-4. The Authors then blindly graded the results. (3) The results were that the students' work improved slightly, but their efficiency improved markedly, with time savings ranging from 11.8% to 32.1%, depending on the nature of the task. (4) Based on these findings, the Authors recommend that legal educators begin thinking now about how to incorporate AI in to teaching and that, for legal practitioners, the time to embrace AI is now. (4)
The paper provides a quick overview of the interaction of technology and lawyering in the past fifty years. Earlier innovations "improved lawyer efficiency rather than fundamentally altering the core skills needed to be an effective lawyer." (5) Early evidence suggests that LLMs might become the great equalizer. Some studies show that they help weaker students considerably but do not really assist strong students at all. (8-9) The Authors' results provide more evidence of this pattern. (16) While the performance of AI on its own varies widely across different fields, there is already strong evidence that AI holds great promise when used to assist humans in performing intellectual work. (12) The Authors also surveyed their test subjects about their impressions of how helpful the AI was. Their "respondents accurately perceived how useful GPT-4 was for specific tasks." (31)
Reflecting on the findings, the Authors observe that "AI tools like GPT-4 and a limited amount of training can substantially improve the efficiency with which they complete a broad array of legal tasks without adversely affecting (or even slightly improving) the quality of that work product." (31) But if anything, the Authors think their results understate the potential uses of AI for attorneys, because (1) more specialized AI tools will become available; (2) users will become more familiar with the capabilities of LLMs and use them more effectively, and (3) the rate of improvement in LLMs is so rapid. (31-34)
In light of their robust findings, the Authors' normative conclusions include some surprises. They start with a qualified defense of the first-year law school curriculum and suggest that use of AI should be banned or substantially limited in traditional first-year doctrinal classes so that students can develop their legal reasoning skills without being able to use technological shortcuts. In addition, because AI enables everybody to work more efficiently but helps weaker students improve their work, while not improving the work of strong students, formative assessments should not involve use of AI. (38) However, the Authors also advocate for the creation of courses for upper-level students that train them in the use of AI. (39)
Lawyers and law firms should likewise be exploring how best to use AI to improve efficiency and quality of work product, while also being cognizant of the dangers associated with sloppy use or misuse of AI. (39-41) Their clients should know that attorneys and law firms need to achieve some efficiency gains through the use of AI. Clients may want to bring more work in house. However, clients also need to understand that AI will not always promote efficiencies that mean less expense. In adversarial situations, including litigation but also contract negotiation, AI will just allow the game to be played with new layers of complexity, but the game may not end more quickly or at lesser expense. (41-42) Judges too ought to allow attorneys to use AI to assist in writing briefs in order to reduce the costs of litigation and improve access to justice. (42-43)
https://lawprofessors.typepad.com/contractsprof_blog/2024/01/whats-all-the-fuss-about-lawyering-in-the-age-of-ai.html
