Putting West and LexisNexis Case Law Digesting Systems to the Relevancy Test

Susan Nevelow Mart's (Faculty Services Librarian, UC Hastings Law Library) has uploaded Reining in the Results: The Use of Human Indexing and Computer Algorithms in West’s Headnotes & Key Numbers and LexisNexis’s Headnotes & Topics as Tools for Finding Relevant Case Lawto SSRN in draft form. The article was presented at the Conference on Legal Information: Scholarship and Teaching, held at the University of Colorado Law School as part of its Boulder Summer Conference Series. So while it appears to be a work-in-progress, it should be very near final.

Mart took an admittedly small sample, "ten pairs of matching headnotes from legally important federal and California cases and reviewed the cases in the results sets generated by each classification," to compare these two very different systems, West's human indexing and LexisNexis' algorithmic indexing systems. So any results have to be qualified from the start. Not a complaint, the task is simply too much for one person to perform comprehensively.

Among Mart's findings: the average percentage of relevant cases found using West’s Key Numbers was 83.2%, LexisNexis’s More Like this Headnote was 62.3%, and Lexis Topics was 40.5%. Mart writes:

Since a major difference between the two systems would seem to be the degree of dependence of algorithms for creating classification topics and assigning headnotes to each topic, the role of human editors appears to be a definite advantage in returning relevant results. It seems that it is only the Key Number search results that are limited to those cases where the language of the headnote is present, and if limiting terms are present, where those terms also appear. This is definitely an advantage, but the advantage is not inclusive of all relevant cases.

Why? Because the LexisNexis classification system also returns a substantial percentage of relevant cases not found in West's Key Number System: 44% in LexisNexis More Like This Headnote and 28.2% in Lexis Topics.

See also Mart's findings for her study of using headnotes as limiters in KeyCite and Shepard’s. Mart conclusion in a nutshell: don't rely on one vendor's tools for comprehensive case law research. Mart's advice and article is highly recommended. [JH]