The WestSearch Straitjacket For Legal Research - Thinking Beyond The Keyword: Part I

Thomson Reuters - Legal (TR Legal) advertises sweeping claims for the benefits of WestlawNext (WN) at Customers. WestlawNext.com. Given the scale of the advertising blitz, these claims merit careful examination. Though it remains too early to reach a conclusion, initial evidence from its advertising suggests that TR Legal oversells WN for the benefits of its search engine, WestSearch. Despite benefits unique to its design, WestSearch has not only limitations from its own design, but the same type of keyword limitations that any other search engine has. As TR Legal markets WestSearch as if it should supplant other uses of WN for legal research. Its marketing reveals a one-size-fits-all reductionism at cross-purposes with WN’s versatility, but in perfect harmony with WN’s increased costs.

Lawyers must think like lawyers to correctly and comprehensively answer legal questions. Legal research therefore requires legal analysis and related problem-solving skills. As Kent Olson observes, “[l]egal research is not simply gathering information, but being able to analyze that information and grasp its significance. It is a process that requires a siginficant amount of time and focus.” (Principles of Legal Research (2009), at 8) Yet marketers for WN attach supreme value to speed in legal research. By failing to distinguish WestSearch and WN, they claim that WN - when they mean WestSearch - allows legal researchers to accomplish their research “fast” but without any loss of “confidence.” ("Meet The People Who Made [WN] Possible") WN retains less obvious features essential to research goals other than speed. As WN’s marketers try to attract interest in WestSearch, WN’s developers have left legal research educators back doors to help their students learn how to overcome WestSearch’s shortcomings, even if the adaptations will, in some instances, almost double “retail” costs.

In the first part of this series, we will separate the marketing hype from the realities of WestSearch. We will find that it does not necessarily advance effective legal research, even if it (modestly) advances effective keyword searching. We will identify some of its limitations, using them to underscore the value of “traditional” research methods still available on WN where WestSearch falls short. Examples alllow us to anticipate increased “retail” costs for traditional research uses of WN. In the second part of the series, we will examine a study for TR Legal that allegedly demonstrates the time-saving benefits of WestSearch. By underscoring WestSearch’s limitations, the study’s biases will help us better understand its benefits. We will also better appreciate the perils of reducing legal research to keywords in multiple databases. Legal researchers, of course, can not wear this kind of straitjacket. They must think beyond keywords.

TR Legal’s marketing campaign features comments by customers, including such law firm partners as Daniel Kelly, of Reinhart Boerner Van Deuren, and Kathryn Conde, of Nutter McClennen & Fish. Kelly says that WN "gives us a quick view of the question that we have to answer and then it leads us right down the road, each step of the way, to figure out exactly what the answer needs to be and how to present that." Conde claims that WN saves her clients money by producing "quick and correct answers" to legal questions. Other smiling customers, including librarians and associates, endorse WN (or rather WestSearch) in the same enthusiastic ways. Oddly enough, almost of them work at Reinhart Boerner or Nutter McClennen.

At Customers. WestlawNext.com, customers can "go behind" WN by exploring "The Science of WestSearch." Its developers associate characteristics with WestSearch that no other search engine has. Mike Dahn, Vice President of New Product Development, says that it ends "the tyranny of the keyword,” even though it depends on user-selected keywords. How are these apparently conflicting ideas reconciled?

To understand what Dahn means, we will begin with a fact: keyword searching has had a record of limited effectiveness.  Keyword searches in large databases fail to uncover all, and only, the documents that keyword searchers need to correctly answer their questions. Keyword searchers often end up with too many unhelpful documents or miss too many helpful ones. The constraints arise from the limitations of keywords, the incomplete knowledge of keyword searchers, and the “keyword irreducibility” of complicated legal questions.

First, keyword searches work only as well as keywords track the right concepts. The concepts meant may diverge from the keywords selected. In response to the problem, Dan Dabney helped develop KeyCite. Before joining West Publishing Company, he identified three aspects of this problem in The Curse of Thamus: An Analysis of Free-Text Document Retrieval 78 Law Libr. J. 5, 17-20 (1986). Whether as individual words or words in expressions, keywords imperfectly correspond to concepts because of ambiguity, synonymy, and syntactic complexity. Document authors use different words or expressions for the same concept (synonymy), the same words or expressions for different concepts (ambiguity), and words or expressions whose complex syntactic relationships designate conceptual relationships (syntactic complexity). Examples to follow will illustrate these types of language-related problems.

Second, keyword searches work only as well as the keyword searcher can identify all of the legal concepts needed to correctly answer a question. If you miss applicable concepts because you lack adequate context for the question, then you will also neglect keywords that might be useful to include in your search.

Third, a keyword search works only as well as a legal question lends itself to keyword compression. A complicated question involves a wide range of concepts, facts, and relationships among them. No one keyword search will prove sophisticated enough. And any number of keyword searches may not suffice to cover the entire conceptual and factual territory.

Its developers designed WestSearch to end “the tyranny of the keyword” by mitigating problems of synonymy, inadequately understood contexts, and complicated questions. (Peter Jackson & Khalid Al-Kofahi, Human Expertise And Artificial Intelligence In Legal Search (2010), at 3, 6) WN users may start their research by entering "simple descriptive words, terms and connectors, or even a citation" in WestSearch. (TR Legal, Westlaw Next Search Brochure (2010)) They need not identify a specific source or type of source. WestSearch’s algorithm weights and links topically related legal documents for importance and relevance, whether or not (1) the documents have the keywords entered; (2) the user has entered keywords to cover all of the applicable concepts; and (3) a complex question can be adequately compressed in keywords.

WestSearch represents a “further evolution “ from the Natural Language search engine of “Westlaw Classic” (WC) and other conventional search engines, which rank documents for relevance according to keyword properties. Conventional search engines deploy  “term-frequency-inverse-document- frequency”. Thus the less frequently a WC case database has the keywords of a Natural Language query, the more of these keywords will appear in top-ranked cases, the more closely together the keywords will appear in them, and the more often top-ranked cases will use these keywords. (Robert A. Berring and Elizabeth C. Edinger, How To Find The Law (12th ed. 2005) at 94, n.4) WestSearch’s design displaces the primacy of keyword properties in ranking search results.

So how does WestSearch work? WestSearch developers have issued a number of patent applications for the new technology. We can begin to understand WestSearch’s ranking based on at least two of them. The following explanations therefore represent a simplification.

WestSearch ranks legal documents for relevance to the issue and importance in use. To rank documents for importance, WestSearch has been designed to capture more than KeyCite’s data on citation frequency. It also captures crowdsourcing by WN users, or patterns in documents they link to, view, print, or save in folders. One crowdsourcing factor concerns the extent to which WN users click though document hyperlinks, especially among WN users who tend to do most of the click-throughs. (Jackson and Al-Kofahi, at 5; U.S. Patent Application No. 20100332520, published Dec. 30, 2010)

To rank documents for relevance, WestSearch requires a set of “starter” documents, such as cases, that satisfy keyword criteria of a Boolean, or “plain English,” search query. (U.S. Patent Application No. 20080033929,  “Exemplary Operation” at [0028 - 0050], published Feb. 7, 2008) “Term-frequency- inverse-document-frequency” supplies one of the “search capabilities”  for retrieving starter cases.” (Patent Application No. 20080033929, at [0021]; TR Legal, WestSearch: Westlaw Next Search Technology (2011), at 3) Search algorithms automate comparisons between starter cases and other cases. Comparisons involve extent of textual similarity between starter cases and other cases. The comparisons also involve shared “meta-data” or document characteristics - the number and importance of shared Key Numbers; the number of shared citations to cases and statutes; and the number of shared citing cases. Each compared case receives a rank, or relevance score, based on these comparisons and the case’s meta-data. The meta-data of a ranked case include the number of sources citing it; the number of sources cited by it; the number of starter cases related to it by shared Key Numbers, shared citations, shared citing sources, or other links; the number of relationships between it and starter cases; and the number of these relationship types. Thus the top-ranked cases may be linked by their Key Numbers to each other, and frequently cited for the same points of law. They may also be linked by shared citations and citing cases.

The ranked cases, in turn, have “meta-data” features that support further links to primary and secondary legal authorities. Based on relative weights, or ranking scores, attached to their meta-data features, ResultsPlus determines what additionally linked primary and secondary authorities appear in the results list, and in what order of relevance. (Patent Application No. 20080033929, at [0015]). ResultsPlus “suggests materials related to retrieved cases, including ALR annotations and Am Jur articles.” (WestSearch: Westlaw Next Technology, at 3) “Suggested” materials also include treatises, law reviews, and legal briefs and trial documents. (TR Legal, ResultsPlus Fact Sheet (2008); U.S. Patent Application No. 20050228788, published Oct. 13, 2005) Inventors of ResultsPlus developed it from an automated classification technology called the “Categorization and Routing Engine, or CaRE. (TR Legal Press Release, ResultsPlus for Statutes Search Now Available on Westlaw, Mar. 29, 2003) CaRE aids TR Legal editors when they assign Key Numbers to headnotes by “suggesting” Key Numbers for unclassified headnotes.  (To learn more about the CaRE invention, see U.S. Patent No.7,065,514 (issued Jun. 20,2006).

Cases represent an “exemplary embodiment” of the WestSearch technology. The technology has comparable applications specific to other categories of legal documents, such as statutes, regulations, administrative decisions, briefs, law reviews, and and treatises. For each “content category,” a search query retrieves a set of starter documents. Search algorithms automate comparisons between starter documents and other documents, ranking each compared document by extent of textual similarity and shared meta-data, and by the document’s weighted, meta-data characteristics. ResultsPlus then links the ranked documents with other primary and secondary sources. (See Jackson’s comments on ResultsPlus in: Jason Wilson, On WestlawNext, State of the Art & Steve Jobs: A Conversation With Peter Jackson, Chief Scientist for Thomson Reuters (Jun. 24, 2010)). Thus WestSearch has been designed to rank search results not just across, but within these “content categories.” It “is not simply a global or federated search of multiple databases containing different document types. Rather, each content type has its own search engine, employing a total of thirteen different algorithms.” (Human Expertise And Artificial Intelligence In Legal Search, at 6)

Nevertheless, if we rely on the Brochure, WestSearch generally works so well that its users can be confident that the top-ranked search results include only relevant and important documents, and that they miss no relevant and important documents. So if we credit the advertising, WestSearch’s developers have untied the Gordion knot of online searching. "The algorithm [used in the search engine] retrieves legal documents of high recall, as well as high precision." (WestSearch: Westlaw Next Search Technology, at 4) Because the search engine ranks relevance in the right ways, its users can expect to retrieve most of WN’s documents on the legal issue of interest (high recall). Because the search engine ranks document importance in the right ways, its users can also expect that most of a search’s results will include such documents (high precision), particularly among the top-ranked results. (On WestlawNext, State of the Art & Steve Jobs: A Conversation With Peter Jackson, Chief Scientist for Thomson Reuters)

Unfortunately, “the tyranny of the keyword” can also stand for an alien marketing ideology - the belief, contrary to common sense, that keyword searching exhausts legal research methods. If we think beyond the keyword, we will realize the difference between effective legal research and effective keyword searches - and the difference in cost. So suppose a single use of WestSearch typically achieved high precision and high recall. WN may not "transform" legal research even in two instances where its developers tell us it should. We have no evidence that running a WestSearch makes it any easier, faster, or cheaper to answer complicated questions than starting with secondary sources or consulting experienced practitioners. We also have no evidence that alternatives to WN work less well at helping legal researchers identify all the legal principles at issue when they lack subject expertise. In fact, we have the following reason to think WN works too well in one undesirable respect, by inflating costs.

Olson recommends that researchers learn more about unfamiliar topical contours of their questions. They should do  “some preliminary research … to understand the context of a particular problem and to get some sense of the rules of an area of law.”  (Legal Research In A Nutshell (10th ed. 2010), at 32) Subject treatises, for example, can help them in traditional areas of law, and legal encyclopedias may help if no subject treatises are available. Law reviews, legal newspapers, and legal blogs can afford insights about new or emerging areas of law. Preliminary research may also help researchers learn the topic’s “legal jargon” to prepare better keyword searches. (Finding the Law, at 13-14

Of course, WN search results may alert legal researchers to issues they would otherwise overlook. But they need not start with a WN search to remedy this deficiency. Secondary sources in a law library appear more likely than WN searches to save legal researchers time, and their clients unnecessary search costs, as long as researchers have ready access to a law library, and know how to effectively use the sources or receive instruction. At "retail," transactional rates, a WN search costs $60, and then $42 to view a section of a “premium state [or] speciality title.” (TR Legal, WestlawNext Pricing Guide for Commercial Plans (Feb. 2010)) WN search results may include a larger collection of relevant secondary sources. But you may incur unnecessary cost to run even one search on WN, and view several documents, at least if you can gain the needed context from a treatise or legal encyclopedia in your institution’s law library. Moreover, it will cost almost twice as much in WN than in WC to view “speciality title” sections. (TR Legal, Westlaw Pricing Guide For Private Price Plans (Apr. 2010))

Though hardly decisive, available evidence at least suggests that WestSearch will not achieve high recall and precisions for certain types of questions, even though they frequently arise in legal research. WestSearch’s design causes at least two important problems; the complexities of language cause others for all search engines.

WestSearch will not accommodate research to find and understand the required legal concepts, if finding and understanding them depends on factors other than those ranked by WestSearch's algorithms. In Does WestlawNext Really Change Everything?, Ron Wheeler challenges WN's effectiveness for questions whose answers depend on "the obscure or less popular results" that crowdsourcing would not uncover. (103 Law Libr. J. 359, 366 (2011)) In addition, he says that "[t]he legal researcher using WestlawNext and doing broad searching will be exposed not only to fewer cases, but also to a far narrower range of related content." (372) In other words, as a legal researcher learns more about a legal question or problem, the researcher may lose opportunities to better define it by missing relevant legal authorities or secondary commentary.  So WestSearch may not achieve high recall for broad search queries. In Part II, we will see why it may not, based on Patent Application No. 20080033929.

Moreover, WestSearch does not close significant gaps between concepts and keywords used to express them, even if it narrows such gaps. The slipperiness of language limits the usefulness of all keyword search platforms, requiring legal researchers to think less like keyword searchers and more like lawyers. Let us now turn to examples of these limitations. The examples involve three types of questions quite common to legal research. Where WestSearch might lead to (expensive) dead ends, other research methods may lead to correct and comprehensive answers. Thus if a researcher understands a legal problem’s context, the researcher may start by identifying a promising source for an answer, and then navigate its content through its map-like system of organization. In our examples, legal researchers will need to use research aids, such as annotated codes, legal encyclopedias, and treatises, and will need to use associated navigational devices, such as indexes, topical outlines and tables.

These alternative methods will cost more to use in WN than in WC. We will base our transactional cost comparisons on two pricing plans for private-sector customers: Westlaw Pricing Guide For Private Price Plans (Apr. 2010) and WestlawNext Pricing Guide for Commercial Plans (Feb. 2010). Our price comparisons will hardly settle opinion on WN’s overall cost impact, especially because almost no WC or WN customer pays “retail.” But they will appear to fit the wonder-bra principle of cost shifting: squeeze the costs in in one direction and they pop out in another.

Consider first the problem of synonymy. Suppose a terminally ill man stores sperm for his wife to use after his death. The couple wants to ensure that a child born under these circumstances has the same right to inheritance as if the child had been born while his father was alive. Do California couples have a legal procedure to protect this form of inheritance? They do, under Cal. Prob. Code §249.5, but that provision references “posthumous conception,” “genetic material,” and a maximum period when the “child was in utero.”. Technical language of this kind abounds in statutes and regulations. Use of at least one of these phrases increases the odds of a successful keyword search. So would starting with a specific source, such as West’s Ann. Cal. Code. But an attorney new to this area of law may overlook such helpful phrases when searching WN’s entire collection of California legal materials. Suppose the attorney enters store sperm terminally ill in WestSearch, limiting the content to just “California” as the jurisdiction. The attorney will not find Cal. Prob. Code §249.5 in the search results, and may end up wasting money by viewing documents not on point. But the attorney could instead use the index to West’s Ann. Cal. Prob. Code, if available, at a savings of at least $60 and perhaps considerably more. Under the entry for “Children and Attorneys,” the attorney would find the entry “Posthumous conception, probate proceedings, Prob 249.5 et seq.” Whatever their shortcomings, the index entries require the user to think like a lawyer.

Now consider the problem of ambiguity with respect to a sample question. Under what circumstance, if any, can an attorney appearing as amicus curiae file an appeal on behalf of a party? This unlikely circumstance occurred in Matter of Additional Magistrates for St. Louis County, 580 S.W.2d 288 (Mo. 1979). A court issued an order making appellant “a party as amicus.” The Additional Magistrates court decided that the appellant retained standing to appeal, because the lower court treated the appellant as a party. For keyword searching, the ambiguity arises from  the pedestrian sense of “party to a case” and the more exceptional sense of “amicus as a party.” A skillful WN user would need to run several searches of federal and state case law  just to eliminate irrelevant cases, such as cases  in which courts barred amici from raising new issues on appeal. Nevertheless, even a few advanced WestSearches (whether Boolean or in “plain English”) will not likely uncover Additional Magistrates. (If Additional Magistrates was frequently cited or “crowdsourced” among WN users, WN searches might stand a better chance of success.) The cost of several searches may approach $200. But a legal encyclopedia - 4 AmJur. 2d. Amicus Curiae §6 - affords a legal researcher a crucial alternative, because it has a citation to Additional Magistrates. Browsing the “Amicus Curiae” outline of AmJur. 2d will point a researcher to “II. Rights and Powers of Amicus Curiae - §6 Generally.” That amorphous “Generally” would tempt the curiosity of an experienced researcher. But temptation has its costs. Retrieving this section in WN will cost about twice as much it would in WC.

Finally, legal researchers may need to answer syntactically complex questions. Dahn claims that WestSearch can “break down the words and phrases in legal documents … [and] recognize the syntactic relationships among them.” (WestSearch: Westlaw Next Search Technology, at 3) But his claim does not survive the test of syntactic complexity. Dabney offers us an example of a syntactically complex question:

“If a person a waives his or her right to trial by jury in one trial, can a jury trial still be demanded in a subsequent new trial of the same matter? The key words for this question, 'trial,' 'jury,' 'waiver,' and 'retrial,' are common in judicial opinions, but discussions of the specific point of law of the question are relatively rare. A computer cannot reliably find cases that are on point because too much of the meaning of the desired cases is tied up in the syntactical relationships between the words, which are not 'understood' by the computer." (78 Law Libr. J. 5, 19-20)

To try answer a syntactically complex question, a keyword searcher must first identify legal phrases or keyword combinations specific to the legal concepts involved in the question. Sometimes no specific legal expressions apply, or the researcher can not anticipate expressions that would apply. Sometimes the applicable legal expressions do not help. Legal language often causes these difficulties.

Although Dabney was discussing Boolean searching, his observation also applies to Natural Language searching in WC and “plain English” searching in WN. Entering "simple descriptive terms" in WestSearch will no more help a researcher answer Dabney's question than using Natural Language or Boolean searches in WC. To correctly and comprehensively answer the question, a researcher should instead identify a likely secondary source and use its table of contents, topical outline, or index. Suppose you are a California attorney. Anticipating criminal law as the likely context, you could start with Witkin & Epstein, Cal. Criminal Law (3d ed. 2000). Using WN or WC, you could browse the table of contents to find the relevant discussion:

Chapter XIV. Criminal Trial - X. RIGHT TO AND SELECTION OF JURY - A. Right to Jury Trial - 4. Waiver of Right. - h. [§ 459] Effect of Waiver on Retrial.

WN users can still access the table of contents of Cal. Criminal Law and other secondary sources. But it will cost them more to do so. Under a private plan’s  "retail," transactional rate, it costs $24 to retrieve Cal. Criminal Law § 459 from WC; however, it costs $42, or almost twice as much, to retrieve this section from WN.  This difference in cost has significant consequences for charges to clients and cost recovery, because the difference will have iterations every time WN users rely on WN’s print-based features. Thus Cal. Criminal Law § 459 cites to People v Solis, 66 Cal.App.4th 62, 77 Cal.Rptr.2d 570 (1998). The case would lead you to another helpful secondary source:

Although there are no published opinions in California specifically addressing whether a jury trial waiver remains in effect for a subsequent retrial of the same case, the contention has been considered and rejected in many other state and federal courts. (United States v. Groth (6th Cir.1982) 682 F.2d 578; United States v. Lee, supra, 539 F.2d 606, 610; People v. Mixon (1994) 271, 111. App.3d 999, 208 Ill.Dec. 385, 387, 649 N.E.2d 441, 443; People v. Hamm (1980) 100 Mich. App. 429, 298 N.W.2d 896, 898; State v. Di Frisco (1990) 118 N.J. 253, 571 A.2d 914, 930; see also Annot., Waiver of Right to Trial by Jury as Affecting Right to Trial by Jury on 573-573 Subsequent Trial of Same Case in Federal Court (1984) 66 A.L.R.Fed 859, 869, § 7 and cases cited.)” (77 Cal.Rptr.2d 570, 572-73).

(Links omitted)

It costs $24 to retrieve the ALR document from WC, but $46 to retrieve it from WN. And if the ALR led to other secondary sources, the cost difference repeats itself.

In summary, we have seen how three types of legal questions can disable keyword searching as a useful research method. The questions tend to to disassociate or sever the ways in which search algorithms match keyword combinations with documents having the right legal concepts. Therefore, they typically require legal researchers to think beyond the keyword. Answering them correctly may require source-specific, research methods more finely tuned to use of analogies, distinctions among similar or related legal rules, and other forms of legal reasoning. The research may take more than a few minutes, especially if the researcher must use multiple research aids and methods. Fortunately, WN, no less than WC, still provides users research aids to leverage the print-based organization of particular databases. But using them on WN will cost more.

By overemphasizing WestSearch as the principal or exclusive way to use WN, WN's marketers reduce legal research to "simple descriptive terms,” applied to searches across a wide range of different types of sources. Yet the analysis legal research often requires can not be reduced to “simple descriptive terms.” So WN’s design reveals a troubling reductionism that  perpetuates another form of “keyword tyranny.” As it has been marketed, the WN model represents a radical departure from the model of traditional research that Wheeler describes in his article. The “traditional model” requires a researcher to understand the context of a legal question, the likeliest sources for answering it, and the structure of these sources. Under the traditional model, researchers also understand the trade-offs involved in using Boolean and Natural Language searches, search engines, and print-based or “pre-coordinate” methods of research.

In Part II of this series, we will examine TR Legal’s evidence that  WestSearch supports fast research by achieving high recall and precision. TR Legal relies on an “independent” study whose biases reveal WN’s reductionist tendencies and invite worries that the study’s designers lack independence. But the study will help us assess WestSearch’s benefits.

[Anon.]