CSI: Google Scholar

Whether Google Scholar is going to seriously challenge the wexis's of the world is not my concern here.  Nor is its current lack of serious search functionality.  It is about what happens when you search Google Scholar because whether we like it or not,  if it is Google, "they" will use it.  Especially if Google decides to include case opinion results in the general Google search results.  So, in order to be prepared, I investigated how the new kid in town might influence or change someone's research results.

My test project:  Find cases in N.Y. courses on cosmetic surgery that didn't go so well.

My limits:  New York cases

My search terms:
I did what I imagine most law students would do.  This means my search was: cosmetic surgery

(Perhaps they might add negligence, but I'll keep it simple for the sake of the blog.)

So what happened?
I retrieved 50 cases from the three levels of court in N.Y. which included published and some unpublished decisions. Not so bad. However, four of the cases were listed twice.  First with its reporter citation and then further down in the list a slip opinion.  So the numerical result was really 46.  Note that one of these cases also had an appellate level opinion listed; however, you would need to know the publication patterns for N.Y. reporters to realize this as Google Scholar does not use their muscle to make the connection for you. 

The order that the cases were placed in remains a mystery.

Interestingly, two of the first four cases were fairly famous cases in N.Y., but not for torts centered around cosmetic surgery.  They are famous for parents raising religious objections to medical procedures performed on their child.  Neither of these cases, In re Siefirth nor Matter of Sampson, mention the phrase "cosmetic surgery." Seifirth was decided in 1955.  The term "cosmetic surgery" was not even being used. Sampson dealt with a blood transfusion and had nothing to do with cosmetic or plastic surgery  

So why are they in  my results list?

• Was Google adding odd metadata like they are doing with the book scanning project? 
• Was the search engine just sloppy? 

No, neither.

I learned the answer from Andrew Larick-Plumb's now infamous post on the Case Western law library blog. Apparently, Google scholar is not searching the text of the the cases.  It is searching the text of the documents listed in the "How Cited" tab that appears next to the case list and analyzes linking patterns to create a results list.  Sampson showed up in the hit list because in the literuture it is often linked with Seifirth. Both are early cases that discuss the parents' right to exercise their religious freedom in choosing health care for their children.

Comparing to Wexis:

I did not compare the case results from Google Scholar to other free resources because there are no other free resources that allow you to search across all three levels of NY courts.  I reran the same search in appropriate N.Y. cases databases on Lexis and Westlaw and limited my results to cases that were decided after 1949.  Of course nither Seifirth nor Sampson were in the results lists from either company because they do not contain the phrase "cosmetic surgery."  But neither were 10 other cases.

Westlaw gave me 45 hits and Lexis gave me 44 hits.  Between Westlaw and Lexis, there were three unique cases.  In Westlaw, three cases showed up that did not show up in Lexis though 2 of the 3 did show up in Google Scholar.  In Lexis, there was one case that was not reported in either Westlaw or Google Scholar, and one case that was reported in Google Scholar but was not reported in Westlaw.  The two cases on Lexis that were not reported on Westlaw were both reported only in the New York Law Journal which explains why it was not picked up in the case law database I chose on Westlaw (NY-CS).  That one case that was unique to Westlaw was an unpublished decision from a lower N.Y. civil court which helps explain why it was not picked up by either Lexis or Google Scholar.

With respect to Google Scholar as compared to wexis, there were a total of 10 cases picked up by Google Scholar that were not reported in either Westlaw or Lexis.  Only one of the nine seems to be relevant and I am not sure how to explain why the search in Westlaw and Lexis did not pick up that one case.  It was a 2008 slip opinion from the trial level that was about a cosmetic surgery gone bad.  None of the other 9 cases that were unique to Google Scholar had the sought-for phrase.  Considering the small set of results, this is rather a large subset to sift through.  I larger set of results would make it very annoying to work with.

From the other side, there were 12 cases reported on Westlaw and Lexis that should have been picked up by Google Scholar if it indexed its cases instead of creating linking patterns based on citations in literature, and had a full set.  In fairness, the missed cases were unpublished decisions that were mentioned only in an Appendix to the reporter or not published at all.  We are not going to miss them too much.

Conclusion:

Certainly, this method of case discovery has some interesting benefits.  It will, of course, pick up cases that are note worthy enough to deserve merit by an author.  From an academic perspective, this might be a way to jump start a note topic or seminar paper.  It isn't the best way to do so, but if we are looking for ways to use this new tools, this could be a good way.

Will I teach students about it?  Yes, of course.  And I will go through this same exercise with them so that they understand the process of the tool and where it fits in their tool bag.  I think once they run into snafus like what I found with Seifirth and Sampson, they may be more discriminatory about relying on it. (VS)