More on evaluating research on new programs

Based on some of the feedback I have received about  my post from Wednesday, I am expanding my discussion about why we need to consider our students when we are implementing new programs based on research. This is an area where I have personal experience, in two different ways.  I am in the unique position of having attended classes at or worked at six different law schools in four areas of the country; UNC Law (my alma mater), Duke Law, UCONN Law, Whittier, ASU--Sandra Day O'Connor, and Vermont Law School.  But I also have personal experience evaluating the research from peer institutions when making decisions about new initiatives and classes.  Based on my experience, the single most important variable when evaluating whether a program will achieve desired results is the students.  Students aren't a monolithic, one-dimensional variable. There are multiple sub-variables to consider.

It would be more than a blog posting, more like a journal article, for me to detail why I would suggest considering each and every factor I listed in my last post. I will examine a couple of key factors as examples of how and why to careful evaluate research before implementing new programs based on research from other schools.  I will be making some generalizations based on my experience; your experience may be different.  My goal is to encourage you to look carefully at how students may impact the research results, and how this may impact the success of a new program. 

One of the critical factors to examine is whether the research was conducted at a school with day only, or day and evening students.  Day and evening students have some dramatic differences.  Demographically, evening students tend to be older, have more work experience, are more likely to be supporting a family, and much more likely to be working while in law school. There are great benefits to schools having evening programs; my experience is they are more focused students, devoted to becoming lawyers, and more mature than their daytime counterparts.  But time is at an even greater premium for these students than for day students. Evening students with families or working even part-time don't have any extra time to relax, let alone participate in supplemental programs, even when it will be of great benefit in the long run.  Time and money constraints have a dramatic impact in the programs they will attend, how they respond to new programs, and the time they can put into extracurricular programs, such as Bar/Bri and PMBR.  I haven't seen any research on the success of evening students as compared to day students on the bar exam, but my guess is there would be a difference.  If a supplemental bar program or bar prep class is evaluated using evening students or day and evening students, I would expect the success rate to be much lower than if the same class is evaluated using day law students only.  It's not a measure of the program, but an outcome of the time constraints of the students.

The location of the law school is also a significant factor to be considered when evaluating research. A city school with numerous other schools in the area is going to be very different from a rural school with few or no other law schools within hundreds of miles.  A school without other schools in the area is more likely to serve a student body with a diverse range of abilities.  If a law school is the only one within a hundred mile radius, some students will attend, even if they could have gone to a higher-ranked law school, because they are locked to the region.  Evaluating programs in a school that has an LSAT range of 147-165 is different from evaluating programs at a school where the LSAT ranges from 150-153. Let me emphasize that LSAT is not destiny, but it is a factor when evaluating whether a program will work with your students. Teaching to a wide variety of abilities results in different teaching methods, and in some cases, different outcomes.  This factor overlaps with the public/private issue; if the only other law school in the area is private, or much more expensive, you will see some of these effects as well.

The history of the law school is a very important factor, with multiple variables.  A new law school is creating a culture and a legacy. They don't have alumni war stories about the bar exam to rely on for student buy-in of programs.  Without a strong culture and legacy, students also don't have misinformation to the same degree as students who have a wealth of bad advice built into the culture of the school.  New law schools also don't have the stigma, or burn-out, if they don't have a great record with the bar exam.  An older law school with more than a few lackluster years can develop a culture of failure than sends self-defeating messages to the students. One such message is that no one from Law School X passes the bar exam on the first try, so take it the first time as a trial run, or just for practice. If a school is implementing a new program while simultaniously trying to overcome the burden of law student stigma, the results of a new program will not be reliable for a couple of years.  The results of the program need time to be decoupled from the efforts to change the law school culture. 

Another variable relating to the history of a law school is the history of the academic success program. A law school with a well-established, reputable ASP program that has outreach during the 1L year will find it much easier implementing a program for 3L's.  When the students already trust ASP, they will buy-in sooner, and put more effort into what you are asking them to achieve.  Similarly, if a law school has not had ASP, but is looking to establish a 3L bar prep program for the first time, they need a  different marketing strategy and should expect a more conservative student response.   I am a strong advocate for starting ASP programs incrementally, starting with 1L's, and gradually introducing programs for upperclass students.  The other effect ASP will have on the success of a new program relates to the skills base of the students. Law schools with a well established 1L ASP that focuses on basic skills will have 3L's with a better foundation for bar courses. It's hard to build a foundation when students have already made it through 2-3 years of law school; you wouldn't try to pour a foundation after building a superstructure.   Any program that is starting with 3L's without a 1L program will need more time to achieve results, and an even longer time if the school isn't planning on creating a 1L program to introduce skills to students at the beginning of law school.

This is not an exhaustive list of factors to consider when thinking about implementing a new program. I hope I have provided an illustration of the kinds of factors to consider when considering implementing new programs based on the research of other schools.   I made some generalizations about students based on my experience, and you may disagree with some of them.

Lastly, if your school is considering implementing a new program, and would like to talk to me about some of the things to consider, I would be happy to chat with anyone on this topic.  (RCF)