The Empirics and Ethics of USNWR Gaming

Posted by Bill Henderson and Jeff Lipshaw

The Wall Street Journal article on gaming the USNWR this morning prompts us to a joint posting of something we've been discussing with each over the summer.  We see this particular exercise - the use of part-time programs to affect a school's standing - as a real-world intersection of empirical data, the meaning one derives from that data, and the ethics of manipulating the data. 

Put simply, in a world in which small moves in the USNWR rankings (much less big ones!) take on meaning within the context of the "rankings game," it is no wonder that the players play the game.  Indeed, the players are co-opted into the "rules" of the game to the extent that observers from the outside question the moral sense of those playing the game (the point here).  As Nancy Rapoport put it so eloquently in the Journal piece and on her blog, what is the difference between using the gray areas of the USNWR formulas to move up the rankings, or manipulating the gray areas of GAAP to report higher earnings?  And what is the implication of this gaming in terms of the ethical leadership of those teaching new lawyers?

The Data

Let's review the data.  Regarding the proposed inclusion of part-time students in the USNWR rankings formula, there is no need to speculate on which schools will be affected:  the ABA-LSAC Official Guide to Law Schools publishes LSAT and UGPA medians for full-time, part-time, and combined entering classes.   For example, below is an LSAT/UGPA grid from the recently released 2009 edition.  An inspection of the LSAT and UGPA medians suggests that this school is going to get absolutely hammered by the proposed change:  -4.0 for LSAT points and -.26 for UGPA.

The school above, however, appears to be one of the biggest gamers of the part-time loophole.  Before reviewing the data, however, let us offer some context.

Here is the problem.  We aren't at all suggesting that schools with part-time programs are gaming.  But as the Journal article and our data shows, under the current system, the part-time loophole opens the door to gaming because schools are able to keep students whose statistics are a drag on the numbers out of the calculation.  For example, if FT and PT numbers were combined, there is at least the possibility that a large number of older students with excellent work experience will be less likely to gain admission.  During his guest blogging stint at Prawfsblawg back in June, Bill Araiza, who formerly served as Associate Dean of Loyola LA (which has a part-time program), summarized the tradeoffs if we see the worst case:

[The inclusion of part-timer in the rankings formula] is potentially quite pernicious, as it will put pressure on law schools to curtail part-time (especially evening) programs' focus on older students whose life and work experience may offset any deficits on the more standard admissions credentials (i.e., LSAT scores and GPA).  This is especially true with regard to LSAT scores.  ...  If those students are going to start counting for U.S. News ranking purposes admissions committees are going to start giving those students less of a break on that criterion, even if their post-college accomplishments give all kinds of reasons to expect the applicant to succeed in law school. 

Note what is really be said here:  "If US News combines full with part-timers, the part-time programs will be subject to the same foolish, myopic tradeoffs as the full-time."  People with excellent work and life experience also apply to full-time law programs.  Prof Araiza implicitly acknowledges that we don’t give these characteristics the proper weight when it could affect our school’s US News ranking.  Last month, Bob Morse of US News weighed in on this pattern of tradeoffs, observing that the combined rankings really do reflect the entire school more accurately, suggesting that the problem of schools gaming the statistics or changing substance to adapt to the form is really one for the law school accreditor: the ABA.  In short, “we just report the numbers; we can’t be responsible for the foolish or myopic tradeoffs.”

To illustrate the effect of US News on law school admissions decisions, let’s assume for the sake of argument that part-time programs are a sanctuary for whole person review rather than vehicle for manipulating US News.  Therefore, to get a glimpse of how law school admissions would operate in world without rankings pressures, we compare full and part-time admissions among schools with both programs.  Below is a summary of average median LSAT and UGPA scores for the 96 fully or provisionally accredited ABA law schools with part-time programs (from the 2008 ABA-LSAC Official Guide):

Note the difference:  Within the universe of schools that offer both full and part-time programs, the typical part-time student has a lower UGPA (-.10) and a lower LSAT (-3.4) than his or her full-time counterpart.   Of course, we are not implying that the part-timers are weaker students; with part-time largely exempt from US News, admissions officers are free to make the type of sensible tradeoffs prescribed by Bill Araiza.  As compared to a 22 year-old, running a business or non-profit for five years or a tour of duty in the military ought to be worth a couple of LSAT points or .3 UPGA.

The "combined" column tell us the price to be paid for this wholistic review:  If these admissions policies are kept in place and US News adopts the proposed methodology, the typical law school with a part-time program can expect a drop of .03 units in UPGA and 1.2 LSAT points.

Now let’s relax the part-time-as-sanctuary assumption and be open to the possibility that, at some schools, the part-time program has become a cynical way-station to collect tuition from lower credentialed students that it would have admitted to the full-time program in the pre-US News era.  For example, consider the following facts that apply (in reality, this is not a hypothetical) to the same law school summarized in the above LSAT/UPGA grid—i.e., the one that will get hammered by the proposed US News formula change:

• In 1992, the first year that US News ranked all fully accredited ABA law schools, this law school enrolled 177 full-time and 74 part-time students.  In the early to mid-1990s, it fluctuated between Tiers (or "quartiles) 3 and 4.
• In 2004, the school had 89 full-time and 84 part-time students--quite a change--and broke into the second tier.
• According to data in the 2009 ABA Official Guide, the pattern is now fully reversed:  76 full-time and 121 part-time.  Further, the part-time yield is remarkable:  149 offers and 121 matriculated (81.2%)--that is better than Yale's full-time yield of 78.4%.  In recent years, the school has fluctuated between Tiers 2 and 3.
• In addition to full-time day, the school has part-time that runs during the day and at night. That seems odd.

Last year, the dean who presided over this transformation was hired by another lower ranked law school.  The news coverage made it very clear that he was hired for his track record of taking a school from Tier 4 and Tier 2.  Indeed, the dean publicly proclaimed that very goal to the local media.

Can you guess one of his strategies for moving up in the rankings at his new school?  Shrinking the full-time program while expanding the part-time program?   Here is a comparison of figures from the 2008 and 2009 ABA-LSAC Official Guides.

• 2008:  287 full-time, 95 part-time
• 2009:  211 full-time, 170 part-time

The large increase in PT enrollment was achieved by doubling the number of part-time offers, from 171 to 352.  On the school's website, the profile of the 2007 entering class has full-time day and part-time evening numbers.  Below is screen-shot of the actual page:

Note the qualifying text under the grid: "An additional 70 students were admitted as Part-Time Day."  Gee, during the most recent admissions cycle, the school added 75 part-timers while dropping 76 full-timers.  Further, the school's website says: "Applicants to the full time day division who do not meet the admissions criteria may be offered admission into a part time day division." 

Another section of the website explains how this new program works: "[A] new Part-Time Limited Day Program … allow[s] incoming students to complete their first year in three continuous semesters—fall, spring and summer. Instead of taking five classes in the fall, students take three classes and make up the remaining two classes—Torts and Criminal Law—the following summer. … By the beginning of the second year, students in the [PT Day] program are attending law school full time with their 2L peers. …"

And the US News payoff from this rescheduling of student courses?

• 2008 medians:  LSAT, 153 FT / 153 PT; GPA, 3.24 FT / 2.98 PT
• 2009 medians:  LSAT, 155 FT / 151 PT; GPA, 3.34 FT / 3.13 PT

A 2 point gain in LSAT and a .10 gain in UPGA. The school also moved from the Tier 4 to Tier 3.  The proposed changes to the US News rankings will shut down these shenanigans, which, we suspect, have nothing to do with admitting experienced older students who will enrich the classroom. Note that this dean has been lauded for his mastery of US News--indeed, it appears to be the reason why he was hired. 

What Does the Data Mean?

If you look at the data compiled by the Journal today, one of the striking things is how small many of the moves would be.   That's not surprising.  In April 2007, Lipshaw (who is no statistician!) observed that if you charted schools by the number at each peer assessment ranking, you ended up with a chart like the one at the left (based on the then current rankings):  excepting the top seventeen schools, right-skewed bell curve.    For large groups of schools, there is no perceived difference in reputation, and getting all worked up about a move up or down by a couple of digits seems misplaced.  Just like there probably isn't a whole lot of difference between the performance of the students who get between 3.0 and a 2.7 GPA equivalent on exams with a mandated mean of 2.85, there really isn't a whole lot of difference in small moves within the group bunched between a 2.8 peer assessment score (the schools at that level are BYU, Florida State, Alabama, Miami, Oregon, Pittsburgh, and San Diego) and a 1.7 peer assessment score (schools at that level are California Western, Capital, New England, Northern Illinois, Roger Williams, South Texas, St. Mary's, Texas Wesleyan, and Touro).  Perhaps a jump from a 1.7 to a 2.8 would signify a real change, but we could question the significance of most of the small moves inside that range.

But people, including academics, give data meaning--not the other way around.  So if meaning arises here (for whatever reasons, including behavioral or cognitive biases, including our round number and base ten biases) because of ordinal ranking, and the way USNWR splits the tiers, and each spot between 1 and 188 means exactly the same thing, it's not surprising that players (deans, faculty, students) ignore the bell-shaped curve, and care, for example, whether the school is ranked 96 or 102.

Lessons in Ethics and Leadership

What does this matter?  At least three reasons:  (1) most of the time, the "gaming" gains don't actually accomplish very much in terms of movement up or down, weakening the rationalization that it's being done for the "greater good" of the school (not the kind of good cost-benefit judgment we'd like to see our students exercise!) (2) gaming compromises the role-modeling credibility of the legal educators who do it; and (3) all legal educators now have to deal with the "well, everybody does it" rationalization for pushing the edge of the legal and ethical envelope, making their jobs harder, just like ethical corporate lawyers pushing back against CEOs and CFOs have to contend with the "everybody does it" rationalization for accounting manipulation, backdating options, or other arguably benign exploitations of the rules.

We've observed that this kind of sub-optimization to “make the numbers” happens out in the real world all the time.  If a company rewards its executives for short-term numerical targets, like inventory turns, or working capital turns, or quarterly earnings, or sales, that differ either from the fundamental number (say, annual earnings) or, worse, damage the company in the long run (e.g. to make this year’s number we cut all R&D spending) you can expect that the executives will respond to the incentives.

To bring it closer to home, it’s exactly the ethical problem we worry about when a law firm sets billable hour goals for its partners or associates.  What then drives the determination whether work actually needs to be done?  The client’s need or the lawyer’s statistics?   One has to wonder about schools teaching “padding your hours” ethics at the same time they are doing this gaming.  The point of ethical leadership is to demonstrate a separation of your self-interest from the choice to do the right thing.

Below the fold you can find the chart of the 90+ fully accredited ABA schools that have part-time programs based on the size of the gap between part-time and full-time/part-time combined statistics.  It is sorted first by LSAT gap, then by UGPA.  Slightly more than half gain 1 or more LSAT point by having a part-time program.  As we noted above, these are all the part-time programs, and we aren't suggesting any gaming other than the publicly acknowledged instances we've discussed.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

Law School	Gap between FT & PT/FT combined
	LSAT	UGPA
Toledo, University of	-3	-0.29
Seton Hall University	-3	-0.07
Southern Methodist University	-2	-0.16
Quinnipiac College	-2	-0.14
Mcgeorge School of Law	-2	-0.1
Illinois Institute of   Technology	-2	-0.08
Hofstra University	-2	-0.06
Fordham University	-2	-0.05
St. John's University	-2	-0.05
Brooklyn Law School	-2	-0.04
Michigan State University -   Dcl College	-2	-0.03
Western New England College	-2	-0.01
Thomas M. Cooley Law School	-2	0
Yeshiva University	-2	0
Chapman University School of   Law	-2	0
Rutgers University-Camden	-2	0
Akron, University of	-2	0.02
Cleveland State University	-1	-0.09
William Mitchell College of   Law	-1	-0.08
Loyola University-Chicago	-1	-0.08
Hamline University	-1	-0.07
Stetson University	-1	-0.06
Louisville, University of	-1	-0.05
Indiana University -   Indianapolis	-1	-0.04
St. Louis University	-1	-0.04
Suffolk University	-1	-0.04
Widener University	-1	-0.04
American University	-1	-0.04
Northern Kentucky University	-1	-0.03
George Mason University	-1	-0.03
Denver, University of	-1	-0.03
Seattle University	-1	-0.02
Arkansas, Little Rock,   University of	-1	-0.02
Connecticut, University of	-1	-0.02
Texas Wesleyan University	-1	-0.02
San Diego, University of	-1	-0.02
Pace University	-1	-0.02
New England School of Law	-1	-0.02
Maryland, University of	-1	-0.01
Georgetown University	-1	-0.01
Florida International School   of Law	-1	-0.01
Catholic University of America	-1	-0.01
Rutgers University-Newark	-1	0
Loyola Marymount   University-Los Angeles	-1	0
Depaul University	-1	0
Alabama, University of	-1	0
Inter American University of   Puerto Rico	-1	0.01
Dickinson School of Law	-1	0.07
Touro College	0	-0.24
Southwestern University	0	-0.1
Baltimore, University of	0	-0.09
New York Law School	0	-0.06
Dwayne O. Andreas School of   Law	0	-0.06
Puerto Rico, University of	0	-0.05
Duquesne University	0	-0.05
South Texas College of Law	0	-0.05
Capital University	0	-0.05
Houston, University of	0	-0.04
William S. Boyd School of Law	0	-0.04
San Francisco, University of	0	-0.04
Memphis, University of	0	-0.03
Wayne State University	0	-0.03
Widener University-Harrisburg	0	-0.03
Golden Gate University	0	-0.03
Whittier College	0	-0.03
George Washington University	0	-0.03
Syracuse University	0	-0.02
Tulsa, University of	0	-0.02
Missouri-Kansas City,   University of	0	-0.02
John Marshall Law School	0	-0.02
Regent University	0	-0.02
Santa Clara University	0	-0.02
Valparaiso University	0	-0.02
Pontifical Catholic University   of P.R.	0	-0.01
Marquette University	0	-0.01
Detroit Mercy, University of	0	-0.01
Temple University	0	-0.01
Lewis And Clark College	0	-0.01
Oklahoma City University	0	-0.01
Loyola University-New Orleans	0	-0.01
Creighton University	0	-0.01
Florida Coastal School of Law	0	-0.01
Northern Illinois University	0	0
Nova Southeastern University	0	0
Miami, University of	0	0
Georgia State University	0	0
California Western School of   Law	0	0
North Carolina Central   University	0	0.02
Thomas Jefferson School of Law	0	0.03
Southern University	0	0.16