Does Academic Support Matter? A Brief, Preliminary Response to Blinded by Science and its Progeny

In August of 2020, an author published Blinded by Science? A Reexamination of the Bar Ninja and Silver Bullet Bar Program Cryptids.  The first of three articles analyzing FIU Law’s Academic Excellence Program ("AEP") and its methods, Blinded by Science contended that:  (1) transfers and attritions, and not teaching students the science of learning, constituted a significant cause of FIU Law's bar pass rate; and (2) the AEP overstated the impact of our academic support measures upon those rates. 

After many months of remaining silent as we prepare a more extensive rebuttal, the recent release of a third article prompts us to address the matter, if only in this truncated fashion.  Our conclusion, which we articulate in the second post, will surprise the reader.

-------

Blinded by Science and its progeny employ flawed methodologies that lead to an overstatement of the impact of transfers and attrition.  Through cherry-picking favorable information and ignoring confounding data, the articles weave a pastiche of unsupportable claims.  Blinded by Science lacks any discussion of common and necessary data analysis tools, like data modeling, regression analyses, or correlation matrices, for instance.

Instead, it asks the reader to look at raw data and draw unsifted assumptions.  Those assumptions include, among others, that:  (1) FIU Law’s part-time students do not count towards an analysis of the school’s pass rate (they do); (2) the February bar exam does not count towards an analysis of the school’s pass rate (it does); (3) all FIU Law transfer-in students had LGPAs above 3.00 (they did not); (4) a transfer 1L LGPA of 3.00 or higher all but guarantees bar passage (it does not); (5) no students transfer out of FIU Law (they do); (6) no dismissed students are ever readmitted (they are); and, most problematically, (7) non-academic attrition is the same as academic attrition (it is not). 

This last assumption best demonstrates the article’s flaws.  Blinded by Science looks at ABA Form 509 data to claim that FIU Law’s academic dismissal rate is high and thus likely disproportionately impacts its bar passage.  However, instead of focusing only on academic attrition, i.e., the students a law school dismisses, it instead conflates dismissals with non-academic attrition.  For instance, on page 14 of Blinded by Science, the author says that FIU “removed students from that cohort at an attrition rate of 10.8%,” (emphasis added) and then noted that this was higher than other schools with similar LSAT medians.  But, when one looks to the ABA Form 509 to which the author cites for this number, we see that the 10.8% is the combination of academic dismissals and non-academic dismissals (including transfers out).  The author repeats this error, and the “removed” language, in each of the data years he traces.  Given that FIU Law’s non-academic attrition comprised between 38-56% of its overall attrition during this period, this conflation significantly alters the analysis. 

Another striking assumption occurs in the author’s latest article.  There, he notes that BARBRI has isolated five factors that account for 79% of the variance in bar passage.  He focuses on just one of those factors, 75% UGPA, and observes that “[t]he first year that FIU had a higher 75th UGPA matriculating class than [the state’s flagship school] and the highest in the State of Florida was 2012. And unsurprisingly, three years later, in 2015, its bar passage dominance began.”  Thus, the author posits that a change in just one metric – 75^th UGPA – caused FIU’s “bar passage dominance.” 

What the article does not mention, however, is that FIU Law’s 75% UGPA actually dropped that year by .03.  Thus, the article suggests that FIU’s almost 11-point increase in absolute bar pass rate from 2014 to 2015 somehow was caused by a .03 decline in 75% UGPA.  The author also ignores the other four BARBRI factors, each of which would likely negatively impact FIU’s bar pass rate.

Another:  While relying so heavily on the theory that the 75% UGPA metric accounts for FIU’s increased bar pass position, the author assumed that this statistic remained the same in perpetuity.  Instead, we see that the 75% UGPA of the flagship school surpassed FIU’s in three of the five subsequent years.  If the author is correct that the 75% UGPA metric is so powerful that it foretold FIU Law’s increased bar rate placement, one would expect that the same result would occur in reverse.  But, in those three years when the 75% UGPAs reversed, the predicted reversal in bar rate placement did not occur.

Another:  While positing that FIU Law’s transfer/ attrition rates likely positively and substantially impacted its bar pass rates, the articles do not mention that the group that achieved FIU Law’s strongest bar passage result (2019, 95.7%) was the cohort with one of the fewest numbers of dismissals and transfers-in since the law school’s first 1L cohort in 2002.

Another:  Blinded by Science does not account for the impact of students transferring out of FIU Law.  While suggesting that transfers-in must be substantially improving the law school’s pass rate because of those students’ academic strength, the article does not report that in the period in question, high-GPA FIU transfers-out often substantially counterbalanced the number of transfers-in.  Because students usually transfer “up,” these transfers-out likely had a far higher bar pass likelihood than transfers-in.  As such, losing these students likely substantially offset any benefit from transfers-in.

There are many other “anothers” that we will detail in our forthcoming article.  In the meantime, our next post in this series will actually draw one rather surprising conclusion:  We actually agree with Blinded by Science.

[Louis Schulze & Raul Ruiz, FIU Law]