Wednesday, June 13, 2018

Reasonable Expectations About Education Data Puts a Damper on "Solving Problems No One Has Solved"

Christopher Elmendorf and Darien Shanske recently published the article “Solving ‘Problems No One Has Solved’: Courts, Causal Inference, and the Right to Education” in the Illinois Law Review.  Elmendorf and Shanske advance several important arguments but two in particular. First, the reason that the unavailability of quality data on public schools presents a serious obstacle to litigants in education rights case. This lack of data and the uncertainty it produces around the operation of public schools, Elmendorf and Shanshke rightly note, has been exploited by both the left and right to advance depending on the specific case and legal theory. Their core insight is that while this has been the state of affairs for decades, it need not be: it is completely within the state’s control to design and provide access to these systems. They argue further that judges should do more to direct the state to collect and provide this access in a way that sheds light on key educational questions.

This argument about the accessibility of information is paired with their second key insight: thanks to major advancements in administrative data sets and methodological advancements in data analysis, we are in an unprecedented position to discern the causal relationship between key aspects of our education system. Judges and litigants, Elmendorf and Shanske argue, should embrace this revolution in “causal methods” to banish much of the uncertainty in what “works” in education—a key to adjudicating educational rights and defining appropriate judicial remedies.

This is an important and compelling argument--one we are glad that Elmendorf and Shanske have made. But (you knew there would be a ‘but’) as educational researchers, we  (Ethan Hutt & Morgan Polikoff) think it is important to calibrate expectations about what the embrace of the causal revolution is likely to secure. In our reply, we advance three primary arguments.  First, we underscore just how intractable the problem of access to education information has been historically. Whether one looks at early administrative data from the 19th century, the development of measures like NAEP in the middle of the 20th, or modern administrative data on course taking patterns, data collection efforts have been shaped by political pressures. Collecting accurate information about American schools is hard and it is made even more challenging by the lack of uniformity in our system. So even when we have statistical uniformity it may mask important underlying variation (and inequality).

Second, though the causal revolution is a pivotal moment in education history, it is not a panacea. Despite the major changes in data collection, methodological innovations, and computing power, there are many questions that are unlikely to give way to these new developments. There are still many questions that are unlikely to yield definitive answers and many non-technical questions about the design and operation of our school system will remain.

Finally, to try and ground our consideration of how Elemendor and Shanske’s proposal might work in practice, we consider it in the context of the case of Williams v California. This case sought to secure educational rights by requiring the state to fulfill its responsibility to monitor the operation of schools through the production of better educational data.  We wrote our reply in the hopes of introducing and engaging a broader audience around issues of considerable importance at the intersection of education data, research, and litigation. We would welcome any and all replies to our reply.

| Permalink


Post a comment