Sunday, October 18, 2009
Edward Cheng has sent me the following response to my post on his article A Practical Solution to the Reference Class Problem.
Many thanks to Alexandra for raising a number of good questions about the implications of my article. I’ll try to address two of them here.
i) The question of value.
One undoubted limitation of the use of model selection methods (at least in the regression context) as a means for resolving reference class type problems is that you need to have a measure of outcome. Thus, the ideas in the paper work well when what we want to predict is the market value of a house or the pre-exposure risk of cancer. Where they do not work straightforwardly are areas determining commonality in class action cases, because there, you really don’t have an obvious target for prediction
One possibility for analyzing commonality through this lens is to use cluster analysis and the “cluster selection” tools that accompany them. (Thanks go to Richard Nagareda for spurring this idea.) Cluster analysis is about figuring out how to sensibly construct groups, and I think may be a fruitful avenue. More details to come as my work progresses.
The other big issue that Alexandra raises is about the “relevance” of the predictors. How do we know that we’ve gotten all of the important predictors, or put differently, how do we know when our model is “right”?
As a response, I have to admit that I am in many ways advocating for a far more practical and data-driven perspective than what we conventionally see in social science studies of law. I think we need to view model selection methods as an attempt to make the best predictions given the available data. Take property valuation for example – I’d argue that we’re not really interested in the true model of property valuation; all we want is a reasonably accurate prediction of what the house would have sold for on the market. Might we get greater accuracy ultimately if we understood the underlying phenomenon better? Possibly. But until we do, I think the model selection methods are powerful ways of making do with what we have. And arguably, that’s what the legal system does anyway. We aren’t in the business of ultimate truths. We’re in the business of resolving cases based on the evidence at hand.