Back to blogging after diverting my energy in recent weeks to putting out a few fires.
I’ll assume here that you’ve read the first post of this new series. So you know that overfitting is a terminal problem for the Shining Path (SP) estimates published in the statistical report of Peru’s Truth and Reconciliation Commission (TRC). According to the TRC’s own stated standard for overfitting, the models in no fewer than 47 out of the TRC’s 58 geographical strata are inadmissible. And many of the TRC’s models are not just somewhat out of bounds; they’re egregiously overfit.
What about underfitting?
That is, does the TRC extrapolate any SP estimates from models with inadmissibly bad fits, even according to the TRC’s own stated fitting standards? Yes they do. The impact of this violation is small, however, since the TRC extrapolates from a really badly fitting model only in stratum 32. Yet stratum 32 is quite an interesting case. So I devote the rest of this post to it.
Recall that Silvio Rendon and the TRC use different methods to estimate SP-caused deaths. The details need not concern us here. The salient point for this post is that Daniel Manrique-Vallier and Patrick Ball (MVB), two authors of the original TRC report, assert that Rendon’s method is biased toward underestimation.
MVB do head-to-head comparisons with Rendon’s SP estimates in 9 strata and MVB’s published numbers place Rendon’s estimates below the TRC’s in 8 out of these 9 strata. These results are broadly consistent with the idea that Rendon’s method is biased downward although they are equally consistent with the idea that the TRC’s method is biased upward.
It turns out, however, that Rendon’s SP estimates are higher than the TRC’s in stratum 32. Here are the numbers:
TRC – 328
Rendon’s preliminary estimate – 751 (before multiple imputation)
Rendon’s main estimate – 877 (after multiple imputation)
Unfortunately, MVB don’t say that stratum 32 goes against the grain of their argument. To the contrary, Figure 4 in their supplementary materials wrongly claims a tie, placing the estimates of both Rendon and the TRC just below 600.
Figure 4 also asserts that any estimate below around 330 (eye balling the graph) is impossible. Yet their own (dubious) methodology for defining “impossibility regions” would place this boundary at 170.
I’ve argued before that all of MVB’s figures are misleading and should be corrected. This is because they airbrush statistical uncertainty away and assume perfect accuracy, both in the underlying data and in the matching of deaths across sources. But figure 4 reaches a new level of wrong.
My first series argued that MVB’s defense of the TRC work is weak. The discovery of mistakes can only diminish further its persuasiveness.
Finally, please keep your eye on the ball which is the TRC report itself. Here are the main take-home points so far in this series.
- The overfitting problem – this is sufficient to dismiss the whole SP portion of the TRC report..
- The underfitting problem – this makes the TRC’s problems a little bit worse
Of course, MVB should correct the mistakes in their stratum 32 figure. Beyond that I wonder whether there are other mistakes out there waiting to be discovered.