This post continues my series on Silvio Rendon’s rejoinder to the reply of Daniel Manrique-Vallier and Patrick Ball (MVB) to Silvio Rendon’s critique of the statistical work of Peru’s Truth and Reconciliation Commission (TRC).
OK that’s complicated so how about this?
The present series covers a second paper by Rendon that definitively shreds the TRC’s estimates.
Today we cover yet another dimension of unacceptably low quality in the TRC’s work. Many of the TRC’s Shining Path (SP) models predict that some lists, or combinations of lists, used by the TRC record negative numbers of people killed.
You must be doing a double take right now. So let’s go back to
basics on how the TRC’s estimates actually work. The data for each of the TRC’s 58 geographical strata consists of three lists of people killed, plus information on the overlaps across these lists. In particular, the TRC’s SP data for each stratum gives the number of deaths appearing on:
- list A alone
- list B alone
- list C alone
- lists A and B but not list C
- lists A and C but not list B
- lists B and C but not list A
- lists A, B and C
The goal is to estimate the number of people in each stratum who were killed by the SP but who didn’t make it onto any of these three lists.
(Note – Here I’m only discussing estimates of SP-caused deaths because MVB have not contested Rendon’s estimates for deaths attributed to the State, which are about 40% higher than the TRC’s estimates.)
There are two key steps to the estimation:
- Fit a model to the above 7 data-points.
- Extrapolate this model to an estimate of unrecorded deaths. This second step is enabled by an assumption that the deaths not recorded on any list are just like the deaths that are recorded on a list – except for the obvious difference that the unrecorded deaths are…well…unrecorded.
Here is a short video that explains the danger of extrapolating beyond your data range as this estimation method requires. The discussion is set in the different context of linear regression models but the main ideas still apply. The extrapolation method is likely to give inaccurate results if deaths not appearing on any list are qualitatively different from deaths appearing on at least one list. In fairness, however, the same risk applies to Rendon’s estimates so let’s just set it aside for the rest of this post.
Each model of the 7 data-points in a stratum predicts the 7 data-points themselves as well as the unknown data-point (deaths not on any list). These predictions should not exactly equal the true recorded figures; this would constitute extreme over-fitting, as discussed in post 1 of this series. Nor should the fit be extremely loose, as discussed in post 2 of this series. The fit should be somewhere in between to provide a decent chance at good prediction of unrecorded deaths. This triple of pictures below illustrates the argument nicely although, again, in a different context.
The point of the present post is that many of the TRC’s predictions for its known data-points are bat-shit crazy. In no fewer than 26 of the TRC’s 58 strata at least one such prediction is a negative number (Table 5).
In fact, Rendon shows that 40 out of the 58 SP models either fit perfectly or predict a negative number of deaths for at least one list or list combination (Table 5). And 57 out of the 58 models fail the TRC’s own standards for over-fitting or under-fitting or they make negative predictions.
There’s quite an irony here. MVB hang their whole critique of Rendon’s alternative SP estimates on their claim that some of these estimates are, in some convoluted sense, “impossible”. I argue here that this is wrong and Rendon strongly bolsters this point is his second paper. Still, that was the MVB critique. Now we learn that many of the TRC’s estimates are, literally, impossible: no list of deaths can contain negative number of them. So this critique is analogous to a jet engine calling a cat noisy.
To be clear, the TRC never predicts a negative number for deaths not recorded on any list. The negative predictions are always for numbers that are already known. But this in-sample character of their impossible predictions makes the problem worse, not better. The issue was readily apparent back when the TRC made its estimates but they plowed on anyway.
Recall, furthermore, that the TRC used an unconventional indirect method for its SP estimates. It’s worth noting, therefore, that Rendon’s conventional direct method rules out negative estimates as a built-in feature. So the TRC’s indirect method opens the door to the negative numbers problem.
I feel like I’m beating a dead horse but there’s still an important remaining issue concerning the data to which I’ll turn next.
War, Numbers and Human Losses 