I’ll start this post by reacting to some interesting comments to part 4 of this series which was, you may be surprised to learn, preceded by part 1, part 2 and part 3. I’ll assume that readers have some familiarity with these posts but I’ll also try to go slowly and remind readers of things we’ve discussed before.
Recall that there is a statistical report done for the Peruvian Truth and Reconciliation Commission (TRC), Silvio Rendon’s critique of this statistical report and a reply to Rendon from Daniel Manrique Vallier and Patrick Ball (MVB) who worked on the TRC statistical report.
Let’s focus first on the data.
The TRC statistical report and Rendon’s critique are both based on what I’ll call “the original data,” which consists of 3 (after some consolidation) lists containing a total of 25,000 unique (it is claimed) deaths, many appearing on multiple lists.
There are several issues concerning the original data:
First, only summary information from the original data is in the public domain. The following table from the TRC statistical report shows the form of the publicly available original data:
We can see, for example, that there are 627 people recorded nationwide as killed by the State (“EST”) who appear on the list of the CVR (the TRC itself) and the DP (Defensoria del Pueblo) but not on the ODH (NGO’s) list.
There are 59 such tables, one for each geographical stratum. Each one looks like the above table but, of course, with smaller numbers. Both the TRC and Rendon base their statistical work on these tables.
The problem is that these 59 tables alone do not allow us to examine the underlying matching of deaths across lists that they summarize. Matching is a non-trivial step in the research that involves a lot of judgment. I will examine the matching issue in an upcoming post. Suffice to say here that until this step is opened up we are not doing open science.
To be fair, it appears possible for at least some researchers to obtain the detailed data from the Peruvian government and perform their own matching. According to Patrick Ball:
“People with access to the detailed TRC data” is not an intrinsic category of people: it’s just people who have asked nicely and persisted (sometimes, like us, over several years), until they got access to the data. It seems to me that with sensitive data, obtaining the relevant information is incumbent upon the researcher: Rendon could have inquired of the Peruvian Ombudsman office to get the TRC data. It’s not secret, it just requires a bit of work to obtain, and he chose not to do so.
I don’t like this. The data should simply be available for use. Patrick may be right that, effectively, it’s open to all nice and persistent people. But the data should also be available to mean and non-persistent people as well.
Let’s move on to a few observations on the MIMDES data, the detailed version of which is in the public domain. (Apparently it’s not online right now but has been in the past and hard copies can be obtained.).
First, the public availability of the MIMDES data undermines excuses for forcing researchers to jump through hoops for the TRC data. They are both detailed lists of people killed in the war. These list are both held by the Peruvian government. Why is it OK to circulate one list while requiring researchers to be nice and persistent for the other?
Second, I know nothing about the data collection methodology for the MIMDES data. OK, perhaps I should obtain and study the MIMDES reports. But the MVB reply paper introduces the MIMDES data into this whole discussion so they should describe the MIMDES data collection methodology in their paper. (They also should have described the data collection methodologies for the lists used in the TRC’s statistical report.) But the MIMDES methodology seems particularly important since Patrick Ball, in his comments on this blog, urges us to treat the MIMDES sample as more or less representative of all deaths in the war. I would need to know something about how MIMDES performed its work before entertaining such a notion.
MVB have matched the MIMDES deaths against the TRC’s deaths and the resulting figures are central to their reply to Rendon’s critique. For three reasons, however, I recommend that we take these merged TRC-MIMDES figures with a grain of salt, at least for now. First, MVB don’t explain how they do the matching. Second, they say their work is unfinished. Third, it is difficult at present for anyone to match independently since the TRC data are not really open. (Remember that you need the detailed TRC data in order to match it against the MIMDES data.)
That said, for the rest of the post I’ll take the numbers from MVB’s TRC-MIMDES merge as given just as I’ve done in my earlier posts.
Patrick and Daniel especially emphasize one point in their separate comments on post number 4 (in which they focus exclusively on Rendon’s Shining Path (SP) estimates). Recall that Rendon’s main estimate starts with estimates from just the geographical strata that allow for direct estimation (after multiple imputation) and then uses spatial extrapolation (kriging) to extend these estimates to the whole of Peru. But, MVD argue, the estimates in the selected strata are biased downwards because the fact that there’s rich enough data to do direct estimation already suggests that there are relatively few undocumented deaths left to discover in these strata. Conversely, MVB suggest, the strata where data is too sparse for direct estimation probably contain relatively many undocumented deaths.
This is a creative idea with some potential but I think that, if it exists, its effect is probably small. One of Rendon’s alternative estimates cuts Peru up into just 10 regional strata which cover the entire country rather than the 59 more localized strata in MVB’s stratification scheme. This 10-stratum estimate is not subject to MVB’s selection bias argument. The SP estimate in this case is around 1,000 deaths more than Rendon’s main estimate (which requires strata selection and spatial extrapolation). So perhaps MVB have identified a real bias although, if so, it seems to be a small one. There are, of course, multiple changes when we move from Rendon’s main estimate to the 10-stratum one. But MVB need their suggested bias to be huge in order to produce the 10,000 plus additional deaths required to make their TRC estimate look accurate. The above comparison doesn’t suggest a bias effect of this order of magnitude.
The frailty of MVB’s statum-bias critique is exposed by the games they play to portray Rendon’s direct stratum estimates for the SP as systematically lower than the SP numbers in their merged TRC-MIMDES dataset.
They begin by deleting the multiple imputation step of Rendon’s procedures. Daniel Marique Vallier explains:
Thus, showing that the application of capture-recapture to those strata leads to contradictions, automatically invalidates all the rest of the analysis. This is because those more complex estimates depend on (and amplify) whatever happens with the original 9; you can’t extrapolate from strata that are themselves inadequate. That’s what we have shown. Specifically, we have shown that the application of capture-recapture to those 9 strata (Rendon’s necessary condition) results in estimates smaller than observed counts (contradiction). This means that the basic premise, that you can use those strata as the basis for a full blown estimation, is faulty (modus tollens). Anything that depends on this, i.e. all the rest of the conclusions, is thus similarly faulty (contradiction principle).
This makes no sense. Applying multiple imputation before capture recapture increases the estimate in every stratum. These higher estimates then feed through the spatial extrapolation to increase the national estimates. Deleting the multiple imputation step decreases the estimates at both the stratum and national levels. Manrique Vallier argues, in effect, that doing something to increase Rendon’s estimates can only compound the problem of his estimates being too low. This is like saying that drinking a lot of alcohol makes it dangerous to drive so (modus tollens) sobering up can only make it more dangerous to drive.
Next MVB try to dismiss all of Rendon’s work based on their claim that some of Rendon’s point estimates (which they have lowered) are below their merged TRC-MIMDES numbers. Simultaneously, MVB apply a far more lenient standard to evaluate Patrick Ball’s Kosovo work (see MVB’s comments on post 4). For Kosovo, they argue, it’s not a problem for most estimates to be below documented numbers as long as the tops of Ball’s uncertainty intervals exceed these numbers. Moreover, it’s even OK for this criterion to fail sometimes as long as Ball’s broad patterns are correct. I actually agree with these standards but consistency requires applying them to Rendon’s Peru work as well.
That said, Patrick Ball’s last comment makes a good point about the serious challenges to data collection in Peru. By contrast, it’s easier to collect war-death data in Kosovo and lots of resources were devoted to doing just this.. So the true numbers in Peru might be substantially larger than MVB’s TRC-MIMDES ones. I agree that this is possible but I would want to know a lot more about the various data collection projects in Peru before taking a strong stand on this point.
Finally, I return to Rendon’s ten strata estimate which appears immune to all the criticism contained in MVB’s reply. The central estimate is about 2,000 deaths above MVB’s TRC-MIMDES national count for SP deaths, leaving considerable room to accommodate the discovery of more deaths, especially in light of the uncertainty interval. That said, it would be interesting to see stratum by stratum comparisons with TRC-MIMDES to see whether there are any substantial discrepancies.
To summarize, perhaps Rendon’s SP estimates are somewhat low. But MVB’s reply does little to undermine Rendon’s critique.beyond this minor observation.