This is the latest installment in a series that considers the statistical report done for the Peruvian Truth and Reconciliation Commission (TRC), Silvio Rendon’s critique of this statistical report and a reply to Rendon from Daniel Manrique Vallier and Patrick Ball (MVB) who worked on the TRC statistical report. The present post continues to discuss the MVB reply.
(Note that I may not resume this series until Silvio Rendon’s rejoinder is published. Meanwhile, I’m also working with Josh Dougherty of Iraq Body Count on an offshoot post that will cover the practice and pitfalls of matching deaths across multiple lists.)
Today I’ll comment on nine figures from the MVB reply: figure 1 in the main body of the paper and figures 2-9 in the appendix.
I won’t produce any of the figures here because they are misleading and a picture is worth a thousand words. The main features I object to are that the figures substitute lower (preliminary) stratum-level estimates for Rendon’s main estimates and suppress the uncertainly surrounding these estimates. Moreover, MVB portray some of these these lowered point estimates as falling within an “impossibility region,” a characterization which further assumes that MVB’s matching of deaths across sources was perfectly executed on fully accurate data.
Nevertheless, the figures do convey some interesting simulation-based information that addresses the question of when a direct estimation approach outperforms MVB’s indirect one and vice versa. Each of the nine figures uses data from a stratum for which one can directly estimate Shining Path (SP) deaths. (There are nine such strata before multiple imputation and two more, not covered by the figures, after multiple imputation.)
The X axis in each picture represents all the possible true values for the number of SP-caused deaths (with the true values indexed by N). MVB perform simulations that estimate the number of SP-caused deaths many times for each stratum and for each N using both direct capture-recapture and MVB’s indirect capture-recapture methodology. MVB then calculate the deviation of each estimate from the underlying true value, square these deviations (so that negative deviations do not cancel out positive ones) and take the mean of these squared deviations across all simulation runs for each value of N. Finally, they graph these “mean-squared errors” for each method and each N in all nine strata.
For eight out of the nine strata the direct method outperforms (i.e., has lower mean-squared errors) the indirect method for values of N below some critical value and the the indirect method outperforms the direct one above this same critical value. (For one stratum the reverse is true but there is never a big difference between the two methods in this stratum so this doesn’t seem to matter much.) For three strata the critical value for which the best performing method switches from direct to indirect is inside of MVB’s “impossibility region”.
In eight out of the nine strata the indirect method outperforms the direct method when the true number of people killed by the SP is set equal to the estimate that the TRC actually made for that stratum (using the indirect method). Essentially, this rather unsurprising result says that the indirect method performs well in simulations of cases for which the TRC’s indirect estimate delivered a correct result. And the indirect method also performs well when the TRC’s estimate is not spot on but still reasonably close to being correct.
The direct method tends to outperform the indirect one in simulations that start from the assumption that the direct estimate is correct. Nevertheless, in three out of the nine strata the indirect method actually wins this contest.
Overall, these simulation results tend to favor the indirect method over the direct one, especially when the true numbers are assumed to be rather high.
That said, the direct method in the simulations does not match Rendon’s main method because, again, MVB omit the multiple imputation step of Rendon’s procedures. Incorporating multiple imputation should shift the balance back towards Rendon. And, again, I would like to see a similar exercise performed on Rendon’s alternative approach that covers the whole country with ten strata.
Here’s one last point before I sign off. As of now, the MVB reply is still just a working paper, not yet published in Research and Politics. The main advantage of posting a working paper before publication is that you can respond to feedback. Thus, it would be great and appropriate for MVB to take advantage of the remaining time window by purging the misleading material about impossible point estimates without uncertainty intervals from the published version of their paper. (See post 4 and post 5 of this series in addition to the present one for further details.) This move would help lead us toward more fruitful future discussions.