I’ve opened several recent posts with something like “Here’s a nice article but it comes up short in a particular way that I’d like to discuss.” Please take such formulations at face value and don’t imagine that I’m stretching to find something polite to say. I’ll tell you directly when I think I see a terrible article.
This article is terrible.
All I can think of to soften the blow is that I like Rob Muggah and am sure he’s trying to be helpful. But he should have done his homework before tossing this one out there.
The very first sentence explodes instantly at the click of a mouse.
At least 240,000 Syrians have died violently since the civil war flared up four years ago, according to the Violations Documentation Center in Syria.
Click on the link and you see 122,683 deaths, not “at least 240,000.” (This number will rise over time so those who click after my post goes up should see a higher number.)
Go to the Syrian Observatory for Human Rights if you want to figure out how Muggah got confused. Here I just note in passing that the Violations Documentation Center is far more transparent in its work than is the Syrian Observatory for Human Rights. This follows a common pattern that higher numbers seem to be associated with weaker documentation.
Predictably, this “click and collapse” opening is not a one-off:
In the case of DRC, the International Rescue Committee (IRC) commissioned a series of mortality surveys between 2000 and 2007…. The researchers initially claimed that 5.4 million Congolese died as a result of the armed conflict from 1998 to 2006. Under pressure from critics, the estimate was later revised downward to around 2.8 million deaths, with roughly 10% attributed to violence.
I was, initially, shocked to learn that the IRC had conceded so much ground to its critics rather than remaining in deep denial as I thought it had. I experienced a warm feeling of the possibilities for rational discussion mixed with self-reproach for having, somehow, missed this important event.
But click on “revised downward” and you see that the IRC’s bow to its critics is an artifact of a surprising reading error; Muggah mistakes a quote from the Human Security Report’s critique of the IRC work as a quote from the IRC itself. (This kind of feels like a schoolboy fantasy….I call you a pooh pooh head and this means you concede that you are, indeed, a pooh pooh head.)
And this is only half of the error.
The Human Security Report’s position boils down to two main points.
- The IRC strung together a series of surveys to underpin its 5.4 million estimate. The first two waves only covered small slivers of the Democratic Republic of Congo (DRC) but the IRC estimated, inappropriately, as if these places were nationally representative. However, just as we should not treat political opinion in Massachusetts as representative for Texas we should not treat death rates where the IRC provides humanitarian relief as representative of death rates for the whole of the DRC. The Human Security Report pointed out that if you throw out these two waves of nonrepresentative surveys, but still accept the rest of the IRC’s work, then the overall estimate falls from 5.4 million deaths to 2.8 million deaths. (This is the sense in which the IRC voluntarily “revised downward” to 2.8 million.)
- The Human Security Report goes on to argue that the baseline death rate the IRC used to calculate excess deaths was too low. (Go here if you’ve forgotten about excess deaths and baselines.) If you take a more realistic baseline then the IRC estimate falls from 2.8 million to 900,000.
The excess death estimates for the final three surveys, the only ones to cover the entire country, were not affected by the methodological errors evident in the first two surveys. Here, the major problem, as mentioned above, lay with the inappropriately low baseline mortality rate. The impact of changing this rate to a more appropriate one was dramatic. The estimated excess death toll dropped from 2.8 million to less than 900,000. This is still a huge toll, but it is less than one-third of the IRC’s original estimate for the period. [Human Security Report]
It seems to me that this and other distortions come from extreme impatience rather than from Muggah having an ax to grind. It is as if he conducted his research while driving a car full of screaming children and talking on the telephone.
Here is a particularly irksome paragraph for me (I work closely with Iraq Body Count and am on the Board for Every Casualty):
…Advocacy groups such as Every Casualty and the Iraq Body Count (IBC) restrict their databases to incidents where there is evidence of a corpse and the circumstances of the individual’s death. Each victim must have a name. And because they limit their counts to precisely documented cases of violent deaths, they are often criticised for under-counting the scale of lethal violence.
First of all, Every Casualty doesn’t have a database, let alone one that imposes Muggah’s created rules. It does strive to bring about a world in which nobody dies in a war without his/her death being recorded and recognized. Ideally, recognition includes names but no one I’m aware of thinks that war death databases should exclude unnamed victims. IBC would love to have a name for every victim in its database but, unfortunately, most remain unnamed to this day. In short, Muggah’s requirement that “each victim must have a name” is a mirage.
Muggah’s “often criticised” link in the above quote is, perhaps, a borderline case but I would also classify it as a click and collapse. It is true that databases that document deaths on a case by case basis will tend to undercount because it is very difficult to capture every single case (although it appears that Kosovo Memory Book pulled off this feat after an extraordinary effort made over many years.). Yet, Muggah spends a good chunk of his article discussing competing figures on the number of deaths in Iraq and it turns out that the article behind this link (by Beth Osborne Daponte) covers the same Iraq numbers in great depth and concludes:
While each approach has its drawbacks and advantages, this author puts the most credence in the work that the Iraq Body Count has done for a lower bound estimate of the mortality impact of the war on civilians. The data base created by IBC seems exceptional in its transparency and timeliness. Creating such a data base carefully is an incredibly time-consuming exercise. The transparency of IBC’s work allows one to see whether incidents of mortality have been included. The constant updating of the data base allows one to have current figures.
Muggah’s short summary of this article is that is criticizes IBC and similar efforts for undercounting.
A few weeks back I introduced readers of the blog to the three fundamental ways to account for war deaths: documentation, counting and estimation. Quoting from an earlier post, you can:
1. Document war deaths one by one, listing key information such as names of victims, dates of deaths, how people were killed etc.
2. Count the number of war deaths. Doing this is an obvious outgrowth of documentation but also has status independent of documentation. You can, for example, count deaths that may be documented to varying standards, e.g., counting unidentified bodies along side identified ones..
3. Estimate the number of war deaths using a statistical procedure.
Muggah refers rather indiscriminately to “counts” throughout his article, even appearing to extend the concept to estimates of excess deaths. This juxtaposition of counting and excess deaths suggests an illuminating thought experiment.
Imagine trying to count excess deaths deaths in a war. Assume you are lucky enough to have a list of everyone who has died since the outbreak of the war. You would have to consider every person on the list and make a call on whether or not that person would have survived if the war had never happened in the first place.
What do you do, for example, with people who die of strokes? There could be a few cases of people dying on the operating table after power outages that can be clearly traced back to insurgent activity. You would have a good case for classifying these as excess deaths, albeit at the cost of engaging in piss-poor monocausal social science since war-induced power outages can only be one among multiple causes of stroke deaths.
Suppose now that the rate of stroke deaths increased dramatically after the outbreak of war but that it’s only possible to identify a war-related cause for a handful of stroke deaths. The excess death concept suggests that, nevertheless, you should view the increase in the death rate as caused by the war. (This attribution of causality is debatable but that’s not my concern here.) My point is simply that you can estimate the number of stroke deaths above and beyond a baseline pre-war rate but you can’t count them one by one because any individual case can’t be pinned monocausally on the war.
Thus, excess deaths are tied inextricably with estimation whereas violent deaths can be documented, counted or estimated. In fact, it is no accident that we can approach violent deaths from more angles than we can approach excess deaths. The excess deaths concept is rooted in a counterfactual scenario – how many people would have died if there had never been a war? – and is, therefore, more abstract than the violent deaths concept.
Muggah is right that there is a lot of disagreement about measuring war deaths. Yet there is a hopeful sign in the way that several dozen casualty recording organizations are now at the final stages of agreeing on a shared set of standards after a series of fruitful meetings organized by Every Casualty. I hope to be able to post something soon on this positive development.