In this post and this post I criticized Columbia Journalism Review(CJR) for writing about Iraq war-death numbers without investigating the methodologies for the production of the numbers and for suppressing the uncertainty that surrounds the numbers.
Yet I need to credit CJR for one thing – they do investigate errors and make some corrections. My personal experience with both academic journals and journalistic publications suggests that such willingness to correct errors is not typical at all.
The first error is to cite this paper as claiming something that it doesn’t actually say. Indeed, the CJR claim doesn’t survive first contact with the paper’s abstract.
Some readers maybe be shocked to learn that even peer-reviewed journal articles often make false claims about other published literature. In future blog posts I will show you plenty of such false citations.
CJR cites a paper that compares two datasets and estimates their overlap. One is Iraq Body Count, with is already familiar to readers of this blog. The second is the “sigacts” database for Iraq, which is the official data of the US military which was brought into the public domain by Wikileaks.
So there are two datasets recording deaths in the Iraq war. The question is – to what extent are they recording the same deaths? CJR cites the journal article as showing that IBC captures less than 1/4 of the sigacts deaths. The article actually estimates that IBC captures 46% of the sigacts deaths.
…it is estimated that 2035 (46.3%) of the 4394 deaths reported in the Wikileaks War Logs had been previously reported in IBC.
(In fact, the cited article is terrible and the true percentage is much higher than 46% but I will come back to this issue in a future post.)
Now it gets more interesting. Here is a peer-reviewed article that falls straight into the same falsehood as CJR does:
the emergence of the Wikileaks “Iraq War Logs” reports in October 2010  prompted the Iraq Body Count team to add to its count, but a recent comparison of recorded incidents between the two databases revealed that the Iraq Body Count captured fewer than one in four of the Iraq War Logs deaths.
(Citation 70 is to the same article that CJR cites.)
Thus, 46% is represented as “fewer than one in four” in a peer-reviewed journal article. Any reader who bothers to follow the citation can spot the falsehood instantly.
Welcome to the real world.
At least we can trust doctors…maybe….a little bit? The recent Physicians for Social Responsibility report on war deaths avers:
Generally, however, the students found that only every sixth individual death in the Logs had a match in the IBC database
Even worse – 46% is portrayed as 1/6. (The “Logs” refers to sigacts and, yes, the matching was done by students in a course project.)
I have some sympathy for CJR here. They are operating inside a hall of mirrors in which multiple, seemingly respectable, sources are dead wrong. And CJR did correct the error when it was brought to their attention.
In the end there is a happy message here – we are not doomed to drab lives of pinging back other peoples’ waste products if we follow one simple rule. Do not cite a source without first reading that source.
The other CJR correction is, perhaps, less interesting. As we have already discussed here the CJR article shrinks the IBC methodology to a shadow of its real self, portraying IBC as only tracking a few dozen newspapers and TV broadcasts and only recording deaths of named individuals. CJR now admits that this was wrong although its correction still don’t recognize that sigacts integration is well under way. So this is not the greatest correction in the world but it’s better than leaving the original alone.
Should we give CJR an award for standing up and admitting to their fallibility unlike many others who never admit error?
Not really. Doing this would be like commending a husband for going ten years without beating his wife. Correcting error and not beating your partner are things you’re supposed to do. Still, it is fair to criticize those who don’t rise to even this low standard.