A Zombie Graph Lives on Courtesy of the Lancet

I’ve got a new article just out in The Conversation.  Here’s the short version:

  1. The Lancet publishes a false graph
  2. The problems of the graph are exposed, several even in letters to the Lancet.
  3. The Lancet just leaves the graph up.
  4. A Washington Post reporter stumbles onto the false graph, thinks it’s cool and reprints it.
  5. I tell the reporter that  he just published a false graph
  6. The reporter does a mea culpa and pulls the graph down
  7. I write up this sequence of events for The Conversation
  8. The Conversation sends it to the Lancet
  9. The Lancet declines to comment and leaves the false graph up
  10. The Conversation publishes the piece
  11. Someone else sees and believes in the graph?

Spewing Rancid Effluvia at Iraq Body Count – Part 1

This post follows up on this one. However, rather than calling it “A Debate about Excess Deaths – Part 2” I went with the above title which is  more descriptive of what’s actually going on here..

In fact, it’s bizarre that the Iraq Body Count (IBC)  database has been  dragooned into a debate about excess deaths.  IBC exclusively records violent deaths.  The concept of excess deaths, on the other hand, was created to account for the possibility that war violence can lead, indirectly, to non-violent deaths.  So the IBC database is not going to be particularly relevant to a debate about excess deaths.

To understand why we’re here you have to recall the following sequence of events.

  1. Hagopian et al. publish a paper claiming 1/2 a million excess deaths in Iraq.
  2.  Stijn van Weezel and I publish a critique saying that this number is greatly exaggerated.
  3.  Hagopian et al. publish a comeback claiming they are right and we are wrong.  (NEWS FLASH – their critique is actually published.  I wasn’t aware of this when I wrote my previous blog post.)
  4.  Stijn and I will publish a rejoinder.  (We’ve already signed off on page proofs but the paper isn’t out yet.)

I will blog our rejoinder (event 4) when it appears.  Now I just want to address some points that, due to space constraints, Stijn and I were forced to ommit from our paper.

One of the main arguments Hagopian et al. use to defend their excess death estimate is the very model of a modern ad hominem attack.  I am a co-author on the critique paper but I am discredited because I have worked with IBC which itself is discredited (the claim) – therefore, the excess death statistics of Hagopian et al. are correct.  With a bit more research Hagopian et al. might have bolstered their logic by pointing out that I support Crystal Palace in  Premiership Football but the Pride of South London is now teetering on the brink of relegation – thus, they are right and I am wrong about Iraq.

For the excess deaths debate the above paragraph should be enough.  However, Hagopian et al. sling so much rancid effluvia at IBC that I feel I have to correct the record.

This post is a start.

Hagopian et al. write:

Spagat has published extensively using the data of Iraq Body Count, a passive media-based measure of 2003 Iraq war mortality…This method has been discredited, however, as it understates mortality (Ahmed, 2015; Burkle & Garfield, 2013; Carpenter et al. 2013; Siegler et al., 2008)  As evidence, an important finding in our work is that small arms fire contributed substantially to mortality (63%); these events rarely make the sort of headlines tracked by the Iraq Body Count.

It’s hard to find any true statement or respectable citation in the above excerpt.  But you have to start somewhere so I’m going to go with the very end.

Notice, first of all, the weasely wording – there are five co-authors but none of them have bothered to learn what the percentage of deaths attributed to gunfire in the IBC database actually is.  They just venture, incorrectly, that IBC only tracks headlines, and that gunfire events rarely make it into these.

How do we quantify “rare”?  Maybe 10%?  That seems way too high for “rare”.  Maybe 1% or 0.1%?  I’m not sure.

In reality, IBC assigns gunfire to 54% of the deaths in its database during the period covered by the Hagopian et al. survey (March 2003 through June 2011).  And this number understates the full IBC percentage because IBC has a separate category of “executions” which are overwhelmingly gun deaths, although I can’t quickly separate gun executions from non-gun executions.

On top of that the Hagopian et al. survey (known as the UCIMS) has two separate modules; one is household based and the other is sibling based.  (In the former people are asked about deaths within their households and in the latter people are asked about deaths of siblings.)  These two modules lead to separate estimates based on different techniques.  And what is the sibling-based UCIMS estimate for the percentage of gunfire deaths?  Errr….54%, same as IBC.

So Hagopian et al. serve up the gunfire percentage as a prime defect of the IBC database when, in fact, IBC and the UCIMS are very much compatible on this metric.  Indeed, the preponderance of gun deaths has been a prime talking point for IBC since shortly after the invasion phase of the war (when air strikes predominated). So the Hapopian et al. insight is an old one.

You’d think that Hagopian et al. would be pleased by confirmation from IBC and would be happy to cite this agreement.  Instead, sadly, they manufacture a falsehood about IBC – that it rarely records gun deaths when the truth is that most deaths  in the IBC database are gun deaths.  They then swipe at IBC from atop their fictitious creation..

And this point about gun deaths is just a tiny drop in the sea of slime Hagopian et al. sling at IBC.  I’ll return soon for more cleansing.

Secret Data Sunday – ABC News (in the US) Stonewalls over their Dubious Iraq Public Opinion Polls

Below is an email that I sent to Kerry Smith, the Senior Vice President for Editorial Quality at ABC news, back in November of 2016.

She did not reply..


Dear Ms. Smith,

I am a professor of economics specialized in the quantitative analysis of armed conflict.  I have a big body of work focused on data quality issues that arise during data collection in conflict zones, especially survey data.

Back in 2011 I wrote a paper with Steven Koczela, now a prominent pollster with MassINC Polling, that uncovered substantial evidence of fabricated data in polls fielded in Iraq by D3 Systems.  We sent our paper to various interested parties for comments, including Mathew Warshaw of D3 Systems and Gary Langer who had just moved from ABC to found Langer Associates.  We included Mr. Langer in the circulation list because ABC news had used D3 Systems for a series of polls in Iraq that now required urgent re-evaluation.

D3, backed by Langer Associates, responded by threatening to sue me and Mr. Koczela.  See this, this and this.   My university has supported me against this censorship attempt but, unfortunately, Mr. Koczela felt that he could not defend himself and signed an agreement to keep his mouth shut about this particular piece of work.  (This why only my name appears on the first link above.)  Eventually, the legal threat disappeared when I wrote to Mr. Warshaw asking him explain what, specifically, he objected to in our analysis.  He did not reply.

To his credit Mr. Koczela continued working on this issue, unearthing a large number of datasets for opinion polls conducted in Iraq by D3 Systems and other polling companies.  These have provided remarkably strong evidence of data fabrication already.  For example, see this eye-popping analysis.

Many of the D3 Iraq surveys that I now have were conducted for the US State Department.  Mr. Koczela made the State Department aware of the problem at some point and they hired Fritz Scheuren, a former president of the American Statistical Association to investigate.  His analysis confirmed the fabrication problem using an analysis rather different from mine.  Unfortunately, Dr. Scheuren signed a nondisclosure agreement but I believe he would confirm in general terms the main gist of this work and he could also give you an authoritative opinion on my analysis.  (scheuren@aol.com)

Notice that after the Huffington Post article Langer Associates did post a response to my 2011 paper.   This is, however, exceptionally weak as I explain in these articles.  Langer Associates have not addressed the new evidence that has emerged since Mr Koczela’s FOIA either.

I emailed Mr. Langer for the data from the ABC Iraq polls but he did not reply.  I asked Mr. Warshaw for the same data and he referred me to ABC news.  I am now requesting the data from you.

 At the risk of belabouring the obvious, I note that people with strong intellectual cases to make do not start by threatening to sue and finish by withholding their data.

Most importantly, ABC needs to take action to correct the historical record of the Iraq war.  These polling numbers are all over the web sites of ABC news and its partner organizations in these polls.  This work must be retracted.

It is, of course, your journalistic obligation to correct the historical record but, at the same time, I think it’s to your advantage to do so.  Fixing this problem would demonstrate a strong commitment to quality and accuracy.  I doubt you would even lose your Emmy Award.  Surely you won’t be punished for pursuing the truth wherever it leads.  I will do anything I can to help in this regard.

I suggest that we meet to discuss these issues further.  I would be happy to fly to New York at my own expense for this purpose.  Alternatively, we could talk by phone, skype or some other technology.



Professor Michael Spagat

Head of Department

Department of Economics

Royal Holloway College

University of London

Egham, Surrey TW20 0EX

United Kingdom


+44 1784 414001 (W)

+44 1784 439534 (F)


Blog:  https://mikespagat.wordpress.com/

War, Numbers and Human Losses: The Truth Counts

Department of Corrections: Mike Spagat – Edition 2

I’ve just gone through and corrected this post.

The present post is, therefore, a follow up to this earlier one which, weirdly, was about correcting the claim that Keynes had claimed that he always corrected himself when he’s wrong.  It seems that Keynes didn’t actually say this although he did correct himself when wrong.

Anyway, I can’t say that I was jumping up and down with joy when I read the comment from Ben Prytherch and realized that I might be wrong on figure 14a of a paper of Cirillo and Taleb.  But I can honestly say that I’m glad he sent it in because he gave me an instant chance to avoid a dead end and get back on track.

I don’t know when edition 3 will go up.  I’m counting on my readers to get me there as soon as I earn the privilege.

88462092 (1)




The Iraq Sanctions Myth Rears its Ugly Head at Scripps College

This unusually interesting article just appeared in the the Los Angeles Times (coming to me via Michael Shermer).

The obvious angle is the spectacle of protesters at one women’s college (Scripps) attempting to follow the  path of another (Smith) in disinviting a female commencement speaker (Madeleine Albright) who had shattered gender barriers in a male-dominated world.  That said, harsh treatment of commencement speakers seems to occupy a central role in American identity almost analogous to the English tradition of replacing regular train service with buses. So maybe the gender connection is just a coincidence here, not the driving force.

There is a lot to chew over in this letter from some Scripps faculty members.   Here I just want to highlight the issue of central concern to the blog – they fall hook, line and sinker for The Iraq Sanctions Myth.


In fact, the Scripps Scholars even nibble on the Tony Blair version of the myth which he used in front of the (apparently) credulous Chilcot Inquiry to justify the 2003 invasion of Iraq.

More Iraqis died as a result of 13 years of U.S.-led and controlled sanctions than as a result of the 2003 US invasion and subsequent occupation of Iraq.

The old evidence the Scripps Scholars use to support their sanctions claims would be like citing Colin Powell’s Presentation to the UN Security Council on Iraq’s WMD Program to prove that Iraq really had WMD.

All that said, there is room to criticize of Madeleine Albright over her interview with Lesley Stahl in which she did say that it was worth sacrificing half a million Iraqi children in pursuit of the sanctions policy.  Stahl’s question turned out to be based on a false premise that both Stahl and Albright accepted at the time and Albright’s answer was appalling.

There is a rough analogy here with the notorious Milgram experiments during which some participants were fooled into believing they were administering huge electrical shocks to other participants but kept pushing the button anyway amidst the screams of increasingly keyed up actors.  We know that nobody was actually shocked during these experiments but we might still criticize those who thought they were seriously harming other people but kept going.

Perhaps the Scripps Scholars will now correct the gaping error in their indictment of Madeleine Albright?.  This shouldn’t take too long.

Don't make him hold his breath too much longer.
Please correct your error before I run out of air.


Department of Corrections: Mike Spagat Edition

Hello old friends.  Once again I need to apologize for neglecting you but I  plan to atone for my sins by flooding you with interesting stuff over the next few weeks.

I’ve been reading the new book by Philip Tetlock and Dan Gardner on “Superforecasting”.  It is spectacularly good and I will definitely come back to it on the blog but for now I draw attention a a relatively minor point of the book.

Those of you who have clicked the “About” link on my web page know that it contains this:


Well guess what?  No one can find a solid record of Keynes actually saying this and it appears that, in fact, he didn’t say it.


….and, yes, I’m aware of the irony of using precisely this quote to define a blog which devotes a lot of effort to correcting errors.  So I must issue a correction and an abject apology.

That didn’t feel too bad, which is good because I’m sure it won’t be the last time I issue a correction on this blog.

Meanwhile, I pledge to live by the sentiment of the apocryphal quote.

Columbia Journalism Review does at least have an error correction mechanism

In this post and this post I criticized Columbia Journalism Review(CJR) for writing about Iraq war-death numbers without investigating the methodologies for the production of the numbers and for suppressing the uncertainty that surrounds the numbers.

Yet I need to credit CJR for one thing – they do investigate errors and make some corrections.  My personal experience with both academic journals and journalistic publications suggests that such willingness to correct errors is not typical at all.

The first error is to cite this paper as claiming something that it doesn’t actually say.  Indeed, the CJR claim doesn’t survive first contact with the paper’s abstract.

Some readers maybe be shocked to learn that even peer-reviewed journal articles often make false claims about other published literature.  In future blog posts I will show you plenty of such false citations.

CJR cites a paper that compares two datasets and estimates their overlap.  One is Iraq Body Count, with is already familiar to readers of this blog.  The second is the  “sigacts” database for Iraq, which is the official data of the US military which was brought into the public domain by Wikileaks.

So there are two datasets recording deaths in the Iraq war.  The question is – to what extent are they recording the same deaths?  CJR cites the journal article as showing that IBC captures less than 1/4 of the sigacts deaths.  The article actually estimates that IBC captures 46% of the sigacts deaths.

…it is estimated that 2035 (46.3%) of the 4394 deaths reported in the Wikileaks War Logs had been previously reported in IBC.

(In fact, the cited article is terrible and the true percentage is much higher than 46% but I will come back to this issue in a future post.)

Now it gets more interesting.  Here is a peer-reviewed article that falls straight into the same falsehood as CJR does:

the emergence of the Wikileaks “Iraq War Logs” reports in October 2010 [69] prompted the Iraq Body Count team to add to its count, but a recent comparison of recorded incidents between the two databases revealed that the Iraq Body Count captured fewer than one in four of the Iraq War Logs deaths[70].

(Citation 70 is to the same article that CJR cites.)

Thus, 46% is represented as “fewer than one in four” in a peer-reviewed journal article.  Any reader who bothers to follow the citation can spot the falsehood instantly.

Welcome to the real world.

At least we can trust doctors…maybe….a little bit?  The recent Physicians for Social Responsibility report on war deaths avers:

Generally, however, the students found that only every sixth individual death in the Logs had a match in the IBC database

Even worse – 46% is portrayed as 1/6.  (The “Logs” refers to sigacts and, yes, the matching was done by students in a course project.)

I have some sympathy for CJR here.  They are operating inside a hall of mirrors in which multiple, seemingly respectable, sources are dead wrong.  And CJR did correct the error when it was brought to their attention.

In the end there is a happy message here – we are not doomed to drab lives of pinging back other peoples’ waste products if we follow one simple rule.  Do not cite a source without first reading that source.

The other CJR correction is, perhaps, less interesting.  As we have already discussed here the CJR article shrinks the IBC methodology to a shadow of its real self, portraying IBC as only tracking a few dozen newspapers and TV broadcasts and only recording deaths of named individuals.  CJR now admits that this was wrong although its correction still don’t recognize that sigacts integration is well under way.  So this is not the greatest correction in the world but it’s better than leaving the original alone.

Should we give CJR an award for standing up and admitting to their fallibility unlike many others who never admit error?

Not really.  Doing this would be like commending a husband for going ten years without beating his wife.  Correcting error and not beating your partner are things you’re supposed to do.  Still, it is fair to criticize those who don’t rise to even this low standard.