L’Affaire Burnham: Ten Years Later

I just gave this presentation at the annual meeting of the American Association for Public Opinion Research (AAPOR) in Toronto.

The title of the talk is taken from Richard Kulka’s Presidential Address to AAPOR.  Back in 2009 AAPOR censured Gilbert Burnham of Johns Hopkins University for refusing to disclose basic information about his methodology for (over)estimating violent deaths in the Iraq war.

This action led to a massive discussion on the AAPOR listserve which Kulka analyzed in his Presidential Address.  Read it.  It’s great.

My presentation tells the story about what happened next.  In short, AAPOR moved from the cutting edge of open science to the middle of the pack.  AAPOR’s Transparency Initiative favors openness of everything except data.  Meanwhile, AAPOR grandees who conceal their data get big awards.

That said, I return from the conference with a sense that the worm is turning on AAPOR and open data.  We’ll see.

Please have a look at the slides and let me know what you think.

Addendum – I just remembered this post in which I describe how an epidemiologist friend of Gilbert Burnham’s tried to dissuade me from ever mentioning the AAPOR censure of Burnham.   It’s another example of how researchers prioritize getting along with powerful people above getting to the truth.



Fake Data and the War in Iraq

Here are the slides from a talk I gave yesterday at RHUL’s PIR department.

It pulls together many threads that I’ve developed on the blog but I won’t give the links here since they are embedded in the document.

The Hurricane Maria Death Toll Numbers: An Extension of my Piece in The Conversation

I just published a piece on the Hurricane Maria death toll numbers.  

Space is always limited with such pieces so I’ll extend it a little bit here.

Excess deaths estimates tend to have really wide uncertainty intervals and the Harvard study on the Hurricane Maria death toll is no exception.  The reason for such extreme uncertainty is obvious, once you think about it, but very few people seem to have thought about it.

Suppose we estimate that country X suffered 10,000 deaths last year and we put a 95% uncertainty interval of 9,000 to 11,000 around that estimate.  That’s a reasonably tight estimate: plus or minus 10%.

Suppose now that in a normal year country X suffers 8,000 deaths.  However, in the year we do our estimate there was a war, hurricane, tidal wave or something else that seems to elevate the death rate.  The purpose of our estimate is to quantify this elevation.

A standard excess deaths estimate is 2,000.  We obtain this simply by subtracting 8,000, the normal or baseline death rate, from 10,000, the central estimate of the rate in the special year.

If we treat the normal (baseline) rate as a certainty then it is also easy to  place a 95% uncertainty interval around our excess death estimate.  This uncertainty interval runs from 1,000 (9,000 – 8,000) to 3,000 (11,000 – 8,000).  But this is an interval of plus or minus 50%.around our central estimate of 2,000.

The point is that when you subtract off a baseline then you magnify the size of the swing when you measure this swing in percentage terms around your excess death estimate.  

This means that what might be a pretty large sample size for determining the total number of deaths is actually a pretty small sample size for determining excess deaths.

Seen through this lens it’s clear that the Harvard study has a tiny sample size.  So it is no surprise that they published a preposterously wide uncertainty interval of about plus or minus 87%.

The moral of the story is that excess death surveys need very large sample sizes compared surveys aimed just at measuring total deaths.


A Debate about Excess War Deaths: Part II

My rejoinder (with Stijn van Weezel) to Hagopian et al is out. Hooray!   See this earlier post for background.

Please have a read.  It’s short and sweet.  Here’s the abstract:

Spagat and van Weezel have re-analysed the data of the University Collaborative Iraq Mortality Study (UCIMS) and found fatal weaknesses in the headline-grabbing estimate of 500,000 excess deaths presented, in 2013, by Hagopian et al. The authors of that 2013 paper now defend their estimate and this is our rejoinder to their reply which, it is contended here, avoids the central points, addresses only secondary issues and makes ad hominem attacks. We use our narrow space constraint to refute some of the reply’s secondary points and indicate a few areas of agreement.

And here’s the first paragraph:

Hagopian et al. (2018), the reply paper to Spagat and van Weezel (2017) which is, in turn, our critique of Hagopian et al. (2013), does not address either of our two central points. These are as follows (Spagat and van Weezel, 2017). First, any appropriate 95% uncertainty interval (UI) for non-violent excess deaths is at least 500,000 deaths wide and starts many tens of thousands of deaths below zero. Second, we find no local spill over effects running from violence levels to elevated non-violent death rates.1 Both these results refute the ‘conservative’ estimate of several hundred thousand non-violent excess deaths given in Hagopian et al. (2013). The fact that Hagopian et al. (2018) ignore these two points suggests that the authors of that paper are unable to respond.

In other words, Hagopian et al. can’t address our main points so they search for errors they might be able to catch us out on.

And they do actually find an error.  However, as we argue in the rejoinder, it doesn’t lead anywhere.

Specifically, we assumed, wrongly, that they drew a stratified sample.  (In fact, we didn’t digest a separate paper they wrote explaining their sampling scheme in detail.)  What this means in practice is that the number of clusters per governorate in their sample is out of line with population proportions.  For example, governorate A might have twice the population of governorate B but four times the number of clusters.  But this is a random outcome rather than being by design (our mistake).

We actually spilled a fair amount of ink in our critique discussing the importance of incorporating a stratification adjustment into the estimation.  However, we also did all our estimates both with and without such an adjustment.  And it turns out that even without a stratification adjustment it’s still true that:

any appropriate 95% uncertainty interval (UI) for non-violent excess deaths is at least 500,000 deaths wide and starts many tens of thousands of deaths below zero.

Making the adjustment widens the UI’s further but this point doesn’t matter materially.

Moreover, we still think it’s a good idea to do ex post stratification.  The Hagopian et al. sample is small and the realized numbers of clusters per governorate are pretty far out of wack with population proportions.  These imbalances would get ironed out in a large sample but this didn’t happen in the actual small sample.  We think it’s best to adjust for this imbalance.

For me, the highlight of the Hagopian et al. response is the section on death certificates which shows a strong desire by this team to have their cake and eat it too.  When households reported deaths to interviewers the interviewers then asked to see death certificates.  Hagopian et al. report that interviewees were usually able to show these certificates.  However, sometimes interviewees said that they didn’t have death certificates and sometimes interviewees said that they have them but were unable to produce one when prompted.  Nevertheless, Hagopian et al. just go ahead and assume that every single reported death is 100% certain to have happened regardless of death certificate status.  So they want to use death certificate checks in general terms to demonstrate the high quality of their data but when the outcome of a particular death certificate check casts a shadow on a particular datum they ignore this outcome.

Consider the following analogy.  I run a bar.  I ask everyone ordering an alcoholic drink if they have an ID showing they are 21 years old.  If they say they do have one then I ask to see it.  Most people just show an ID.  But some people say they have an ID, although they are unable to produce one when prompted.  Other people say they don’t have an ID.  I serve alcoholic drinks to all three types of people.  The police then investigate me to determine whether or not I’m selling to underage drinkers.  I tell them that I am certain that I never ever do this.  The reason I’m so certain is that I always ask my customers for ID’s and most of the people I serve drinks to actually show me one.

Somehow I don’t think the police would be convinced by this logic.

OK, those are the highlights – time now to read the whole thing!

Spewing Rancid Effluvia at Iraq Body Count – Part 1

This post follows up on this one. However, rather than calling it “A Debate about Excess Deaths – Part 2” I went with the above title which is  more descriptive of what’s actually going on here..

In fact, it’s bizarre that the Iraq Body Count (IBC)  database has been  dragooned into a debate about excess deaths.  IBC exclusively records violent deaths.  The concept of excess deaths, on the other hand, was created to account for the possibility that war violence can lead, indirectly, to non-violent deaths.  So the IBC database is not going to be particularly relevant to a debate about excess deaths.

To understand why we’re here you have to recall the following sequence of events.

  1. Hagopian et al. publish a paper claiming 1/2 a million excess deaths in Iraq.
  2.  Stijn van Weezel and I publish a critique saying that this number is greatly exaggerated.
  3.  Hagopian et al. publish a comeback claiming they are right and we are wrong.  (NEWS FLASH – their critique is actually published.  I wasn’t aware of this when I wrote my previous blog post.)
  4.  Stijn and I will publish a rejoinder.  (We’ve already signed off on page proofs but the paper isn’t out yet.)

I will blog our rejoinder (event 4) when it appears.  Now I just want to address some points that, due to space constraints, Stijn and I were forced to ommit from our paper.

One of the main arguments Hagopian et al. use to defend their excess death estimate is the very model of a modern ad hominem attack.  I am a co-author on the critique paper but I am discredited because I have worked with IBC which itself is discredited (the claim) – therefore, the excess death statistics of Hagopian et al. are correct.  With a bit more research Hagopian et al. might have bolstered their logic by pointing out that I support Crystal Palace in  Premiership Football but the Pride of South London is now teetering on the brink of relegation – thus, they are right and I am wrong about Iraq.

For the excess deaths debate the above paragraph should be enough.  However, Hagopian et al. sling so much rancid effluvia at IBC that I feel I have to correct the record.

This post is a start.

Hagopian et al. write:

Spagat has published extensively using the data of Iraq Body Count, a passive media-based measure of 2003 Iraq war mortality…This method has been discredited, however, as it understates mortality (Ahmed, 2015; Burkle & Garfield, 2013; Carpenter et al. 2013; Siegler et al., 2008)  As evidence, an important finding in our work is that small arms fire contributed substantially to mortality (63%); these events rarely make the sort of headlines tracked by the Iraq Body Count.

It’s hard to find any true statement or respectable citation in the above excerpt.  But you have to start somewhere so I’m going to go with the very end.

Notice, first of all, the weasely wording – there are five co-authors but none of them have bothered to learn what the percentage of deaths attributed to gunfire in the IBC database actually is.  They just venture, incorrectly, that IBC only tracks headlines, and that gunfire events rarely make it into these.

How do we quantify “rare”?  Maybe 10%?  That seems way too high for “rare”.  Maybe 1% or 0.1%?  I’m not sure.

In reality, IBC assigns gunfire to 54% of the deaths in its database during the period covered by the Hagopian et al. survey (March 2003 through June 2011).  And this number understates the full IBC percentage because IBC has a separate category of “executions” which are overwhelmingly gun deaths, although I can’t quickly separate gun executions from non-gun executions.

On top of that the Hagopian et al. survey (known as the UCIMS) has two separate modules; one is household based and the other is sibling based.  (In the former people are asked about deaths within their households and in the latter people are asked about deaths of siblings.)  These two modules lead to separate estimates based on different techniques.  And what is the sibling-based UCIMS estimate for the percentage of gunfire deaths?  Errr….54%, same as IBC.

So Hagopian et al. serve up the gunfire percentage as a prime defect of the IBC database when, in fact, IBC and the UCIMS are very much compatible on this metric.  Indeed, the preponderance of gun deaths has been a prime talking point for IBC since shortly after the invasion phase of the war (when air strikes predominated). So the Hapopian et al. insight is an old one.

You’d think that Hagopian et al. would be pleased by confirmation from IBC and would be happy to cite this agreement.  Instead, sadly, they manufacture a falsehood about IBC – that it rarely records gun deaths when the truth is that most deaths  in the IBC database are gun deaths.  They then swipe at IBC from atop their fictitious creation..

And this point about gun deaths is just a tiny drop in the sea of slime Hagopian et al. sling at IBC.  I’ll return soon for more cleansing.