War Death Estimates that are Lighter than Air

I’m in the middle of reexamining the data collected by the University Collaborative Iraq Mortality Study (UCIMS) (This is joint work with my former student Stijn Van Weezel.)

The number of excess deaths estimated by the UCIMS is 405,000, 461,000, 500,000, more than 500,000…. well, that’s the point….it’s not clear exactly what the UCIMS estimate is but it has a natural tendency to rise.14345365-Hot-air-balloon-flying-up-to-the-sky-rising-high-as-a-symbol-of-adventure-and-freedom-on-a-blue-summ-Stock-Photo

The abstract of the paper states:

From March 1, 2003, to June 30, 2011, the crude death rate in Iraq was 4.55 per 1,000 person-years (95% uncertainty interval 3.74–5.27), more than 0.5 times higher than the death rate during the 26-mo period preceding the war, resulting in approximately 405,000 (95% uncertainty interval 48,000–751,000) excess deaths attributable to the conflict.

OK, this seems crystal clear; the central estimate is 405,000.  (It’s rather absurd to carry the numbers out to the nearest thousand despite an uncertainty interval 700,000 deaths wide but at least we know that the estimate centres around 400,000.)  The estimate of 405,000 is confirmed three times in the paper, not that confirmation should be necessary since the abstract must surely contain the right number.

But wait, there’s more:

Our household survey produced death rates that, when multiplied by the population count for each year, produced an estimate of 405,000 total deaths. Our migration adjustment would add an additional 55,805 deaths to that total. Our total excess death estimate for the wartime period, then, is 461,000, just under half a million people.

To support their upward adjustment the UCIMS authors say that there are 2 million refugees outside the country, that these divide into 374,532 (not 374,531?) households and that 14.9% of Iraqi refugee households suffered at least one death.  The 14.9% figure comes from a reference that seems to be unavailable but let’s just accept it.    These numbers would, indeed, imply around 56,000 deaths but not 56,000 excess deaths.

Readers of this blog will recall that excess deaths are deaths above and beyond some baseline level.  The excess-deaths concept is meant to capture deaths that would not have happened if war had been avoided.  The UCIMS estimated the baseline to be 2.89 per 1,000 per year (maybe 2.89857 would have been a better estimate?).  This is an extremely low baseline and, of course, if we raise it then then the excess death estimate of 405,000 will fall but leave this point aside.

Here I just note that even if all 56,000 estimated deaths from the refugee households occurred in a single year the death rate for these households would be around 2.8 per 1,000 for that year, slightly below the baseline used by the UCIMS.  So even if we lard on far more deaths than the 14.9% figure suggests it would still be quite a challenge to squeeze a positive number of excess deaths out of this situation.  It seems that refugees have, on average, done better than the people left behind in Iraq.  So integrating refugees into an excess-death calculation should lower the estimate, not raise it as claimed by the UCIMS authors.

12-2-guard-against-inflation

Still, no one associated with the article seems prepared to stop even at 461,000.  Indeed, the very next sentence after the one quoted above switches from “just under half a million people” to “about half a million excess deaths”:

Our total excess death estimate for the wartime period, then, is 461,000, just under half a million people.

Discussion

We estimate about half a million excess deaths occurred in Iraq following the US-led invasion and occupation (March 2003–2011).

Surely the PLoS editors will take the punch bowl away from the inflation party:

….their final estimate is that approximately half a million people died in Iraq as a result of the war and subsequent occupation from March 2003 to June 2011.

I guess not.

The next step is for lead author Amy Hagopian to use the media to pretty much convert the half a million number into a lower bound:

“We think it is roughly around half a million people dead. And that is likely a low estimate,” says Hagopian.

Finally, something important has been lost in the shuffle as we have traced the trajectory of the ICIMS estimate from 405,000 up to 500,000+.

The uncertainty  interval has disappeared.  

We started out with an uncertainty interval of 48,000 to 751,000 and we ended up with 500,000 as “likely a low estimate.”  Somehow, the back-of-the-envelope calculation on Iraqi refugees airbrushed all downside uncertainty off the books.  The last remaining question seems to be: by how much should we pad the half-a-million figure?

 

Advertisements

Columbia Journalism Review lowers journalistic standards while lecturing on said standards: Part I – Suppressing Uncertainty

Isn’t Columbia University supposed to be good at journalism?  And isn’t Columbia Journalism Review considered to be a decent publication?

It sure doesn’t look that way based on this remarkably poor article by Pierre Bienaimé which turns out to be right up my alley.

The article is an embarrassment of riches for the blog since it spreads numerous falsehoods about the Iraq conflict while simultaneously bidding for a record on the number of journalism traps.that can be packed into a single little article.

So this is a teaching moment.  But I’ll have to spread the material out over several posts.

Where to start?

I have a huge amount of respect for Beth Daponte and she has a quite a nice quote in the article:

When asked about journalistic responsibility, Daponte explains that the press must “take each study and really look at what it does say and what it doesn’t say. This is where journalists really do need to have at least a rudimentary understanding of statistics and confidence intervals, and what sampling really means.”

Bienaimé immediately botches the explanation of what a confidence interval is, surely leading some readers to conclude that Daponte herself does not know what she’s doing.

A confidence interval is the degree of certainty researchers attain that their estimates—in this case, of the death toll—fall within a certain range.

That was CJR, not Beth Daponte, explaining confidence intervals.

Given that he does not know what a confidence interval is it is not surprising that Bienaimé reports only central estimates with no quantification whatsoever of the uncertainty surrounding these estimates.  This is a cardinal sin when reporting on statistical estimates.

Bienaimé compounds the error by reporting central estimates to a preposterous number of significant digits, creating a spurious sense of diamond-sharp precision.

The 2006 study estimated that the war had caused roughly 654,965 excess deaths….

A study published in PLOS Medicine in 2013, applying the same surveying methods used in the Lancet studies on a broader scale,estimated 461,000 excess deaths throughout the war from 2003 to 2011.

Maybe the 2006 study was little off, with the true number being 654,963?

In fairness, Bienaimé just reproduces the quantitatively ignorant presentation of the original paper but at least the original gives a confidence interval.  And perhaps one can make a case for carrying non-zero digits to the 1,000’s place as the second study does.

Totally suppressing all uncertainty is unconscionable.

For your convenience the 95% uncertainty interval for:

1.  the 2006 study is 392,979 to 942,636 (with no information provided on in the digits in the tenth place although I suspect the top should really go up to 942,636.199):

2.  the 2011 study is 48,000 to 751,000.