More Evidence of Fabrication in D3 Polls in Iraq: Part 2

On Tuesday I provided some eye-popping comparisons on one Iraq survey fielded by D3/KA against another Iraq survey fielded by another company at exactly the same time.  In light of this evidence any reasonable person has to agree that the D3/KA data are fabricated.  Nevertheless, today I give you a different window into the same D3/KA survey.

Recall that one of the main markers of fabrication in these surveys is that the respondents to what I’m calling the “focal supervisors” have too many “empty categories”.  A response category is “empty” for a group of supervisors if it is offered as a possible choice but zero respondents actually chose it.  For example, in Part I to this series we saw that for all public services zero  respondents for the focal supervisors said that the service was “unavailable” or that availability was “very good”.  These are, therefore, both empty categories for the focal supervisors.

Langer Research Associates tried to rationalize all the empties for the focal supervisors by arguing that other supervisors also have empties.  Langer Associates also argued that Steve Koczela and I were unfair to compare the group of focal supervisors with  the group of all the other supervisors.  This is because the number of empties should be decreasing in the total number of interviews and the all-others group did more interviews than the focal group did.  Langer does have a point on this which I addressed in this post.  Here I follow up with a couple of pictures based on the same D3/KA survey discussed on Tuesday.

Each picture takes a bunch of different combinations of supervisors and for each combination plots the number of empties against the number of interviews.  The first plot graphs the data on 100 combinations of three supervisors plus the focals.  The second plot graphs the data on 100 combinations of four supervisors plus the focals.

Empties versus Interviews_three supervisors

Empties versus Interviews_four supervisors

You can see that:

1,  The number of empties is, indeed, decreasing in the number of interviews.

2.  Even after adjusting for this fact the focal supervisors still have overwhelmingly more empties than they should have, given the number of interviews they have conducted.


More Evidence of Fabrication in D3 Polls in Iraq: Part 1

Veteran readers know that I have posted a lot of this subject, including here, here, here, here , here and here.

To recap, a bunch of Iraq polls fielded by D3 Systems and its partner KA Research Limited contain data that appear to be fabricated.  In particular, there is a list of supervisors who consistently preside over non-credible interviews.  Steve Koczela and I dubbed these the “focal supervisors” since we focused our attention on them in our original paper on this subject.

We have known for a long time that D3/KA fielded a large number of surveys in Iraq and that we only had access to a few of them.  This changed recently. when Steve’s Freedom of Information Request to the US State Department came through, providing us with a mass of new Iraq polls.  Some of these were fielded by D3/KA and some were fielded by other companies.  This embarrassment of riches enables all sorts of new tests and comparisons.  I have only scratched the surface of the gold but I can report that lack of credibility of the D3/KA data screams off of the computer screen.

Let’s take a peak at two polls that ask exactly the same questions and were both fielded in April of 2006, one by D3/KA and the other by a company called the Iraq Center for Research and Strategic Studies (ICRSS).

Before looking at some numbers it is worth asking ourselves why the State Department Commissioned two different companies to administer an identical questionnaire simultaneously?  The only reason I can think of is that people in the State Department were suspicious of one of the companies.

In any case, for this short blog post let’s just look at one battery of questions on the availability of various services.  We compare the following two things:

  1.  ICRSS in the regions covered by the focal supervisors  in the comparable D3/KA survey:
  2. The focal supervisors in the D3/KA survey.

Of course, the two surveys should yield roughly the same answers since I hold the zone fixed in the comparisons.

The questions take the following form:

Q3_1:  Please tell me whether the following services for your neighborhood [in the quarter in which you live] over the past month have been very good, good, poor, very poor or not available. … Water supply

The same question is then asked for electricity, telephone service, etc.

Have a scroll through the table below:

Water Supply
Focals ICRSS Survey
Very Good 0 189
Good 0 977
Poor 245 466
Very Poor 198 128
Not Available 0 3
Don’t Know 0 0
NA 0 8
Electricity Supply
Focals ICRSS Survey
Very Good 0 11
Good 0 224
Poor 245 626
Very Poor 198 822
Not Available 0 80
Don’t Know 0 0
NA 0 8
Telephone Service (land line)
Focals ICRSS Survey
Very Good 0 71
Good 0 608
Poor 245 433
Very Poor 198 571
Not Available 0 36
Don’t Know 0 40
NA 0 12
Telephone Service (mobile)
Focals ICRSS Survey
Very Good 0 266
Good 0 1105
Poor 245 185
Very Poor 198 142
Not Available 0 40
Don’t Know 0 21
NA 0 12
Garbage Collection
Focals ICRSS Survey
Very Good 0 57
Good 0 608
Poor 245 667
Very Poor 198 373
Not Available 0 53
Don’t Know 0 0
NA 0 13
Sewage Disposal
Focals ICRSS Survey
Very Good 0 64
Good 0 574
Poor 91 662
Very Poor 352 370
Not Available 0 87
Don’t Know 0 0
NA 0 14
Conditions of Roads
Focals ICRSS Survey
Very Good 0 26
Good 0 532
Poor 148 769
Very Poor 295 388
Not Available 0 39
Don’t Know 0 5
NA 0 12
Traffic Management
Focals Nonfocals
Very Good 0 111
Good 0 834
Poor 245 505
Very Poor 198 207
Not Available 0 58
Don’t Know 0 35
NA 0 21
Police Presence
Focals ICRSS Survey
Very Good 0 255
Good 217 948
Poor 24 390
Very Poor 202 124
Not Available 0 23
Don’t Know 0 10
NA 0 16
Army Presence
Focals ICRSS Survey
Very Good 0 250
Good 217 834
Poor 24 371
Very Poor 202 171
Not Available 0 109
Don’t Know 0 19
NA 0 17


This is what your face looks like now:







In the D3/KA survey:

  • For six of the ten services exactly 245 rate the availability as “poor” and exactly 198 rate the availability as very “poor”
  • In two of the four cases for which the split is not 245-198 the breakdown is exactly 217-24-202
  • Despite the overwhelming preponderance of answers of “poor” and “very poor” nobody ever answers that a service is “unavailable”.
  • There are zero answers of “very good” and “don’t know.”

The above points easily condemn the D3/KA survey to the dustbin of lies but it’s a piece of cake to come up with more.

  • For four services the most common answer is “good” for ICRSS yet zero people give this answer for D3/KA.
  • ICRSS always has some responses of “unavailable” and “very good” but D3/KA always has zero people giving these answers.

This is not a judgement call.  It is blatantly obvious that the D3/KA data are fabricated.

Does the Deep Roots Theory of War Encourage Fatalism about War?

Something weird happened just when I stopped checking my favourite twiterati.

First there was an article by John Horgan.  Then suddenly there was this, this, this and probably much more, all saying that Horgan’s wrong about everything.


Maybe enough is enough.  Michael Shermer already has a good rebuttal to what Horgan wrote about war.   Still, I want to give my own take on the war discussion.

Horgan writes:

The biological theory that really drives me nuts is the deep-roots theory of war. According to the theory, lethal group violence is in our genes. Its roots reach back millions of years, all the way to our common ancestor with chimpanzees.

The deep-roots theory is promoted by scientific heavy hitters like Harvard’s Steven Pinker, Richard Wrangham and Edward Wilson. Skeptic Michael Shermer tirelessly touts the theory, and the media love it, because it involves lurid stories about bloodthirsty chimps and Stone Age humans.

I don’t know if all the people named above would say that war is “in our genes”.  However, it is obvious that very many humans are capable of great violence when they are placed in the wrong circumstances.  The Lucifer Effect by Phillip Zimbardo seems relevant here.  Social situations, such as a prison environment, can tap into a violent side of human nature which is invisible most of the time.  Of course, many humans can be violent on their own but group dynamics often seem to magnify the violence problem.  If this means that group violence is in our genes then I guess I think that group violence is in our genes.

Horgan continues:

I hate the deep-roots theory not only because it’s wrong, but also because it encourages fatalism toward war….Perhaps you believe the deep-roots theory. If war is ancient and innate, it must also be inevitable, right?

I really struggle here but I think I kind of get the point.  It seems to be that if war has been around for hundreds of thousands of year and is so deeply embedded in human nature that it has penetrated all the way down into our genes then what’s the point of struggling against war?  We might as well just accept it.

I can grant that this idea is not crazy.  Still, upon reflection it just doesn’t make sense.

The drive to have sex must be “in the genes” and go really far back in time.  Yet human populations are able to control their growth even though the sex drive is strong and innate.

More to the point, I have never heard anyone say that there is no point in trying to contain our sexual desires and that females will be perennially pregnant because the sex drive is “in the genes”. If anything the opposite is true.  The widely acknowledged strength of the sex drive has led to technological and cultural innovations aimed at avoiding excessive pregnancies.  Perhaps the main reason we take the sex drive so seriously is that we know it is inate and strong so we need to work hard overcome it.

The main point in Steven Pinker’s book “The Better Angels of our Nature” is that we humans have gradually been overcoming our in-built tendencies toward violence. Yet Horgan still cites Pinker as encouraging a fatalistic attitude that humans aren’t capable of winning the war against war.  This accusation is truly puzzling. Surely our long history of success in gradually overcoming violence should encourage us to believe that it is very much possible to continue further along this path toward peace.

Indeed. the link Horgan gives to unpack his claim that the “deep-roots” theory encourages fatalism about war goes to a story about Horgan meeting an ex military guy at a conference who thinks that war will be with us for the forseable future so we had better prepare for it rather than hoping it will go away.  The guy cites experience since Napolean and doesn’t mention anything about ancient, let alone pre-historic wars.  This story strikes me as a nonsequiter to the deep-roots-leads-to-fatalism claim.

What about the substative dispute about the roots of war?

As I understand it Horgan’s vision runs along the following lines.  A long time ago humans were peaceful.  Then some anomalous humans injected twisted ideas about warfare into our culture.  These carried us out of our natural, peaceful state to which our genes predispose us.

A central problem with Horgan’s vision, at least as I understand it, is that when the war idea comes it spreads and entrenches itself.  So the vision itself still seems to accept the premise that humans have the potential for group violence within them, ready to be tapped by entrepreneurs of violence.  That is, if the group violence idea is so alien to the human character why has it proved so attractive and resilient to humans?  Shouldn’t such an unnatural cultural implant be relatively easy to eradicate?


Maybe there is a good answer to these questions.  But for now I will continue believing that humans have inate tendencies toward group violence which can be overcome with enough effort.  We have had much success in violence reduction over the centuries and we should continue to work hard to do better in the future.  .

War Death Estimates that are Lighter than Air

I’m in the middle of reexamining the data collected by the University Collaborative Iraq Mortality Study (UCIMS) (This is joint work with my former student Stijn Van Weezel.)

The number of excess deaths estimated by the UCIMS is 405,000, 461,000, 500,000, more than 500,000…. well, that’s the point….it’s not clear exactly what the UCIMS estimate is but it has a natural tendency to rise.14345365-Hot-air-balloon-flying-up-to-the-sky-rising-high-as-a-symbol-of-adventure-and-freedom-on-a-blue-summ-Stock-Photo

The abstract of the paper states:

From March 1, 2003, to June 30, 2011, the crude death rate in Iraq was 4.55 per 1,000 person-years (95% uncertainty interval 3.74–5.27), more than 0.5 times higher than the death rate during the 26-mo period preceding the war, resulting in approximately 405,000 (95% uncertainty interval 48,000–751,000) excess deaths attributable to the conflict.

OK, this seems crystal clear; the central estimate is 405,000.  (It’s rather absurd to carry the numbers out to the nearest thousand despite an uncertainty interval 700,000 deaths wide but at least we know that the estimate centres around 400,000.)  The estimate of 405,000 is confirmed three times in the paper, not that confirmation should be necessary since the abstract must surely contain the right number.

But wait, there’s more:

Our household survey produced death rates that, when multiplied by the population count for each year, produced an estimate of 405,000 total deaths. Our migration adjustment would add an additional 55,805 deaths to that total. Our total excess death estimate for the wartime period, then, is 461,000, just under half a million people.

To support their upward adjustment the UCIMS authors say that there are 2 million refugees outside the country, that these divide into 374,532 (not 374,531?) households and that 14.9% of Iraqi refugee households suffered at least one death.  The 14.9% figure comes from a reference that seems to be unavailable but let’s just accept it.    These numbers would, indeed, imply around 56,000 deaths but not 56,000 excess deaths.

Readers of this blog will recall that excess deaths are deaths above and beyond some baseline level.  The excess-deaths concept is meant to capture deaths that would not have happened if war had been avoided.  The UCIMS estimated the baseline to be 2.89 per 1,000 per year (maybe 2.89857 would have been a better estimate?).  This is an extremely low baseline and, of course, if we raise it then then the excess death estimate of 405,000 will fall but leave this point aside.

Here I just note that even if all 56,000 estimated deaths from the refugee households occurred in a single year the death rate for these households would be around 2.8 per 1,000 for that year, slightly below the baseline used by the UCIMS.  So even if we lard on far more deaths than the 14.9% figure suggests it would still be quite a challenge to squeeze a positive number of excess deaths out of this situation.  It seems that refugees have, on average, done better than the people left behind in Iraq.  So integrating refugees into an excess-death calculation should lower the estimate, not raise it as claimed by the UCIMS authors.


Still, no one associated with the article seems prepared to stop even at 461,000.  Indeed, the very next sentence after the one quoted above switches from “just under half a million people” to “about half a million excess deaths”:

Our total excess death estimate for the wartime period, then, is 461,000, just under half a million people.


We estimate about half a million excess deaths occurred in Iraq following the US-led invasion and occupation (March 2003–2011).

Surely the PLoS editors will take the punch bowl away from the inflation party:

….their final estimate is that approximately half a million people died in Iraq as a result of the war and subsequent occupation from March 2003 to June 2011.

I guess not.

The next step is for lead author Amy Hagopian to use the media to pretty much convert the half a million number into a lower bound:

“We think it is roughly around half a million people dead. And that is likely a low estimate,” says Hagopian.

Finally, something important has been lost in the shuffle as we have traced the trajectory of the ICIMS estimate from 405,000 up to 500,000+.

The uncertainty  interval has disappeared.  

We started out with an uncertainty interval of 48,000 to 751,000 and we ended up with 500,000 as “likely a low estimate.”  Somehow, the back-of-the-envelope calculation on Iraqi refugees airbrushed all downside uncertainty off the books.  The last remaining question seems to be: by how much should we pad the half-a-million figure?


UK Charity Commission Grants Charitable Status to Every Casualty


Every Casualty (for which I’m on the Board) is now officially a charity.

The UK Charity Commission sets up hurdles for supplicants to cross to make them worthy of the Commission’s approval. This is a solemn process that builds strength.


Some requirements can seem pointless when you’re doing them but after finishing you can’t help thinking that the Commission should impose them on future applicants for all eternity.  A particularly rewarding ritual is to answer a question repeatedly across a lengthy chain of correspondence, paying a lawyer each time.

Next stop – the Bankruptcy Commission.

Where do your Data Come From?


This picture is a wonderful opener to an undergraduate seminar.  The story is that the great statistician Abraham Wald collected data on the placements of bullet holes on planes returning from World War II missions.  He made a little dot on a picture representation of a plane for each bullet hole observed on an actual plane.  As the dots accumulated the starting picture on the left was transformed into something resembling the picture on the right.  (I’ve never seen the actual Wald picture.  Surely it’s not as perfect as the one above but leave that aside.)

In the classroom you explain that engineers then used the picture to figure out where to reinforce their airplanes.  What was the right thing for them to do?  (You need to explain to students that it was not an option to just reinforce everywhere.  Doing this would be too expensive and  weigh the plane down too much.

Some people seem to think that only a genius like Wald can solve this puzzle but I find that about a quarter of the time some student just blurts out the right answer.  That said, students have the advantage of knowing that there must be a little trick or I wouldn’t ask the question in the first place.  And only a fairly small minority of students do get it straight away.

Some students think, wrongly, that anti-aircraft guns were highly accurate.  They then cast around for reasons why the enemy decided not to shoot at some parts of the allied airplanes.  To have any chance of getting the right answer you need to realize that bullets hit the airplanes pretty much randomly all over the place.

Once you clear the hurdle of realizing that the key issue is not targeting a big trap still looms over you.  The dark areas still seem to jump out at most people as the right places to reinforce, although the only argument that could support this view is that these were where the bullets were going….but we just said that the bullets went everywhere so this can’t be right.

In fact, the black areas are just where we have records of bullets hitting the planes.   Why do we have records of bullets hitting some places but not others?  I can only think of one explanation that makes sense.  Planes that are hit in the white areas don’t return.

Main Conclusion – We should reinforce the white areas.

Falling for the conceptual trap and reinforcing the black areas would be a colossal  error.  You would waste a lot of money reinforcing most of your planes, making them  unnecessarily heavy and leaving their most vulnerable parts still exposed.

The power of the example resides in the combination of a highly tempting error that is simiultaneously very costly to make.

Please now take a second look at the title of this post.  Abraham Wald carried out a highly successful bit of data collection and analysis.  The whole key to making it work was the way he thought about where his data were coming from.

The Wald airplane analysis is cool enough in its own right to justify a post.  But I bring it up now to highlight some misguided points some people are making in other contexts.  I’ll draw these issues out in future posts but for now I’ll make them just in the context of the present example.  These are obviously unfair criticisms but, hopefull, people will get what I’m doing here when they see these points made in other contexts.

  1. Wald created a mess because he collected biased data.  Wrong.  Wald did collect biased data but he thought carefully about where his data were coming from and managed to draw highly useful and appropriate conclusions from his data.
  2. I have just discovered that Wald’s data are biased.  It would be interesting to discuss whether he has committed statistical malpractice.  Wrong.  Wald actually discovered himself that his data were biased, pointed this out and integrated this fact into his thinking.
  3. I’m not saying that Wald caused Hitler to invade the Soviet Union but he did gather biased conflict data and then Hitler did escalate the war so you can draw your own conclusion on this one.  OK, this one is much dumber than 1 and 2 but you might be surprised about some arguments currently making the rounds about biased data collection causing the rise of ISIS.

Please stay tuned.