Predicting Armed Conflict Events

Zee Media is launching a news channel and web cite and, somehow or other, I wound up as featured in their web site launch.

Here is the link.

Although I find it a bit harrowing to watch myself so much on camera I would say that Zee came up with an interesting and original way to present the material.

The basic idea for the piece came from this blog post from a few months back.  This research programme has been more about armed conflict than about terrorism whereas the Zee piece has pretty much the opposite priority.  Still, I think their angle works pretty well.

Like many media outlets Zee was very interested in the possibility of prediction.  Hopefully, viewers will come away from the piece with realistic expectations about the potential for prediction.  I doubt we will every be in a position to mine past patterns and then make a useful prediction saying that there will be an attack at a particular time and place.  But I do think that it is possible to make useful predictions about broad patterns in violent events, such as the relative numbers of attacks of different sizes.

I would add that we should put a lot of effort into making predictions because this is the best way for us to learn when our theories are working and when they need to be modified.  It is very easy to cling endlessly to faulty theories when you never test them with predictions.

PS – There were a couple of minor errors in the piece that I’m trying to get corrected, mainly identifying me as a mathematician rather than as an economist.

PSS – All the work that I describe in the Zee piece is joint with Neil Johnson and Stijn Van Weezel.  I mentioned this on camera but this information didn’t make it into the final version.

Chilcot on Civilian Casualties: Part 4

In October of 2004 The Lancet published a paper by Roberts et al. that estimated the number of excess deaths for the first year and a half of the Iraq war using data from a new survey they had just conducted.  (Readers wanting a refresher course on the concept of excess deaths  can go here.)

One of the best parts of the civilian casualties chapter of the Chilcot report is the front-row seat it provides for the (rather panicked) discussion that Roberts et al. provoked within the UK government.  Here the real gold takes the form of links to three separate reviews of the paper provided by government experts.  The experts are Sir Roy Anderson of the first report, Creon Butler of the second report and Bill Kirkup, CBE of the third report.

In the next several posts I will evaluate the evaluators.  I start by largely incorporating only information that was available when they made their reports.   But I will, increasingly, take advantage of hindsight..

For orientation I quote the “Interpretation” part of the Summary of Roberts et al.:

Making conservative assumptions, we think that about 100,000 excess deaths, or more have happened since the 2003 invasion of Iraq.  Violence accounted for most of the excess deaths and airstrikes from coalition forces accounted for most violent deaths.  We have shown that collection of public-health information is possible even during periods of extreme violence.  Our results need further verification and should lead to changes to reduce non-combatant deaths from air strikes.

The UK government reaction focused exclusively, so far as I can tell, on the question of how to respond to the PR disaster ensuing from:

  1.  The headline figure of 100,000 deaths which was much bigger than any that had been seriously put forward before.
  2. The claim that the Coalition was directly responsible for most of the violence.  (Of course, one could argue that the Coalition was ultimately responsible for all violence since it initiated the war in the first place but nobody in the government took such a position.)

Today I finish with two important points that none of the three experts noticed.

First, the field work for the survey could not have been conducted as claimed in the paper.  The authors write that two teams conducted all the interviews between September 8 and September 20, i.e., in just 13 days.  There were 33 clusters, each containing 30 households. This means that each team had to average nearly 40 interviews per day, often spread across more than a single sampling point (cluster).  These interviews had be on top of travelling all over the country, on poor roads with security checkpoints, to reach the 33 clusters in the first place.

To get a feel for the logistical challenge that faced the field teams consider this picture of the sample from a later, and much larger, survey – the Iraq Living Conditions Survey:

ILCS Sample

I know the resolution isn’t spectacular on the picture but I still hope that you can make out the blue dots.  There are around 2,200 of them, one for each cluster of interviews in this survey.

Now imagine choosing 33 of these dots at random and trying to reach all of them with two teams in 13 days.  Further imagine conducting 30 highly sensitive interviews (about deaths of family members) each time you make it to one of the blue points.  If a grieving parent asks you to stay for tea do you tell to just answer your questions because you need to move on instantly?

The best-case scenario is that is that the field teams cut corners with the cluster selection to render the logistics possible and then raced through the interviews at break-neck speed (no more than 10 minutes per interview).  In other words, the hope is that the teams succeeded in taking bad measurements of a non-random sample (which the authors then treat as random).  But, as Andrew Gelman reminds us, accurate measurement is hugely important.

The worst-case scenario is that field teams simplified their logistical challenges by making up their data.  Recall, that data fabrication is widespread in surveys done in poor countries.  Note, also, that the results of the study were meant to be released before the November 2 election in the US and the field work was completed only on September 20; so slowing down the field work to improve quality was not an option.

Second, no expert picked up on the enormous gap between the information on death certificates reported in the Roberts et al. paper and the mortality information the Iraqi Ministry of Health (MoH) was releasing at the time.  A crude back-of-the-envelope calculation reveals the immense size of this inconsistency:

  1.  The population of Iraq was, very roughly, 24 million and the number of people in the sample is reported as 7,868.  So each in-sample death translates into about 3,000 estimated deaths (24,000,000/7,868).  Thus, the 73 in-sample violent deaths become an estimate of well over 200,000 violent deaths.
  2. Iraq’s MoH reported 3,858 violent deaths between April 5, 2004 and October 5, 2004, in other words a bit fewer than 4,000 deaths backed by MoH death certificates.  The MoH has no statistics prior to April 5, 2004 because their systems were in disarray before then (p. 191 of the Chilcot chapter)
  3. Points 1 and 2 together imply that death certificates for violent deaths should have been present only about 2% of the time (200,000/4,000).
  4. Yet Roberts et al. report that their field teams tried to confirm 78 of their recorded deaths by asking respondents to produce death certificates and that 63 of these attempts (81%) were successful.

The paper makes clear that the selection of the 78 cases wasn’t random and it could be that death certificate coverage is better for non-violent deaths than it is for violent deaths.

Still……

There is a big, yawning, large, humongous massive gap between 2% and 81% and something has to give.

screen-shot-2016-05-16-113942-am

Here are the only resolution possibilities I can think of::

  1.  The MoH issued vastly more (i.e., 50 times more) death certificates  for violent deaths than it has admitted to issuing.  This seems far fetched in the extreme.
  2. The field teams for Roberts et al. fabricated their death certificate confirmation figures.  This seems likely especially since the paper reports:

Interviewers were initially reluctant to ask to see death certificates because this might have implied they did not believe the respondents, perhaps triggering violence.  Thus, a compromise was reached for which interviewers would attempt to confirm at least two deaths per cluster.

Compromises that pressure interviewers to risk their lives are not promising and can easily lead to data fabrication.

3.   The survey picked up too many violent deaths.  I think this is true and                we will return to this possibility in a follow-up post but I don’t think that            this can be the main explanation for the death certificate gap.

OK, that’s enough for today.

In the next post I’ll discuss more what the expert reports actually said rather than what they didn’t say.

 

 

 

 

 

 

 

 

 

 

 

Chilcot on Civilian Casualties: Part II

After posting yesterday on Chicot and civilian casualties I’m finding it no easier to pick my second subject from this rich, but frustrating, chapter. So I’ve spun my random selectometer again and landed on the attempt to assassinate “Chemical Ali” aka Ali Hassan al-Majid.

The heading for this section in the report is “Case study of a bombing in a Basra suburb, 5 April 2003”, a formulation which captures in a nutshell the restrained rhetoric of the Chilcot report.

I recommend watching the movie  “Eye in the Sky” in conjunction with reading this section of the report.  The movie does a good job of presenting the trade-off inherent to air-strike decisions, albeit in a form that must be more pristine than is typical in real world decisions.  The choice posed in the movie is:

a.  Destroy a house containing high-ranking terrorists who are in the middle of arming young recruits with suicide vests for an imminent attack.  Doing this will also kill and injure people within some radius of the surrounding the house, including a young girl.

b.  Do nothing, which entails a high likelihood of a large suicide attack in which many people will be killed.

The film is driven by debate over whether it is justifiable to sacrifice the girl in the interest of (probably) saving many more people.  (Everyone assumes that it’s fine to kill anyone in the terrorists’ house and the question of other nearby people [and property] is swept under the rug.)

The general contours of the “Basra suburb” situation are similar to the Eye-in-the-Sky scenario.

It is, perhaps, worth noting that al-Majid (hanged by the Iraqis in 2010) really was an atrocious character.  He spearheaded the gassing of the Kurds in the late 1980’s, signing a decree that ordered:

“Within their jurisdiction, the armed forces must kill any human being or animal present in these areas.”

That said, the immediate motivation for the Coalition’s assassination attempt seems to have been that al-Majid was a key military figure in the South.  Adam Ingram, the Minister of State for the Armed Forces explained:

As the Commander of the Southern Region [of Iraq] …Al-Majid was a key Iraqi military figure whose removal from command was expected to deliver considerable military advantage…thus ultimately minimising casualties on both sides.  The attack on the place where he was believed to be located was therefore entirely lawful.

I also suspect that al-Majid’s nickname plus the widespread, but false, belief that Iraq would use chemical weapons in its fight against the Coalition greased the wheels for the Coalition decision to attack the house they thought al-Majid was in.

Collateral damage was assessed just as it is in (the fictional) “Eye in the Sky.”  However, the outcome in the Basra suburb makes the movie look quaint.  The prediction is that an attack will kill 39 civilians (51 if done at night) plus whoever is in the house  with al-Majid.  (Again, there seems to be an assumption that co-habitants can be omitted from the moral calculus.)  Defence Secretary Geoff Hoon signed off on the attack which was presented with this trade-off, although in the event the average bomb size was reduced from the original proposal.

The strike wound up killing 10 members of an innocent family (the Hamoudi’s) but not al-Majid who may or may not have actually been there at the time.  The Hamoudi’s pressed for compensation to which Ingram responded that these killings were:

“…likely to have been the result of Coalition bombing aimed at General…Al Majid.  There as no deliberate targeting of your father’s home and the losses suffered by your family were quite unintended. I appreciate that this may be of very little comfort to you now.

… the Coalition does take every care to ensure that our military action avoids injury to civilian populations.  That said it is not possible to eliminate the risk to civilians entirely, but I hope you will understand that when civilians are injured or killed in this way, this is a tragic accident rather than a deliberate event.”

 

Ingram’s formulation is really confused and “Eye in the Sky” is helpful to see why this is the case.  None of the characters in the movie are portrayed as keen to take out the little girl.  So there is, indeed, a sense in which her killing is not deliberate. On the other hand, the air strike in the movie is launched with a clear understanding that the girl’s death is a likely outcome.  So, in another sense, she was intentionally killed.  In my opinion, there is no reasonable sense in which the killing of the girl in the movie can be viewed as a “tragic accident”.

The killing of the Hamoudi’s strikes me as essentially the same situation.  There was an internal discussion within the MoD about whether the missile that hit them was on-target or a stray but I don’t think this really matters.  Either way there is no reason to suppose that the Hamoudi’s were specifically targeted.  Nevertheless, their deaths can hardly be viewed as a surprising accident, given that the attack was predicted to kill 39 civilians.

Ultimately, the MoD stuck with its initial view that the attack was legal and took the further step of concluding that no compensation was owed to the Hamoudi’s. This brings me to my final two points:

  1.  So far as I can tell the actual outcome of the attack was not factored into the MoD’s final assessment of the attack’s legality.  They went into the attack thinking that there was a high enough probability of killing al-Majid that it would be OK to kill 39 civilians along the way.  They ended up killing 10 civilians but not al-Majid.  Does the initial judgment still cover this outcome?  (I’m not saying I agree with the initial moral calculus.  I’m just saying that it can’t then be stretched to cover any far-removed contingency.)
  2. Even if the attack was fully justified it doesn’t follow that there is now no need to compensate the families of the victims.  In other words, why shouldn’t the UK bear some of the cost of the collateral damage it inflicts?

PS – The US is an elephant in the room I haven’t yet figured out how to handle. The Chilcot report is about UK involvement in the Iraq war.  The US barely surfaces in the chapter on civilian casualties although, clearly, the US is a much more important player here than the UK is.  But I’m blogging on the Chilcot report so I’ll have to retain a UK focus.

PSS – I’m proud to announce that my posts on Chilcot are mirrored on the web site of Significance, a joint project of the American Statistical Association and the Royal Statistical Society.  It’s a great publication.  Check it out.

 

Chilcot on Civilian Casualties: Part 1

The chapter on civilian casualties in the Chilcot report is stuffed with interesting material to the point that I don’t know where to start.  So I guess I’ll make a somewhat random choice and start with the internal UK discussion on whether or not to compile and release data on civilian casualties in the Iraq war.

Iraq Body Count (IBC), my long-time collaborators, have produced a must-read piece on this subject which influences a lot what I write below.

One thing about the report that surprises me is is the strength and persistence of the interest from Tony Blair about civilian casualties.  As argued by IBC, much of his interest seems to derive from the political/propaganda role of such figures, i.e., winning the “blame game” as IBC puts it.  But Blair dis ask repeatedly over a period of several years for casualty information.

The Ministry of Defence (MoD) consistently opposed all ideas about accounting for civilian casualties, arguing that it’s impossible to get really solid figures so it’s best not to try.    There was a clear recognition in some government circles of the inadequacy of this argument.  For example, the military’s Permanent Joint Headquarters (PJHQ) warned that:

The current line, that there is no reliable way of knowing how many casualties there have been…was perfectly reasonable during the decisive combat phase of Op TELIC….as long-range attacks meant that there was no source on the ground to verify … casualty numbers

Since…the end of decisive combat operations, this line has become more difficult to defend as confirmed cases of civilian casualties where UK forces are involved are recorded locally.

In other words, the UK was sitting on plenty of useful raw information that should have been collated and released. It is true that such data would have been incomplete and flawed but the information could have been usefully triangulated with other data to clarify the humanitarian situation in Iraq.  Indeed, the MoD argument is such a transparent case of making the best into the enemy of the good that I find it hard to believe that MoD officials offered it sincerely.

The Foreign and Commonwealth Office (FCO), like Tony Blair, wanted to arm itself with civilian casualty information so it could battle effectively with critics of the war.  Foreign Secretary Jack Straw argued:

….I am concerned that the current UK/US position – that ‘there is no reliable means of ascertaining the number of civilian casualties, even in post-conflict Iraq’ – leaves the field entirely open to our critics and lets them set the agenda.

We need to find ways of countering the damaging perception that civilians are being killed needlessly, and in large numbers, by Coalition forces.

Thus, Straw assumes that Coalition forces are not killing civilians needlessly and in large numbers and wants to be supplied with numbers he can use to drive home these facts, as he sees them, to the general public.

Yet the interest of the FCO, and Blair, in civilian casualty numbers evaporates when people realize that the Coalition might be killing more people than the “terrorists” are.  For example, a private Secretary to Tony Blair writes:

You asked for an assessment of civilian casualties in Iraq, noting that we cannot let figures of 10-15,000 go unchallenged as if we are responsible for them….

The FCO recommend that we stick to publicising terrorist responsibility for civilian casualties in individual incidents.  Underlying this is concern that any overall assessment of civilian casualties will show that MNF [Multi-National Force – Iraq] are responsible for significantly more than insurgents/terrorists.

This  strategy of generally withholding data on civilian casualties while cherry picking particular enemy atrocities for publicity is alarming in the extreme for at least two reasons:

  1.  If the reality was that the Coalition was needlessly killing many civilians then the UK government was intentionally blinding itself to this fact, thereby foreclosing the possibility of making improvements.
  2. The UK government would disclose civilian casualty figures to its citizens if these figures were good news, thereby betraying the public trust and forcing people to assume the worst in future when data are not made available.

There is more in the report on this subject but I think the essence and outcome of the discussion are clear. The UK did not compile or release civilian casualty statistics because such figures would not have been perfect and because government officials were afraid of what such figures would show.

Chilcot – Brexit Parallels

The release of the Chilcot report is so soon after the Brexit vote that I can’t help comparing the two.

And I know this is strange but I notice several surprisingly close resemblances between the Brexit vote and the decisions (both of the UK and the US) to invade Iraq:

  1.  Both were genuinely dramatic actions that were to be taken without much of a plan for what would come next.
  2. Both were misinformed by false information.  For example, there were campaign claims that Brexit would lead to a flood of money for the National Health Service and that the EU is on the verge of admitting a bunch of Muslim-majority countries.  Of course, the decision to invade Iraq was greased by the false claim that Iraq had (and was developing more) Weapons of Mass Destruction.
  3. Both decisions were widely opposed in Europe but masses of people in the US and UK assumed that  these Europeans are idiots.  (Of course, on the US side this is mostly true of the invasion decision, not Brexit which wasn’t much of a concern in the US.)

I wonder if readers have further thoughts on this.  I find this subject interesting although I’m not sure that it has any practical significance.

Chilcot Cometh

At long last the Chilcot report is out and I’m proud to announce that I’ve already read 0.000019783% of it.

So far I have fixated mainly on the issue for which I threw down a marker earlier on the blog – Tony Blair’s use of bogus child mortality figures before the Inquiry to justify the invasion of Iraq.

This press release is the initial fruit of my reading.  Please have a look and let me know what you think of the high resolution picture.  (Warning – at least on my computer the picture came out small until I clicked the supersizing button on my screen.)

I’ll definitely be back on Chilcot.