Secret Data Sunday – ABC News (in the US) Stonewalls over their Dubious Iraq Public Opinion Polls

Below is an email that I sent to Kerry Smith, the Senior Vice President for Editorial Quality at ABC news, back in November of 2016.

She did not reply..

 

Dear Ms. Smith,

I am a professor of economics specialized in the quantitative analysis of armed conflict.  I have a big body of work focused on data quality issues that arise during data collection in conflict zones, especially survey data.

Back in 2011 I wrote a paper with Steven Koczela, now a prominent pollster with MassINC Polling, that uncovered substantial evidence of fabricated data in polls fielded in Iraq by D3 Systems.  We sent our paper to various interested parties for comments, including Mathew Warshaw of D3 Systems and Gary Langer who had just moved from ABC to found Langer Associates.  We included Mr. Langer in the circulation list because ABC news had used D3 Systems for a series of polls in Iraq that now required urgent re-evaluation.

D3, backed by Langer Associates, responded by threatening to sue me and Mr. Koczela.  See this, this and this.   My university has supported me against this censorship attempt but, unfortunately, Mr. Koczela felt that he could not defend himself and signed an agreement to keep his mouth shut about this particular piece of work.  (This why only my name appears on the first link above.)  Eventually, the legal threat disappeared when I wrote to Mr. Warshaw asking him explain what, specifically, he objected to in our analysis.  He did not reply.

To his credit Mr. Koczela continued working on this issue, unearthing a large number of datasets for opinion polls conducted in Iraq by D3 Systems and other polling companies.  These have provided remarkably strong evidence of data fabrication already.  For example, see this eye-popping analysis.

Many of the D3 Iraq surveys that I now have were conducted for the US State Department.  Mr. Koczela made the State Department aware of the problem at some point and they hired Fritz Scheuren, a former president of the American Statistical Association to investigate.  His analysis confirmed the fabrication problem using an analysis rather different from mine.  Unfortunately, Dr. Scheuren signed a nondisclosure agreement but I believe he would confirm in general terms the main gist of this work and he could also give you an authoritative opinion on my analysis.  (scheuren@aol.com)

Notice that after the Huffington Post article Langer Associates did post a response to my 2011 paper.   This is, however, exceptionally weak as I explain in these articles.  Langer Associates have not addressed the new evidence that has emerged since Mr Koczela’s FOIA either.

I emailed Mr. Langer for the data from the ABC Iraq polls but he did not reply.  I asked Mr. Warshaw for the same data and he referred me to ABC news.  I am now requesting the data from you.

 At the risk of belabouring the obvious, I note that people with strong intellectual cases to make do not start by threatening to sue and finish by withholding their data.

Most importantly, ABC needs to take action to correct the historical record of the Iraq war.  These polling numbers are all over the web sites of ABC news and its partner organizations in these polls.  This work must be retracted.

It is, of course, your journalistic obligation to correct the historical record but, at the same time, I think it’s to your advantage to do so.  Fixing this problem would demonstrate a strong commitment to quality and accuracy.  I doubt you would even lose your Emmy Award.  Surely you won’t be punished for pursuing the truth wherever it leads.  I will do anything I can to help in this regard.

I suggest that we meet to discuss these issues further.  I would be happy to fly to New York at my own expense for this purpose.  Alternatively, we could talk by phone, skype or some other technology.

Sincerely,

 

Professor Michael Spagat

Head of Department

Department of Economics

Royal Holloway College

University of London

Egham, Surrey TW20 0EX

United Kingdom

m.spagat@rhul.ac.uk

+44 1784 414001 (W)

+44 1784 439534 (F)

 

Blog:  https://mikespagat.wordpress.com/

War, Numbers and Human Losses: The Truth Counts

Advertisements

Department of Corrections: Mike Spagat – Edition 2

I’ve just gone through and corrected this post.

The present post is, therefore, a follow up to this earlier one which, weirdly, was about correcting the claim that Keynes had claimed that he always corrected himself when he’s wrong.  It seems that Keynes didn’t actually say this although he did correct himself when wrong.

Anyway, I can’t say that I was jumping up and down with joy when I read the comment from Ben Prytherch and realized that I might be wrong on figure 14a of a paper of Cirillo and Taleb.  But I can honestly say that I’m glad he sent it in because he gave me an instant chance to avoid a dead end and get back on track.

I don’t know when edition 3 will go up.  I’m counting on my readers to get me there as soon as I earn the privilege.

88462092 (1)

 

 

 

The Iraq Sanctions Myth Rears its Ugly Head at Scripps College

This unusually interesting article just appeared in the the Los Angeles Times (coming to me via Michael Shermer).

The obvious angle is the spectacle of protesters at one women’s college (Scripps) attempting to follow the  path of another (Smith) in disinviting a female commencement speaker (Madeleine Albright) who had shattered gender barriers in a male-dominated world.  That said, harsh treatment of commencement speakers seems to occupy a central role in American identity almost analogous to the English tradition of replacing regular train service with buses. So maybe the gender connection is just a coincidence here, not the driving force.

There is a lot to chew over in this letter from some Scripps faculty members.   Here I just want to highlight the issue of central concern to the blog – they fall hook, line and sinker for The Iraq Sanctions Myth.

hook

In fact, the Scripps Scholars even nibble on the Tony Blair version of the myth which he used in front of the (apparently) credulous Chilcot Inquiry to justify the 2003 invasion of Iraq.

More Iraqis died as a result of 13 years of U.S.-led and controlled sanctions than as a result of the 2003 US invasion and subsequent occupation of Iraq.

The old evidence the Scripps Scholars use to support their sanctions claims would be like citing Colin Powell’s Presentation to the UN Security Council on Iraq’s WMD Program to prove that Iraq really had WMD.

All that said, there is room to criticize of Madeleine Albright over her interview with Lesley Stahl in which she did say that it was worth sacrificing half a million Iraqi children in pursuit of the sanctions policy.  Stahl’s question turned out to be based on a false premise that both Stahl and Albright accepted at the time and Albright’s answer was appalling.

There is a rough analogy here with the notorious Milgram experiments during which some participants were fooled into believing they were administering huge electrical shocks to other participants but kept pushing the button anyway amidst the screams of increasingly keyed up actors.  We know that nobody was actually shocked during these experiments but we might still criticize those who thought they were seriously harming other people but kept going.

Perhaps the Scripps Scholars will now correct the gaping error in their indictment of Madeleine Albright?.  This shouldn’t take too long.

Don't make him hold his breath too much longer.
Please correct your error before I run out of air.

 

Department of Corrections: Mike Spagat Edition

Hello old friends.  Once again I need to apologize for neglecting you but I  plan to atone for my sins by flooding you with interesting stuff over the next few weeks.

I’ve been reading the new book by Philip Tetlock and Dan Gardner on “Superforecasting”.  It is spectacularly good and I will definitely come back to it on the blog but for now I draw attention a a relatively minor point of the book.

Those of you who have clicked the “About” link on my web page know that it contains this:

download

Well guess what?  No one can find a solid record of Keynes actually saying this and it appears that, in fact, he didn’t say it.

Ooooops…..

….and, yes, I’m aware of the irony of using precisely this quote to define a blog which devotes a lot of effort to correcting errors.  So I must issue a correction and an abject apology.

That didn’t feel too bad, which is good because I’m sure it won’t be the last time I issue a correction on this blog.

Meanwhile, I pledge to live by the sentiment of the apocryphal quote.

Columbia Journalism Review does at least have an error correction mechanism

In this post and this post I criticized Columbia Journalism Review(CJR) for writing about Iraq war-death numbers without investigating the methodologies for the production of the numbers and for suppressing the uncertainty that surrounds the numbers.

Yet I need to credit CJR for one thing – they do investigate errors and make some corrections.  My personal experience with both academic journals and journalistic publications suggests that such willingness to correct errors is not typical at all.

The first error is to cite this paper as claiming something that it doesn’t actually say.  Indeed, the CJR claim doesn’t survive first contact with the paper’s abstract.

Some readers maybe be shocked to learn that even peer-reviewed journal articles often make false claims about other published literature.  In future blog posts I will show you plenty of such false citations.

CJR cites a paper that compares two datasets and estimates their overlap.  One is Iraq Body Count, with is already familiar to readers of this blog.  The second is the  “sigacts” database for Iraq, which is the official data of the US military which was brought into the public domain by Wikileaks.

So there are two datasets recording deaths in the Iraq war.  The question is – to what extent are they recording the same deaths?  CJR cites the journal article as showing that IBC captures less than 1/4 of the sigacts deaths.  The article actually estimates that IBC captures 46% of the sigacts deaths.

…it is estimated that 2035 (46.3%) of the 4394 deaths reported in the Wikileaks War Logs had been previously reported in IBC.

(In fact, the cited article is terrible and the true percentage is much higher than 46% but I will come back to this issue in a future post.)

Now it gets more interesting.  Here is a peer-reviewed article that falls straight into the same falsehood as CJR does:

the emergence of the Wikileaks “Iraq War Logs” reports in October 2010 [69] prompted the Iraq Body Count team to add to its count, but a recent comparison of recorded incidents between the two databases revealed that the Iraq Body Count captured fewer than one in four of the Iraq War Logs deaths[70].

(Citation 70 is to the same article that CJR cites.)

Thus, 46% is represented as “fewer than one in four” in a peer-reviewed journal article.  Any reader who bothers to follow the citation can spot the falsehood instantly.

Welcome to the real world.

At least we can trust doctors…maybe….a little bit?  The recent Physicians for Social Responsibility report on war deaths avers:

Generally, however, the students found that only every sixth individual death in the Logs had a match in the IBC database

Even worse – 46% is portrayed as 1/6.  (The “Logs” refers to sigacts and, yes, the matching was done by students in a course project.)

I have some sympathy for CJR here.  They are operating inside a hall of mirrors in which multiple, seemingly respectable, sources are dead wrong.  And CJR did correct the error when it was brought to their attention.

In the end there is a happy message here – we are not doomed to drab lives of pinging back other peoples’ waste products if we follow one simple rule.  Do not cite a source without first reading that source.

The other CJR correction is, perhaps, less interesting.  As we have already discussed here the CJR article shrinks the IBC methodology to a shadow of its real self, portraying IBC as only tracking a few dozen newspapers and TV broadcasts and only recording deaths of named individuals.  CJR now admits that this was wrong although its correction still don’t recognize that sigacts integration is well under way.  So this is not the greatest correction in the world but it’s better than leaving the original alone.

Should we give CJR an award for standing up and admitting to their fallibility unlike many others who never admit error?

Not really.  Doing this would be like commending a husband for going ten years without beating his wife.  Correcting error and not beating your partner are things you’re supposed to do.  Still, it is fair to criticize those who don’t rise to even this low standard.