Fabrication in Survey Data: A Sustainable Ecosystem

Here is a presentation I gave a few weeks ago on fabrication in survey data.

It includes some staple material from the blog but, mainly, I set off in a new direction – trying to explain why survey data get fabricated in the first place.

While writing the presentation I realized that these conditions are similar to those that led to the Grenfell Tower fire.  I only hint at these connections in the presentation but I plan to pursue this angle in the future.

Data Dump Friday – Just Three this Week Plus a Cleanup of the Censorship Page


I’ve now put all the State Department public opinion polls conducted in Iraq during 2005 up on the conflict data page.

I’ve also cleaned up the censorship page after I realized that its organization is worse than the organization on the conflict data page.

And, yes, I should unite the two pages since there is now a lot of data posted on the censorship page that isn’t on the conflict data page which doesn’t make a lot of sense.  So I’ll probably set up a mirror posting next week.

Look at this post if you’ve forgotten what the censorship page is about.

Secret Data Sunday – Iraq Family Health Survey

The WHO-sponsored Iraq Family Health Survey (IFHS) led to a nice publication in the New England Journal of Medicine that came complete with an editorial puff piece extolling its virtues.  According to the NEJM website this publication has generated 60 citations and we’re still counting.   If you cast a net wider than just medical publications then the  citation count must run well into the hundreds.

But the IFHS virtues don’t stop there.  The NEJM paper, and the accompanying report, are well written and supply plenty of good methodological information about the survey.  The authors are pretty up front about the limitations of their work, notably that they had to skip interviews in some areas due to security concerns.  Moreover, the IFHS is an important survey not least because its estimate of 150,000 violent deaths discredited the Burnham et al. estimate of 600,000 violent deaths for almost exactly the same time period.  (The Burnham et al. survey hid its methodology and was afflicted by serious ethical and data integrity problems. )

I have cited the IFHS multiple times in my own work and generally believe in it.  At the same time, the IFHS people did several questionable things with their analysis that I would like to correct, or at least investigate, by reanalyzing the IFHS data.

But here’s the rub.  The WHO has not released the IFHS dataset.

I and other people have requested it many times.  The field work was conducted way back in 2006.  So what is the WHO waiting on?

I’ll leave a description of my unrealized reanalysis to a future post. This is because my plans just don’t matter for the issue at hand; the IFHS data should be in the public domain whether or not I have a good plan for analyzing them.  (See this post on how the International Rescue Committee hides its DRC data in which I make the same point.)

There is an interesting link between the IFHS and the Iraq Child and Maternal Mortality Survey, another important dataset that is also unavailable.  The main point of contact for both surveys is Mohamed Ali of the WHO.  Regarding the IFHS. Mohamed seemed to tell me in an email that only the Iraqi government is empowered to release the dataset.  If so, this suggests a new (at least for me) and disturbing problem;

Apparently, the WHO uses public money to sponsor surveys but then sells out the general public by ceding their data distribution rights to local governments, in this case to Iraq.  

This is practice of allowing governments benefiting from UN-sponsored research to withhold data from the public that pays for the research is unacceptable .  It’s great that the WHO sponsors survey research in needy countries but open data should be a precondition for this service.



The Torture Trial

I’m mesmerized by this New York Times article which shows video footage of key people testifying under oath about the US post 9/11 torture program, also known as “enhanced interrogation”.  The most interesting bits are the testimony of military psychologists John Bruce Jessen and James Mitchell.

The two psychologists — whom C.I.A. officials have called architects of the interrogation program, a designation they dispute — are defendants in the only lawsuit that may hold participants accountable for causing harm.

The trial begins in September.  Here are a few thoughts.

First, the case is brought only against the two consulting psychologists.  No government official is named:

The two psychologists argue that the C.I.A., for which they were contractors, controlled the program. But it is difficult to successfully sue agency officials because of government immunity.

The apparent untouchability of government officials is frustrating.  Based on a recent National Security Law podcast I’m not sure that immunity is as absolute as the above quote suggests but I do gather that government workers have quite strong protection against law suits.

I wonder, then, why the CIA hired outside consultants since this move seems to have created a possibility for an embarrassing law suit which, otherwise, may not have been possible.  Maybe the  CIA just couldn’t imagine a court challenge?  Or maybe it is possible, although very difficult, to sue government officials directly so dangling more vulnerable consultants before hungry lawyers could be a good way to divert them away from the CIA itself?  I don’t know.

The price tag for the consulting psychologists is astonishing.

Under the agency’s direction, the two men said, they proposed the “enhanced interrogation” techniques — which were then authorized by the Justice Department — applied them and trained others to do so. Their business received $81 million from the agency.

I guess the CIA didn’t take competitive bids.  Surely somebody would have signed up for $80.5 million.  My understanding is that the enhanced interrogation techniques were, essentially, adaptations of North Korean torture interrogation methods from the 1950’s.  So it would appear that Jessen and Mitchell were overpaid.

Seriously, though, I’m really struggling to work out what Jessen and Mitchell could have provided that the CIA valued so much.  I suspect that their consulting fees exceeded the amount of money the US government spent on counterinsurgency research during a period when the US conducted counterinsurgencies in both Afghanistan and Iraq, making this sum seem gargantuan.  On the other hand these fees are a small fraction of the money spent on a single fighter plane, making them look like a rounding error.  So maybe the CIA was just shoveling counter terrorism money out the door indiscriminately. As a government contractor once explained to me; if you pile enough shit on top of a hill eventually some will start sliding down the hill.

The last main thing that jumps out at me is the claim that at least some of the enhanced interrogation methods do not inflict pain on their targets.  For example,

“Walling,” in which a prisoner is repeatedly slammed against a flexible plywood wall, was “one of the most effective,” Dr. Jessen said. “It’s discombobulating. It doesn’t hurt you, but it jostles the inner ear, it makes a really loud noise.”

Surely it hurts to get slammed into a wall even if it’s true that a flexible wall causes less pain than an inflexible one does.  I find it hard to trust someone who says otherwise.

Enhanced interrogation must be….well….enhanced relative to unenhanced interrogation.  Most people, myself included, have understood that enhancement means more pain.  The underlying idea needs no explanation; a prisoner may be willing to withstand some pain without giving up information but with enough pain he’ll crack.  Many experts dispute this simple idea (e.g., see this monumental book.) but, at a minimum, the more-pain-more-information theory is intuitive.  It seems, however, that the consultants will not invoke it in court.

Rather, the consultants seem poised to argue that their enhancements created non-painful psychological states, such as “discombobulation”, that softened prisoners up without hurting them.  That said, the prisoners in the videos clearly recall experiencing a lot of pain and these recollections seem credible.  My point is simply that the consultants’ case should go better if they acknowledge the pain but argue that inflicting it was worthwhile.  Denying that there was any pain and claiming that their enhancements went in non-painful directions seems like a losing argument to me.


Data Dump Friday – A Film of Cats Doing Funny Things plus Somewhere Between 1 and 16 New Iraq Public Opinion Datasets

This is a great clip.

Moreover, I’ve updated the conflict data page as is my custom on Fridays.

It’s hard to quantify how much is new in these additions.

There is one file designated “STATE MEDIA – All Data”.  Then there are a bunch of files with titles like “Media – Wasit”, Media – Salah Ad Din, etc., going through governorate by governorate.  So I think that the full dataset was broken down into a bunch of mini components that are redundant.

But I figure that it’s better to post exactly what I have rather than investigating everything in detail and posting only what I think you need to have.  This isn’t the first situation like this but this case is so obvious that I figured I should comment.

Have a great weekend.