Show me Your Data

I would love to claim Andrew Gelman as the model for my blogging although, realistically, I’ll never never be able to match his torrent of wonderful material.  Thus, it’s always an honour when Andrew features my work on my blog.

Andrew’s piece brings high-level attention to the issue of fabrication in survey research  in general and to the issue of fabrication in a series of Iraq surveys fielded by D3 Systems and KA Research Limited in particular.

Andrew writes:

I don’t don’t don’t don’t don’t trust surveys where the data are hidden.

I think he slightly slightly exaggerates the trustworthiness of surveys with hidden data but still he’s on the right track..

Andrew’s comment drove me to take a close look at the AAPOR Transparency Initiative, an effort by the American Association for Public Opinion Research (AAPOR).  AAPOR is busy getting institutions to sign on to a pledge to disclose central aspects of their methodology.  This is important work since we know that some people like to hide their methodologies and still hope to be taken seriously.  Nevertheless, AAPOR has not traditionally pushed for disclosure of data so I was skeptical of this initiative.

Thus, I was pleasantly surprised to discover that AAPOR’s disclosure standards have moved forward to include some pressure for data disclosure in addition to AAPOR’s longstanding emphasis on methodological essentials such as sampling designs, question wordings and target populations:

“Finally, reflecting the fundamental goals of transparency and replicability we share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request.  Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis.  In order to protect the privacy of individual respondents such datasets must be de-identified to remove variables that can reasonably be expected to identify a respondent.  Those who commission publicly disseminated research have an obligation to disclose a rationale for why eventual public release or access to the datasets is not possible if that is the case.” (Informational Module 5)

Honestly, it would be much better to say that if you want to be part of the Transparency Initiative then you have to share your data.  It’s hard to understand how an institution that hides its data can claim to be a paragon of transparency.

Still, the glass is at least half full here.  There are clear expectations that survey data should be released and you have some explaining to do if you violate this expectation.


I see that both Langer Associates and D3 Systems are charter members of AAPOR’s Transparency Initiative.  So I have requested the Iraq polls listed on this page from Langer Associates. I’ll make an announcement on the blog when the data arrive.

Dispute Resolution by Mutual Maiming

I’m puzzled by the following sequence of events.  (This story has a very clear summary.)

  1. The UN issues a report entitled “Children and Armed Conflict”.  The report highlights quite a few groups for committing grave abuses against children.  The “Saudi Arabia-led Coalition”  in the war in Yemen is on this UN blacklist.  The report fingers the Coalition for killing and maiming children and for attacking hospitals and schools.  (So far I’m not puzzled.)
  2. UN Secretary General Ban Ki Moon then announces that he is caving in to pressure and will remove Saudi Arabia from the UN blacklist:

“The report describes horrors no child should have to face,” Ban said at a press conference. “At the same time, I also had to consider the very real prospect that millions of other children would suffer grievously if, as was suggested to me, countries would defund many U.N. programs. Children already at risk in Palestine, South Sudan, Syria, Yemen, and so many other places would fall further into despair.”  (The quote is from the same summary story mentioned above.)

Moon stops just short of directly naming his blackmailer but it’s obviously  Saudi Arabia.

Of course, this story is sad and pathetic.  It would be nice to live in a world in which the UN can at least speak the truth and exert moral suasion upon belligerent parties to clean up their acts even if the UN cannot force good behaviour.  Unfortunately, we do not really live in this world.

But here’s the puzzle.  Why do the Saudis think they have accomplished something with their bullying censorship?

Saudi Arabia was named in an obscure report that is read by only a handful of specialists.  Suddenly the report is famous.   What’s the take-home message for people outside the Saudi inner circle?  Is it that the UN screwed up by naming the Saudi-led Coalition but that this mistake has now been corrected and the Saudis are finally getting the respect they deserve?  I don’t think so.

It’s as if a rape victim names her rapist but then recants, saying that he threatened to kill her unless she did so – the rapist then breaths of sigh of relief now that his good name has been cleared.

The only way I can make sense of the Saudi behaviour is to think of it as just a single  move in a long game.   This time Saudi Arabia elevates a black-hole report to a major news item spiced up by Saudi blackmail.

But next time the UN will think twice before embarrassing the Saudis.

Maybe it makes sense that way.  But if so then we should always assume that the Saudis are behaving much worse than the self-censoring UN says they are.

PS (Two hours after posting) – Looking at this again I realize that my title is a little weird.  This is because I started with the title but then the ideas drifted while I wrote and by the end the connection between the post and the title became obscure.

For the record, the idea is that the dispute resolution harmed both parties.   Saudi Arabia comes off as a bully and blackmailer in addition to the original charge of abusing children.  The UN demonstrates that it can’t be trusted to speak the truth.  So, at least in the short run, both sides are damaged by the dispute.

Forecasting World War III – An Exchange of Letters

I’m sure that this reminder of my article in Significance  will awaken warm memories in many of you.

In it, I used an article of Pasquale Cirrilo and Nassim Nicholas Taleb to help me make my case.  I think that the Cirillo-Taleb paper is quite interesting.  However, it is not in any way a shoot-down of Steven Pinker’s masterwork, The Better Angels of our Nature, as Cirillo and Taleb like to claim.   In fact, I argued in an earlier article that there isn’t even any great contradiction between Cirrilo-Taleb and Better Angels.

No matter.

I seem to have provoked Cirrilo and Taleb who wrote a protest letter to Significance about my piece.  To me, it feels like my main crime is that I didn’t dismiss Pinker as an incompetent writer of “popular science.”   Or perhaps the issue is that my short piece leaned more heavily on Better Angels than it did on Cirrilo-Taleb.  In any case, I don’t think that Cirrilo and Taleb help themselves very much with their letter.

Steven Pinker and I now have a joint reply to Cirrilo and Taleb in the current issue of Significance.

Please have a look.




Check out my New Article at

Hello everybody.

Please have a look at this new article that has just gone up on

It is a compact exposition of the evidence of fabrication in public opinion surveys in Iraq as well as the threats and debates flowing from this evidence.

My current plan for the blog is to do one follow up post on some material that was left on the cutting room floor for the article and then move on to other stuff….unless circumstances dictate a return to the Iraq polling issue.

Have a great weekend!