L’Affaire Burnham: Ten Years Later

I just gave this presentation at the annual meeting of the American Association for Public Opinion Research (AAPOR) in Toronto.

The title of the talk is taken from Richard Kulka’s Presidential Address to AAPOR.  Back in 2009 AAPOR censured Gilbert Burnham of Johns Hopkins University for refusing to disclose basic information about his methodology for (over)estimating violent deaths in the Iraq war.

This action led to a massive discussion on the AAPOR listserve which Kulka analyzed in his Presidential Address.  Read it.  It’s great.

My presentation tells the story about what happened next.  In short, AAPOR moved from the cutting edge of open science to the middle of the pack.  AAPOR’s Transparency Initiative favors openness of everything except data.  Meanwhile, AAPOR grandees who conceal their data get big awards.

That said, I return from the conference with a sense that the worm is turning on AAPOR and open data.  We’ll see.

Please have a look at the slides and let me know what you think.

Addendum – I just remembered this post in which I describe how an epidemiologist friend of Gilbert Burnham’s tried to dissuade me from ever mentioning the AAPOR censure of Burnham.   It’s another example of how researchers prioritize getting along with powerful people above getting to the truth.

 

Advertisements

Show me Your Data

I would love to claim Andrew Gelman as the model for my blogging although, realistically, I’ll never never be able to match his torrent of wonderful material.  Thus, it’s always an honour when Andrew features my work on my blog.

Andrew’s piece brings high-level attention to the issue of fabrication in survey research  in general and to the issue of fabrication in a series of Iraq surveys fielded by D3 Systems and KA Research Limited in particular.

Andrew writes:

I don’t don’t don’t don’t don’t trust surveys where the data are hidden.

I think he slightly slightly exaggerates the trustworthiness of surveys with hidden data but still he’s on the right track..

Andrew’s comment drove me to take a close look at the AAPOR Transparency Initiative, an effort by the American Association for Public Opinion Research (AAPOR).  AAPOR is busy getting institutions to sign on to a pledge to disclose central aspects of their methodology.  This is important work since we know that some people like to hide their methodologies and still hope to be taken seriously.  Nevertheless, AAPOR has not traditionally pushed for disclosure of data so I was skeptical of this initiative.

Thus, I was pleasantly surprised to discover that AAPOR’s disclosure standards have moved forward to include some pressure for data disclosure in addition to AAPOR’s longstanding emphasis on methodological essentials such as sampling designs, question wordings and target populations:

“Finally, reflecting the fundamental goals of transparency and replicability we share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request.  Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis.  In order to protect the privacy of individual respondents such datasets must be de-identified to remove variables that can reasonably be expected to identify a respondent.  Those who commission publicly disseminated research have an obligation to disclose a rationale for why eventual public release or access to the datasets is not possible if that is the case.” (Informational Module 5)

Honestly, it would be much better to say that if you want to be part of the Transparency Initiative then you have to share your data.  It’s hard to understand how an institution that hides its data can claim to be a paragon of transparency.

Still, the glass is at least half full here.  There are clear expectations that survey data should be released and you have some explaining to do if you violate this expectation.

Great.

I see that both Langer Associates and D3 Systems are charter members of AAPOR’s Transparency Initiative.  So I have requested the Iraq polls listed on this page from Langer Associates. I’ll make an announcement on the blog when the data arrive.