I would love to claim Andrew Gelman as the model for my blogging although, realistically, I’ll never never be able to match his torrent of wonderful material. Thus, it’s always an honour when Andrew features my work on my blog.
Andrew’s piece brings high-level attention to the issue of fabrication in survey research in general and to the issue of fabrication in a series of Iraq surveys fielded by D3 Systems and KA Research Limited in particular.
I don’t don’t don’t don’t don’t trust surveys where the data are hidden.
I think he slightly slightly exaggerates the trustworthiness of surveys with hidden data but still he’s on the right track..
Andrew’s comment drove me to take a close look at the AAPOR Transparency Initiative, an effort by the American Association for Public Opinion Research (AAPOR). AAPOR is busy getting institutions to sign on to a pledge to disclose central aspects of their methodology. This is important work since we know that some people like to hide their methodologies and still hope to be taken seriously. Nevertheless, AAPOR has not traditionally pushed for disclosure of data so I was skeptical of this initiative.
Thus, I was pleasantly surprised to discover that AAPOR’s disclosure standards have moved forward to include some pressure for data disclosure in addition to AAPOR’s longstanding emphasis on methodological essentials such as sampling designs, question wordings and target populations:
“Finally, reflecting the fundamental goals of transparency and replicability we share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request. Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis. In order to protect the privacy of individual respondents such datasets must be de-identified to remove variables that can reasonably be expected to identify a respondent. Those who commission publicly disseminated research have an obligation to disclose a rationale for why eventual public release or access to the datasets is not possible if that is the case.” (Informational Module 5)
Honestly, it would be much better to say that if you want to be part of the Transparency Initiative then you have to share your data. It’s hard to understand how an institution that hides its data can claim to be a paragon of transparency.
Still, the glass is at least half full here. There are clear expectations that survey data should be released and you have some explaining to do if you violate this expectation.
I see that both Langer Associates and D3 Systems are charter members of AAPOR’s Transparency Initiative. So I have requested the Iraq polls listed on this page from Langer Associates. I’ll make an announcement on the blog when the data arrive.