Don’t Mention the Methodology

Two recent Andrew Gelman posts connect together for me.

The first is on the Open Science Collaboration paper that attempts to replicate 100 important psychology experiments and succeeds for less than half of them.

The second is on “Being polite vs. saying what you think.”

What is the connection between the two posts?

Well, the Open Science team replicates the originals in around 39 of the 100 cases (depending on how you score).  However, a little-known fact is that in 11 of these 39 “successes” the researchers behind the original experiments refused to disclose their methodology.  Open Science couldn’t attempt to replicate these experiments so they scored them as successful replications.

The Open Science scoring methodology makes sense because:

1.  Naming a scientist for refusing to disclose his/her methodology constitutes an ad hominem attack on that scientist, in effect saying: “person X is a bad scientist because he keeps his methodology secret.”  Battles within the field of psychology are bruising enough already without getting so personal and Open Science wants to focus on methodology, not personalities.
2. Open Science has produced no evidence that would invalidate the 11 studies since it didn’t try to replicate them.  Therefore, we should consider these 11 experiments to be valid.

Really?   No, not really.  I made up the “fact” about experiments declared replicated because their authors don’t disclose their methodologies.

Hopefully, we all agree that the above (fake) procedures would be preposterous.  If  your primary concern is methodology then you can’t hand out free passes to scientists who keep theirs secret.

That said, there is a certain perverse logic at work here.  If methodological openness is central to the scientific process then there will never be a nice way to say that someone is hiding their methodology.  But if you’re too polite to criticize methodological silence then you’ve gone well beyond what Gelman already viewed as problematic politeness.

OK, let’s return now to the notorious Burnham et al. survey that estimated quite a lot more violent deaths in Iraq than several other credible sources of information did.  I don’t want to survey all the evidence in one little blog post.  Here I just want to treat the issue of methodological openness.

The original Burnham et al. paper was sparse on details about how the survey was conducted.  The authors later deepened the uncertainty over methodology by issuing contradictory statements about how they did their sampling.

The American Association for Public Opinion Research  supports minimum standards for methodological disclosure in survey research.  The required information covers only such basic things as the exact wording of the questions asked and the details of how the sample is drawn.

The original Burnham et al. publication fell short of the AAPOR standards so AAPOR investigated.  Specifically, an AAPOR committee asked Gilbert Burnham multiple times to disclose the basics of his survey methodology.  Burnham refused so AAPOR issued a formal censure.  At the time AAPOR president Richard Kulko said:

“When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research.

It’s fair to say that not providing your methodology removes your work from the scientific universe.  However, many researchers and journalists seem to believe that one shouldn’t be so crass as to mention this fact.

I got a sense of the way the wind was blowing when I was working on this paper and got comments back from an interested party:

Overall, I think it is good.  However, if you will permit me, I would recommend that you make it less personal and remove any references to Burnham getting censored, etc…. I don’t think it is relevant to the chapter and I believe it detracts from the more important points that you make.

I replied:

The AAPOR censure of Burnham is pretty central to the whole line of argument.  It is one thing to say that a scientific study has been subject to a lot of unanswered criticism as is the case here.  But the AAPOR censure effectively removes the Burnham et al. study from the scientific universe.  If you do a survey and refuse to reveal basics like your sampling methodology and your questionnaire then you are not doing science anymore.

My correspondent replied:

I still believe that the points you make can be made just as forcefully but less personally.  Staying away from naming people but rather dealing with the key issues in an objective and constructive manner.  I don’t wish to get into the details, but the censures by the AAPOR and Hopkins, in my mind, don’t invalidate the surveys themselves; they rather deal with actions by an individual(s).  But many of the points you et al. are making re: past mortality surveys are important and need further constructive discussion.  I believe that some people on both sides have become too personal in their rebuttals – and this should be avoided.

Perhaps my joke about the Open Science Collaboration doesn’t seem like a joke any more.  The message seems to be that one should focus on methodology but it is out of bounds to say that someone won’t disclose his methodology.

By now quite a number of articles have been written that treat the Burnham et al. survey as a serious contribution to science and are too polite to mention Burnham’s refusal to disclose his methodology.  For examples, see this paper, this paper (which cites approvingly both the AAPOR standards and the Burnham et al. paper but does not connect the dots between the two), and this article (but you probably can’t access it unless you have a university library).  There are many such journalistic articles but I won’t try to list them here.

Note that non-disclosure is not just an unfortunate, but unfixable, accident.  A methodology can be disclosed at any time.   It’s not like you’re up there on a quiz show and the $50,000 question comes up:

How did you conduct your survey?

Inexplicably you hem and haw and reply:

I’m not telling you

You lose the $50,000 and are kicking yourself for years.  Why didn’t you just answer the question rather than panicking?

Ladies and Gentleman.  We are living through a very encouraging groundswell of support for replicability and methodological openness in science. (Another interesting example is the “worm wars” controversy during which the researchers on the original paper have been 100% forthcoming about their methodology and data.)  If it was considered poor form to nag scientists about their methodologies back in 2009 such unscientific politeness is now a quaint thing of the past.

It’s time to wake up and talk straight.