Fabrication Conference Highlights

The video record of the conference on fabrication in survey research is now up.

Some of the presentations are well worth viewing. But it’s a little difficult to navigate the recording unless you know what to look for.  So allow me to help you out.

Around the 46-minute mark Jennifer Parsons of the UIC Survey Research Lab talks about a health survey they fielded in Chicago, including in some very poor neighbourhoods.

The Lab works hard to motivate their interviewers to do high quality work.  For example, they have clients explain the importance of their studies to their interviewers.  They also explain to interviewers what counts as falsification.  Crucially, falsification includes interviewing people who were not selected into the sample.  Interviewers are not statisticians and many might honestly believe that it’s OK to interview their friends or people who just happen to be available on the street.

The Chicago survey offered financial incentives to respondents  allowing them earn up to $125 from doing enough modules.  Word quickly spread on the street.  People started approaching interviewers pleading to be interviewed.  Three interviewers were obliging to these walk ups, thus crossing the line into falsification.

The consequences for the survey of such sampling violations are potentially severe since the pool of people who are this desperate to be interviewed might well differ substantially from the more general pool the survey wanted to understand.

The main driver here seems to be that $125 is a good chunk of money in a poor neighbourhood.  Another possible factor, highly relevant for surveys conducted in war zones, is that poor neighbourhoods also tend to be dangerous neighbourhoods.  Some interviewers may have been keen to parachute quickly to safety by doing some quick interviews with readily available people.

This episode reminds me of the notorious Burnham et al. survey estimating the number of people killed in the Iraq war.  (See this,  this and this on Burnham et al.).  The interviewers for this survey entered a neighbourhood, somehow gathered together a bunch of children, explained the survey to them and sent these children out to spread the word.  (I know this is hard to believe….).  So lots of people in each visited neighbourhood knew that interviewers were going around asking about war deaths.

I wonder whether some of these interviewers were approached by people demanding to be interviewed about deaths they knew of and whether some of these interlopers pushed their way into the sample.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s