Some of the presentations are well worth viewing. But it’s a little difficult to navigate the recording unless you know what to look for. So allow me to help you out.
Around the 46-minute mark Jennifer Parsons of the UIC Survey Research Lab talks about a health survey they fielded in Chicago, including in some very poor neighbourhoods.
The Lab works hard to motivate their interviewers to do high quality work. For example, they have clients explain the importance of their studies to their interviewers. They also explain to interviewers what counts as falsification. Crucially, falsification includes interviewing people who were not selected into the sample. Interviewers are not statisticians and many might honestly believe that it’s OK to interview their friends or people who just happen to be available on the street.
The Chicago survey offered financial incentives to respondents allowing them earn up to $125 from doing enough modules. Word quickly spread on the street. People started approaching interviewers pleading to be interviewed. Three interviewers were obliging to these walk ups, thus crossing the line into falsification.
The consequences for the survey of such sampling violations are potentially severe since the pool of people who are this desperate to be interviewed might well differ substantially from the more general pool the survey wanted to understand.
The main driver here seems to be that $125 is a good chunk of money in a poor neighbourhood. Another possible factor, highly relevant for surveys conducted in war zones, is that poor neighbourhoods also tend to be dangerous neighbourhoods. Some interviewers may have been keen to parachute quickly to safety by doing some quick interviews with readily available people.
This episode reminds me of the notorious Burnham et al. survey estimating the number of people killed in the Iraq war. (See this, this and this on Burnham et al.). The interviewers for this survey entered a neighbourhood, somehow gathered together a bunch of children, explained the survey to them and sent these children out to spread the word. (I know this is hard to believe….). So lots of people in each visited neighbourhood knew that interviewers were going around asking about war deaths.
I wonder whether some of these interviewers were approached by people demanding to be interviewed about deaths they knew of and whether some of these interlopers pushed their way into the sample.