Yesterday I gave some background on this discussion.
Today I will start working through the “Exhaustive Review” conducted by Langer Research Associates (hereafter, “LRA” to save space) of this paper. (The lawyer for D3 Systems called the LRA report an “exhaustive review.” . Well, he’s a lawyer he must be right so I’ll stick with his name.)
LRA open by saying that I have posted online “false accusations” about their company. I would ask LRA to specify what they are referring to. I will be happy to apologize and correct any false accusation I have made. I have already made one such correction on my blog and am confident that I will have to do this again.
Steve Koczela and I found suspicious patterns in the interviews that certain supervisors presided over in five polls that were conducted in Iraq. We called these the “focal” supervisors since we focused on them in our paper.
The Exhaustive Review is of four different Iraq polls. The connection between our work and LAR ‘s is that both sets of polls were fielded in Iraq by the same companies at around the same time period. Moreover, the Exhaustive Review makes clear that some of our “focal supervisors” appear in the four polls they analyze so, clearly, we can learn about one set of polls from studying the other set of polls.
LAR stresses on their first page that they “have some documentary evidence at hand” :
KA/D3’s work for us in Iraq included delivery of interviewer and supervisor journals describing their field work experiences, and photos of field work as it occurred. Our review finds that we have both journals and photos of field work from the areas where Koczela and Spagat suggest that field work did not occur.
First, to be clear, it ts possible that some field work did actually occur in the areas of the focal supervisors. In the data I have there is evidence (see tomorrow’s post) for a large number of duplicated or nearly duplicated interviews.. Focal supervisors may have presided over some legitimate interviews and then done duplication with minor changes to evade detection. (Readers, please let me know if you see me anywhere stating flatly that no field work occurred.and I will correct myself.)
Second, having pictures and journals is, indeed, positive for LAR. Certainly, if the focal supervisors refused to supply such required evidence and all the other supervisors did that would that would look like quite a smoking gun in the hands of the focal supervisors.
Still, anyone fabricating data would have to be incredibly stupid to fail to comply with a requirement to deliver journals and pictures. LAR gives no indication that their Exhaustive Review extended to examining these pictures and journals. Certainly it is possible to fake such documents.
LAR should release these journals and photos along with their data. But for now have a look at this description of these documents for one poll drawn from here: Bear in mind that the focal supervisors operated in Anbar, Baghdad and Diyala.
In addition to keeping field notes, teams carried cameras to take photos of interviews when the respondents agreed. The pictures underscore the wide range of Iraq’s population, with some respondents in Western garb, down to a knotted tie; others in traditional clothes such as the hijab (veil) and dishdasha (flowing robes).
Notably, all the photos from Anbar and Baghdad are from the neck down; no respondent in either of these provinces consented to have their faces shown, an indication itself of security concerns there. In other areas – notably the far more secure Kurdish north – respondents smiled genially for the camera.
Nonetheless, even in Anbar, where insularity is high and resentment over the U.S. invasion seethes, an interviewer reported, “I have noticed that the respondents answered very seriously and were not afraid to tell me their answers to these questions.”
Hmmm….complete unanimity on neck-down photos and bland statements about how successful the interviews were. This doesn’t bowl me over.