I’ve finally absorbed the report of the American Association for Public Opinion Research (AAPOR) on polling in the Trump-Clinton election. So I’ll jot down my reactions in a series of posts (see also this earlier post). In keeping with the spirit of the blog I’ll also offer related thoughts on survey-based approaches to estimating numbers of war deaths.
I strongly recommend the AAPOR report. It has many good insights and is highly readable.
That said, I’ll mostly criticize it here.
But before I proceed to the substance of the AAPOR report I want to draw your attention to the complete absence of an analogous document in the literature using household surveys to estimate war deaths.
There has been at least one notable success in survey-based war-death estimation and several notable failures. (two of the biggest are here and here). Yet there has not been any soul searching within the community of practitioners in the conflict field that can be even remotely compared to the AAPOR document. On the contrary, there is a sad history of epidemiologists militantly promoting discredited work as best practice. See, for example, this paper which concludes:
The use of established epidemiological methods is rare. This review illustrates the pressing need to promote sound epidemiologic approaches to determining mortality estimates and to establish guidelines for policy-makers, the media and the public on how to interpret these estimates.
The great triumph that drives the above conclusion is the notorious Burnham et al. (2006) study which overestimated the number of violent deaths in Iraq by at least a factor of 4 while endangering the lives of its interviewees.
Turning back to the AAPOR document, I want to underscore that AAPOR, to its credit, has produced a self-critical report and I’m benefiting here from the nice platform their committee has provided.
The report maintains a strong distinction between national polls and state polls. Rather unfortunately though, the report sets up state pollsters as the poor cousins of the real national pollsters.
It is a persistent frustration within polling and the larger survey research community that the profession is judged based on how these often under-budgeted state polls perform relative to the election outcome.
Analogously, we might say that Democrats are frustrated by the judgments of the electoral college which keeps handing the presidency over to Republicans despite Democrat victories in popular votes. Yes, I too am frustrated by this weird tick of the American system. But the electoral college is the way the US determines its presidency and we all have to accept this. And just as it would be a mistake for Democrats to focus on winning the popular vote while downplaying the electoral college, it’s also a mistake for pollsters to focus on predicting the popular vote while leaving electoral college prediction as an afterthought.
The above quote is followed by something that is also pretty interesting:
The industry cannot realistically change how it is judged, but it can make an improvement to the polling landscape, at least in theory. AAPOR does not have the resources to finance a series of high quality state-level polls in presidential
elections, but it might consider attempting to organize financing for such an effort. Errors in state polls like those observed in 2016 are not uncommon. With shrinking budgets at news outlets to finance polling, there is no reason to believe that this problem is going to fix itself. Collectively, well-resourced survey organizations might have enough common interest in financing some high quality state-level polls so as to reduce the likelihood of another black eye for the profession.
I have to think more about this but at first glance this thinking seems sort of like saying:
Look, for a while we’ve been down here in Ecuador selling space heaters and, realistically, that’s not gonna change (although we’re writing this report because our business is faltering). But maybe next year space heater companies can donate a few air conditioners to some needy people. It’s naive to imagine that there will be any money in the air conditioner business in Ecuador but this charity might help us defend ourselves against the frustrating criticism that air conditioner companies are supplying a crappy product.
In other words, it’s clear that a key missing ingredient for better election prediction is more high-quality state polls. So why is it obvious that the market will not reward more good state polls but it will reward less relevant national ones?
(Side note – I think there are high-quality state polls and I think that the AAPOR committee agrees with me on this. It’s just that there aren’t enough good state polls and also the average quality level may be lower on state polls than it is on national ones.)
Maybe I’m missing something here. Is there some good reason why news consumers will always want more national polls even though these are less informative than state polls are?
But maybe journalists should just do better job of educating their audiences. A media company could stress that presidential elections are decided state by state, not at the national level, and so this election season they will do their polling state by state, thereby providing a better product than that of their competitors who are only doing national polls.
In short, there should be a way to sell high quality information and I hope that the polling industry innovates to tailor their products more closely to market needs than they have done in recent years.