Let’s go back to February of 2016 when the New York Times ran this headline:
Death Toll from War in Syria now 470,000, Group Finds
The headline is more conservative than a caption in the same article which reads:
At least [my emphasis] 470,000 Syrians have died as a result of the war, according to the Syrian Center for Policy Research.
This switch between the headline and the caption is consistent with a common pattern of converting an estimate, that might be either too high or too low, into a bare minimum.
“We use very rigorous research methods and we are sure of this figure,” Rabie Nasser, the report’s author, told the Guardian. “Indirect deaths will be greater in the future, though most NGOs [non-governmental organisations] and the UN ignore them.
“We think that the UN documentation and informal estimation underestimated the casualties due to lack of access to information during the crisis,” he said.
Oddly, none of the news articles say anything about what this rigorous methodology is. The Guardian refers to “counting” which I would normally interpret as saying that the Syrian Center for Policy Research (SCPR) has a list of 470,000 people killed but it is not at all clear that they really have such a list.
This report was the source for all the media attention. The figure of 470,000 appears just once in the report, in a throwaway line in the conclusion:
The armed conflict badly harmed human development in Syria where the fatalities in 2015 reached about 470,000 deaths, the life expectancy at birth estimated at 55.4 years, and the school age non-attendance rate projected at 45.2 per cent; consequently, the HDI of Syria is estimated to have lost 29.8 per cent of its HDI value in 2015 compared to 2010.
The only bit of the report that so much as hints at where the 470,00 number came from is this:
The report used results and methodology from a forthcoming SCPR report on the human development in Syria that is based on a comprehensive survey conducted in the mid of 2014 and covered all regions in Syria. The survey divided Syria into 698 studied regions and questionnaire three key informants, with specific criteria that guarantee inclusiveness and transparency, from each region. Moreover, the survey applied a strict system of monitoring and reviewing to ensure the correctness of responses. About 300 researchers, experts, and programmers participated in this survey.
This is nothing.
The hunger for scraps of information on the number of people killed in Syria is, apparently, so great that it is feasible to launch a bunch of news headlines just by saying you’ve looked into this question and come up with a number that is larger than what was previously thought. (I strongly suspect that having a bigger number which you use to dump on any smaller numbers is a key part of getting noticed.)
That said, the above quote does promise a new report with more details and eventually a new report was released – but the details in the new report on methodology are still woefully inadequate. They divide Syria up, interview three key informants in each area and then, somehow, calculate the number of dead people based on these interviews. I have no idea what this calculation looks like. There is a bit of description on how SCPR picked their key informants but, beyond that, the new report provides virtually no information relevant for evaluating the 470,000 figure. The SCPR doesn’t even provide a copy of their questionnaire and I can hardly even guess at what it looks like.
One thing is clear though – they did not use the standard sample survey method for estimating the number of violent deaths. Under this approach you pick a bunch of households at random, do interviews on the number of people who have lived and died in each one and extrapolate a national death rate based on death rates observed in your sample households. If the SCPR had done something like this then at least I would’ve had a sense of where the 470,000 number came from, although I’d still want to know details.
I emailed Rabie Nasser asking for details but didn’t hear back. Who knows. Maybe my message went into his spam folder. There are other people associated with this work and I’ll try to contact them and will report back if I hear something interesting.
I want to be clear. I’m not saying that this work is useless for estimating the number of people killed in the Syrian war. In fact, I suspect that the SCPR generated some really useful information on this question and on other issues as well. But until they explain what they actually did I would just disregard the work, particularly the 470,000 figure. I’m not saying that I think this number is too high or that it is too low. I just think that it is floating in thin air without any methodological moorings to enable us to understand it.
Journalists should lay off press releases taking the form of “I did some unspecified research and here are my conclusions.”