The Hurricane Maria Death Toll Numbers: An Extension of my Piece in The Conversation

I just published a piece on the Hurricane Maria death toll numbers.  

Space is always limited with such pieces so I’ll extend it a little bit here.

Excess deaths estimates tend to have really wide uncertainty intervals and the Harvard study on the Hurricane Maria death toll is no exception.  The reason for such extreme uncertainty is obvious, once you think about it, but very few people seem to have thought about it.

Suppose we estimate that country X suffered 10,000 deaths last year and we put a 95% uncertainty interval of 9,000 to 11,000 around that estimate.  That’s a reasonably tight estimate: plus or minus 10%.

Suppose now that in a normal year country X suffers 8,000 deaths.  However, in the year we do our estimate there was a war, hurricane, tidal wave or something else that seems to elevate the death rate.  The purpose of our estimate is to quantify this elevation.

A standard excess deaths estimate is 2,000.  We obtain this simply by subtracting 8,000, the normal or baseline death rate, from 10,000, the central estimate of the rate in the special year.

If we treat the normal (baseline) rate as a certainty then it is also easy to  place a 95% uncertainty interval around our excess death estimate.  This uncertainty interval runs from 1,000 (9,000 – 8,000) to 3,000 (11,000 – 8,000).  But this is an interval of plus or minus 50%.around our central estimate of 2,000.

The point is that when you subtract off a baseline then you magnify the size of the swing when you measure this swing in percentage terms around your excess death estimate.  

This means that what might be a pretty large sample size for determining the total number of deaths is actually a pretty small sample size for determining excess deaths.

Seen through this lens it’s clear that the Harvard study has a tiny sample size.  So it is no surprise that they published a preposterously wide uncertainty interval of about plus or minus 87%.

The moral of the story is that excess death surveys need very large sample sizes compared surveys aimed just at measuring total deaths.



My Free Online Course is Ready and About to Launch!

Hello everyone.

I haven’t posted for a while, mainly because I’ve been completely swamped writing with creating my free online course  which launches on Monday.  

The course is on exactly the sort of material I cover on the blog so if you’re following the blog you should seriously consider signing up for the course.  It’s called:

“Accounting for Death in War: Separating Fact from Fiction”

A Zombie Graph Lives on Courtesy of the Lancet

I’ve got a new article just out in The Conversation.  Here’s the short version:

  1. The Lancet publishes a false graph
  2. The problems of the graph are exposed, several even in letters to the Lancet.
  3. The Lancet just leaves the graph up.
  4. A Washington Post reporter stumbles onto the false graph, thinks it’s cool and reprints it.
  5. I tell the reporter that  he just published a false graph
  6. The reporter does a mea culpa and pulls the graph down
  7. I write up this sequence of events for The Conversation
  8. The Conversation sends it to the Lancet
  9. The Lancet declines to comment and leaves the false graph up
  10. The Conversation publishes the piece
  11. Someone else sees and believes in the graph?

An Interesting Job in Casualty Recording!

Check out this job opportunity.


Closing date: 12 April 2018

Programme Development Officer

Posted by Every Casualty Worldwide

Save job

Salary £28,000 –  £32,000
Location London, Greater London

This is an opportunity to make a unique contribution to changing the way that governments acknowledge the human cost of armed conflicts.

Job Description

We are looking for a person with a track record of effective advocacy in the area of human rights, humanitarian aid, international development or peacebuilding who can use their talents and energies to develop this work and our organisation.  This is a 4-days per week, 12-month role, with a possibility of extension. The post is home-based and the post-holder will be expected to operate primarily in London.

About Every Casualty Worldwide

Every Casualty Worldwide (ECW) works towards a world in which every casualty of armed violence, both civilian and combatant, is properly recognised. This is to give a human face to those who have paid conflict’s highest price, and also to enable a wide range of humanitarian, human-rights and peacebuilding goals.

Our project began in 2007 under the auspices of the London based think tank Oxford Research Group. In 2014 ECW became a standalone not-for-profit company, and in May 2016 was granted registered charity status (charity number 1166974). Our vision is succinctly expressed in the Charter for the Recognition of Every Casualty of Armed Violence launched at the British Academy in 2011 and endorsed by a wide range of supporting organisation.

Applicants should note that while it works closely with practitioners, ECW is not itself a casualty recording organisation but seeks to improve the state of casualty recording across the globe, including by responsible states and other conflict parties, to fulfil the core demands of the Charter that every casualty of armed conflict is ‘promptly recorded, correctly identified and publicly acknowledged’.

ECW has been working to fulfil its charitable objectives through research, capacity building, and advocacy. Our research has focused on better understanding and codifying existing and emerging casualty recording practice around the world and the various humanitarian, human rights, and peacebuilding benefits that it brings. Capacity building has revolved around establishing and maintaining the first international Casualty Recorders Network. These activities have underpinned our advocacy with states, who bear prime responsibility for ensuring compliance with existing norms.

In recent years our efforts have concentrated on the development and eventual publication in late 2016, of an international set of Standards for Casualty Recording, in consultation with and endorsed by casualty recording practitioner NGOs and key international agencies who use casualty data, such as the ICRC, the ICC, and UNOCHA. These have provided new openings for advocacy with governments around the world, as these standards allow the conversation to move from the politics around casualty numbers to the need to do casualty recording in any conflict according to a recognised set of principles and methods.

In the UK context, these developments have been further assisted by the publication in July 2016 of the Chilcot Report (“Iraq Inquiry”). Chapter 17 of the report, entitled “Civilian Casualties” draws heavily on the work of Iraq Body Count, one of ECW’s founding network members, and concludes by recommending that in future conflicts,

Government has a responsibility to make every reasonable effort to identify and understand the likely and actual effects of its military actions on civilians

and that

Government should be ready to work with others, in particular NGOs and academic institutions, to develop such assessments and estimates over time

Maximising the opportunity presented by these recommendations is a key element of ECW’s current priorities.

The immediate challenge (2018-19)

Following the publication of the Chilcot Report, the UK Government committed itself to conducting a ‘lessons learned’ investigation across Whitehall, coordinated by the National Security Adviser, into the substantive criticisms of the government made by the report. To assist this process, the MOD set up a Chilcot implementation unit tasked to find practical ways to meet the recommendations of the report.

ECW has been closely involved in the casualty recording element of this process, and is well placed to make significant contributions towards ensuring that the eventual full government response lives up to the spirit and letter of the Chilcot specific recommendations on casualty recording. In order to fully capitalise on this and other emerging opportunities we need to augment our capacity to respond rapidly and effectively to advance the issue of casualty recording both within government and in public arenas.

The UK is only one relevant actor of many, though well-placed to take an influential lead on this issue if it can be persuaded to do so. On the wider international front we also need to maintain and garner further momentum among key players who have already indicated their support for casualty recording in various ways.

The role of Programme Development Officer

This is a newly created role within the organization and the post-holder is expected to play a key part in expanding the impact and effectiveness of ECW, working alongside the co-directors. The role is focused on three organisational priorities. In order of immediate importance, they are:

  1. Ensuring an effective and appropriate UK government response to the Chilcot Report in the area of Casualty Recording;
  2. Maintaining growing international engagement with Casualty Recording on the relevant fora beyond the UK (e.g. various UN fora);
  3. Enhancing the institutional stability and sustainability of ECW.

The post-holder will report to the co-directors, and initially working closely with them while gaining the experience and subject knowledge to be able to work more independently.

Specific objectives to which the post-holder will contribute are:

  • Working with government officials, NGO and other partners to promote and assist development of policy and implementation in casualty recording;
  • Conducting targeted advocacy to stimulate government action, including identifying effective pressure points and opportunities for contributing to other relevant initiatives;
  • Elevating the importance and relevance of casualty recording in public discourse;
  • Ensuring the Casualty Recorders Network is kept informed of ECW activities and able to contribute effectively.

Specific activities to support work in these areas may include:

  • Engaging with the UK environment on a day to day basis, scanning the scene for advocacy opportunities, reaching out to relevant individuals and institutions;
  • Attending and representing ECW at relevant meetings and networking;
  • Making profile- and awareness-raising interventions ranging from short items (e.g. tweets) through to more substantive commentary (e.g op-eds, interviews with experts, etc.);
  • Monitoring national and international initiatives which the casualty recording agenda can support;
  • Keeping abreast of developments in armed conflict and other news relevant to casualty recording;
  • Assisting in the drafting of expressions of interest and full grant bids where appropriate;
  • Contributing web content to ECW website;
  • Contributing to the strategic planning processes of the organisation.

Person Specification

  • At least 3 years policy and advocacy experience in a human rights, humanitarian, peacebuilding, international development or other relevant contexts (whether government, international agency, or NGO), in a role involving networking and outreach;
  • High level of literacy in English;
  • Proven ability to communicate effectively with a wide range of audiences, including the ability to produce compelling text and presentations;
  • Competency with standard office and communications software;
  • Ability to manage their own time and workload, using their own initiative;
  • Experience in maintaining well-organised records of activity;
  • A good understanding of the role of new and traditional media in advocacy;
  • Commitment to ideals expressed in the Charter for the Recognition of Every Casualty;
  • A good understanding of the UK government and parliamentary system, and its connections to the international system;
  • A working understanding of the international system, and the relevant fora for civil society-led or -initiated change;
  • Existing right to work in the UK.

Further particulars

This is a home-working position and the candidate must have a private space in which to work, with a reliable internet connection. This is a 4-days (30 hours) per week position, starting July 2018 or as soon as possible thereafter, but no later than August 2018. The post runs for 12 calendar months.

We would expect normal working hours, i.e. Monday to Friday, between the hours of 9 and 6, with core working hours of between 10am and 5pm, though some flexibility around working hours could be discussed. Most routine team communications as well as collaborative work will be undertaken remotely using Skype. Allowance towards phone and internet costs will be made available. A suitable laptop will be supplied if needed. The position will require easy access to central London for face-to-face ECW team or stakeholder meetings in Westminster. All travel other than for ECW team meetings will be reimbursed.

Salary will be pro-rata to the full-time range £35,000 – £40,000 per annum (i.e. actual gross salary of £28,000 – £32,000) according to qualifications and experience.  The postholder will be required to make a minimum 2% contributions to a statutory pension scheme (NEST), to which ECW will also contribute pro-rata. Although this is a 12-month position, there is a real prospect of the position’s being extended on a more long-term basis (circumstances permitting).  The current position is made possible through a generous grant from the Joseph Rowntree Charitable Trust.

How to apply

Closing date for applications 12 April 2018.

Interviews will be held in London in early May.

Informal enquiries are welcomed prior to submission of a full application, and potential applicants may send a preliminary email to admin[AT]everycasualty[DOT]org

Full applications should consist of a single file (word or pdf) containing a description of no more than 1000 words demonstrating how the applicant meets the person specification and why they are applying for this position, along with a brief CV (including the names and contact details of two referees).

The client requests no contact from agencies or media sales.

A Debate about Excess War Deaths: Part II

My rejoinder (with Stijn van Weezel) to Hagopian et al is out. Hooray!   See this earlier post for background.

Please have a read.  It’s short and sweet.  Here’s the abstract:

Spagat and van Weezel have re-analysed the data of the University Collaborative Iraq Mortality Study (UCIMS) and found fatal weaknesses in the headline-grabbing estimate of 500,000 excess deaths presented, in 2013, by Hagopian et al. The authors of that 2013 paper now defend their estimate and this is our rejoinder to their reply which, it is contended here, avoids the central points, addresses only secondary issues and makes ad hominem attacks. We use our narrow space constraint to refute some of the reply’s secondary points and indicate a few areas of agreement.

And here’s the first paragraph:

Hagopian et al. (2018), the reply paper to Spagat and van Weezel (2017) which is, in turn, our critique of Hagopian et al. (2013), does not address either of our two central points. These are as follows (Spagat and van Weezel, 2017). First, any appropriate 95% uncertainty interval (UI) for non-violent excess deaths is at least 500,000 deaths wide and starts many tens of thousands of deaths below zero. Second, we find no local spill over effects running from violence levels to elevated non-violent death rates.1 Both these results refute the ‘conservative’ estimate of several hundred thousand non-violent excess deaths given in Hagopian et al. (2013). The fact that Hagopian et al. (2018) ignore these two points suggests that the authors of that paper are unable to respond.

In other words, Hagopian et al. can’t address our main points so they search for errors they might be able to catch us out on.

And they do actually find an error.  However, as we argue in the rejoinder, it doesn’t lead anywhere.

Specifically, we assumed, wrongly, that they drew a stratified sample.  (In fact, we didn’t digest a separate paper they wrote explaining their sampling scheme in detail.)  What this means in practice is that the number of clusters per governorate in their sample is out of line with population proportions.  For example, governorate A might have twice the population of governorate B but four times the number of clusters.  But this is a random outcome rather than being by design (our mistake).

We actually spilled a fair amount of ink in our critique discussing the importance of incorporating a stratification adjustment into the estimation.  However, we also did all our estimates both with and without such an adjustment.  And it turns out that even without a stratification adjustment it’s still true that:

any appropriate 95% uncertainty interval (UI) for non-violent excess deaths is at least 500,000 deaths wide and starts many tens of thousands of deaths below zero.

Making the adjustment widens the UI’s further but this point doesn’t matter materially.

Moreover, we still think it’s a good idea to do ex post stratification.  The Hagopian et al. sample is small and the realized numbers of clusters per governorate are pretty far out of wack with population proportions.  These imbalances would get ironed out in a large sample but this didn’t happen in the actual small sample.  We think it’s best to adjust for this imbalance.

For me, the highlight of the Hagopian et al. response is the section on death certificates which shows a strong desire by this team to have their cake and eat it too.  When households reported deaths to interviewers the interviewers then asked to see death certificates.  Hagopian et al. report that interviewees were usually able to show these certificates.  However, sometimes interviewees said that they didn’t have death certificates and sometimes interviewees said that they have them but were unable to produce one when prompted.  Nevertheless, Hagopian et al. just go ahead and assume that every single reported death is 100% certain to have happened regardless of death certificate status.  So they want to use death certificate checks in general terms to demonstrate the high quality of their data but when the outcome of a particular death certificate check casts a shadow on a particular datum they ignore this outcome.

Consider the following analogy.  I run a bar.  I ask everyone ordering an alcoholic drink if they have an ID showing they are 21 years old.  If they say they do have one then I ask to see it.  Most people just show an ID.  But some people say they have an ID, although they are unable to produce one when prompted.  Other people say they don’t have an ID.  I serve alcoholic drinks to all three types of people.  The police then investigate me to determine whether or not I’m selling to underage drinkers.  I tell them that I am certain that I never ever do this.  The reason I’m so certain is that I always ask my customers for ID’s and most of the people I serve drinks to actually show me one.

Somehow I don’t think the police would be convinced by this logic.

OK, those are the highlights – time now to read the whole thing!