I just published a piece on the Hurricane Maria death toll numbers.
Space is always limited with such pieces so I’ll extend it a little bit here.
Excess deaths estimates tend to have really wide uncertainty intervals and the Harvard study on the Hurricane Maria death toll is no exception. The reason for such extreme uncertainty is obvious, once you think about it, but very few people seem to have thought about it.
Suppose we estimate that country X suffered 10,000 deaths last year and we put a 95% uncertainty interval of 9,000 to 11,000 around that estimate. That’s a reasonably tight estimate: plus or minus 10%.
Suppose now that in a normal year country X suffers 8,000 deaths. However, in the year we do our estimate there was a war, hurricane, tidal wave or something else that seems to elevate the death rate. The purpose of our estimate is to quantify this elevation.
A standard excess deaths estimate is 2,000. We obtain this simply by subtracting 8,000, the normal or baseline death rate, from 10,000, the central estimate of the rate in the special year.
If we treat the normal (baseline) rate as a certainty then it is also easy to place a 95% uncertainty interval around our excess death estimate. This uncertainty interval runs from 1,000 (9,000 – 8,000) to 3,000 (11,000 – 8,000). But this is an interval of plus or minus 50%.around our central estimate of 2,000.
The point is that when you subtract off a baseline then you magnify the size of the swing when you measure this swing in percentage terms around your excess death estimate.
This means that what might be a pretty large sample size for determining the total number of deaths is actually a pretty small sample size for determining excess deaths.
Seen through this lens it’s clear that the Harvard study has a tiny sample size. So it is no surprise that they published a preposterously wide uncertainty interval of about plus or minus 87%.
The moral of the story is that excess death surveys need very large sample sizes compared surveys aimed just at measuring total deaths.
War, Numbers and Human Losses 