Citation Distortion: Part I

Apologies for the radio silence.  I went on holiday (for just one week) and then, somehow, have been desperately playing catch up ever since.

This 2009 paper by Steven Greenberg entitled “How citation distortions create unfounded authority….” strikes me as remarkably useful and important.

Suppose I write a peer-reviewed journal article  that includes a claim that “Campbell’s soup prevents breast cancer.”  I immediately drop a footnote citing seven journal articles.

Many readers will probably believe that the cited articles support the breast-cancer claim.   Sadly, nothing could be further from the truth.  In fact, it can easily turn out that none of the cited papers offer any support for my claim and some may even provide contrary evidence.

Faking people out with a blizzard of footnotes is a surprisingly effective strategy.  First of all you can intimidate many readers with your apparent erudition.  Plus people will ask themselves whether they want to invest precious time tracking down a load of footnotes.  They may figure that one or two of the citations could turn out to be flawed but is it really possible that all seven are not as advertised?

Yes, it is possible.

Have a look at pages 36-38 of this paper of mine.  This is just a sliver of a critique of the infamous Burnham et al. (2006) paper that dramatically overestimated the number of violent deaths in the Iraq conflict.  I will definitely return to this paper in future posts but for now I just want to note that there could hardly be a better example of a claim supposedly backed up by a lot of sources that don’t actually check out.

Greenberg’s study covers 242 papers with 675 citations on something incomprehensible (to me) having to do with proteins and Alzheimer’s disease.  Luckily, the only thing that matters for us is that there is a big literature pitting a  side A (my term) against a side B.

Please click on this nice summary picture of the Greenberg analysis:

F1.large

The citations of side-A partisans have a pronounced tendency to back up only side A.  Greenberg calls this “citation bias”.

OK, I know that some of you will never recover from the shock of discovering that people prefer to cite evidence that backs up their own beliefs rather than evidence that calls their beliefs into question.  I apologize for doing this to you.

Quotation-Susan-Hill-innocence-Meetville-Quotes-181873

More interesting is what Greenberg calls “citation diversion”.  This means taking a a paper that supports side B but citing it as supporting side A.  Greenberg shows that three citation diversions mushroom into a whopping 7,848 false-claim chains.

Greenberg also introduces a third category, “invention”, to cover cases such as when a cited paper says nothing about the claim it supposedly backs or when a  citation elevates a mere hypothesis in the cited paper into a fact.

Once a diversion, invention or even an honest mistake is introduced into the literature it is readily perpetuated in subsequent publications.  Researchers may not bother to trace a claim back to its original sources, especially if these lazy souls  have a stake in the claim being true.

In follow-up posts I’ll give you some nice examples from the conflict literature.

P.S. – While I was writing the present post this article appeared totally backing me up.  Take it from me.  You don’t need to bother reading it.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s