A study (Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998 Sep;32(3 Pt 1):310-7) introduced 10 major and 13 minor errors in a fictitious scientific manuscript. The manuscript was sent to all reviewers of the Annals of Emergency Medicine, the official publication of the American College of Emergency Physicians. The Annals has been in print for more than 25 years, and is the most widely read journal in emergency medicine. The work described in the manuscript was a standard double-blind, placebo control study of the effect of the propranolol drug on migraine headaches. The manuscript was reviewed by 203 reviewers. Eighty percent of the reviewers were professors at academic emergency medicine departments, and twenty percent were physicians in private practice.

The analysis of the reviewers' comments produced the following results. Fifteen reviewers recommended publication. The reviewers in this group missed 82.7% of the major errors and 88.2% of the minor errors. Sixty seven reviewers recommended revisions. The reviewers in this group missed 70.4% of the major errors and 78.0% of the minor errors. One hundred and seventeen reviewers recommended rejection. The reviewers in this group missed 60.9% of the major errors and 74.8% of the minor errors.

According to these results, the 15 professors who recommended publication, on average, missed 82.7% of the major errors, and 88.2% of the minor errors. In other words, the professors missed at least 4 out of 5 errors inserted in the manuscript. These errors were defined by the authors as "nonremediable errors that invalidated or markedly weakened the conclusions of the study." It is interesting to note that one of the minor errors included in the manuscript was a misspelling of the drug's name. Out of the 203 reviewers, 30 were convinced in the correctness of the misspelled name and used it throughout their interview. The authors of the study said about the results (with the usual scientific undertone): "the small number of errors identified by the reviewers in this study was surprising. The major errors placed in the manuscript invalidated or undermined each of the major methodologic steps of the study