Richard Phelps writes for the Martin Center about flaws embedded within the academic peer review process.
I recently submitted a manuscript to an education journal, a review essay of another scholar’s work. It opened with a compliment of the author’s “highly-praised and influential work.” To that statement, one reviewer of my manuscript asserted that I used “emotionally loaded language of incredulity, dismissiveness, and hyperbole.”
More “tone policing” comments riddled the review, suggesting that even when my words might sound benign or complimentary, what I really meant was malevolent. There were several examples of another curious critique type as well: Raw declarations that my claims could not possibly be true, without any effort having been made to follow my footnotes to the evidence.
Few of my manuscript submissions to education journals have been reviewed substantively. For those familiar with Paul Graham’s hierarchy of disagreement, the reviews tend to lack a refutation of the central point, refutation in general, or counterargument. Instead, most consist of responding to (perceived) tone or being ad hominem (even anonymously) attacks.
I consider it likely that this reviewer knew that I had written the manuscript. And that may have motivated the decision to volunteer. If I read the editor’s reviewer numbering system correctly, three others had agreed to review but then failed to follow through. Who knows how many were originally asked?
As it is, the traditional peer review system relies on unpaid volunteers from a small population of very busy people to perform an intensive and time-consuming task. Two types of scholars feel most compelled to review papers: Those intrinsically motivated (not necessarily for noble reasons) such as the aforementioned reviewer, and those who “have to,” such as graduate students and not-yet-tenured professors.
Assume an editor with a new manuscript chooses wisely and picks two eminently qualified reviewers, they accept the invitations, and then review the manuscript responsibly. Nonetheless, they are still just two people in an intellectual circle of thousands. The odds are small that their consideration of the manuscript will bring to bear all the relevant knowledge or even a representative sample of it.
