The peer review paradox: The (fool's) gold standard?

A concise summary of the problem, here.

Let me start with a story. In my first years at Kansas. Before the internet, smartphones, personal computers or the like, if you wanted to keep up with the literature critical to your academic profession you needed to have access to a regular, consistent summery of what was being published, the title of the article, the authors and the name of the journal, if only to lead you to the library to locate the article. No links, no Google, no quick and easy search or access. All analog, nothing digital.

As luck -- and it was luck -- would have it, I stumbled across an onion skin paper weekly publication named Current Contents, a summary of the tables of content in all the key journals in my field. Hundreds of articles, weekly. You did your own 'scan' of the tables of content to hope to find something that might be related to and useful to your research, and that was a labor-intensive process that only started with the title. You then had to hope you could find the journal, make a 'xerox' copy of the article, and add it to your literature library -- if it was as relevant as you hoped it would be.

One useful element was right inside the front cover of each issues, a 'citation classic,' referring to an article that had been cited a very large number of times, establishing its value and contribution to thinking in the field. The author of the classic article wrote a story as to the genesis of the article which in itself was valuable, i.e., better understanding the process of doing what would become invaluable research. A colleague, Paul Sackett, told me of an interesting study he ran. He did a content analysis of the stories from a number of issues and found one common, key characteristic residing in the story: the article has been rejected by the 'leading' journal in the field. Think about that a second.

Now, before I get to the linked video story, let me cite another senior colleague at Kansas, Ken Mackenzie. Ken was a chaired professor and not an easy person to know. He was rarely on campus and he held some ideas critical of the way academic research ran. One anecdote he rattled to me, besides telling me that anyone whose primary academic aspiration was to earn tenure did not deserve to get it, was his claim that the key to regular publishing was to get two articles published in a journal, get invited to the editorial board, and then spend the rest of your career rejecting research that questioned your line of thinking. As gatekeeper, you could keep new and original ideas out of the journal, even if they were clearly better than previous research. Ken's recommendation to those aspiring to regular publication was to never challenge anything; just refine a age-old theory, adding almost nothing of value to the field, while yielding an article in a top-ranked journal.

You can see that Ken and Paul had reached quite similar conclusions as to 'peer-reviewed research' in the leading journal(s). And their bottom line is the same as the ecologist featured in the linked video: Nothing truly original or important surfaces from the academic peer-review process, jus groupthink, just smug reassurance of the conventional wisdom. To think and to get published outside the box you needed to find a journal that was outside the box. Over time, good research will win out over maintaining the line established by the leading journal(s). We need to be grateful that this is the case and skeptical of just about anything that emerges from the 'leading' or 'top' journals, just as we need to closely examine the entrenched biases of the editors and reviewers of those same journals when they speak out or get cited by the media. Conventional thinking does produce innovation. As ecologist Allan Savory notes in the video, candlemakers could not envision electric lights, innovation comes from challenging conventional wisdom, not from reaffirming it, usually with rhetoric.

In one topical respect, the peer review process is an ancient form of 'cancel culture,' where the squeaky wheel gets not grease but rather it gets cancelled. So when 'experts' cite the peer review process as the 'gold standard' of knowledge, it may be time to turn off the tv or to run screaming from the college classroom.