Digital collage by JC Johnson using images from Wiki Commons.
ARTICLE BY DR. MARK LEVENSTEIN
“This article has not yet been peer-reviewed.”
“In a recent preprint appearing on bioRxiv…”
“Research now published in the peer-reviewed journal X.”
These may be phrases you are now familiar with after reading them in popular science or general news articles, especially since the start of the COVID-19 pandemic.
But what exactly is peer review? What is the difference between a preprint manuscript and a peer-reviewed article? And most importantly, what information should I trust? These are questions that the general public and even well-informed non-scientists may not be able to fully answer. However, answering them is essential for society as we continue to navigate this new era of alternative facts, fake news, and the increasing access to and practical implications of scientific findings and discoveries.
The goal of this article, then, is to peel back the curtain on this ubiquitous but slightly mystifying academic phenomenon of peer review, often considered a “gold standard” by academics and non-academics alike. I will break down the purpose and process of peer review from the perspective of a physical and chemical scientist and share some opinions and anecdotes. Although I can only speak from my experience in these fields, the general characteristics of peer review I will describe hold true for the biological and life sciences and likely for the social sciences as well. I will also discuss some new developments in peer review intended to improve its quality, consistency, and transparency.
WHY DOES PEER REVIEW EXIST?
Scientists and researchers have a duty to communicate their findings with each other, their funders, and the public, whose taxes have often paid for all or part of a research study. The primary mechanism of this communication is through the publication of articles in scholarly journals, which provides an official and permanent record of the study in the academic literature. Since adding to the agreed-upon knowledge of humanity is not something to be taken lightly, there must be some type of quality control. For the past 50 years or so, this quality control has come in the form of peer review.
At face value, peer review is exactly what it sounds like: Research outputs from scientists are reviewed by their scientific peers to assess their soundness. Ideally, these peers should work in a closely-related field—so they can accurately evaluate the methods and results of the study—and should not be direct collaborators with the authors of the study—to avoid bias or cronyism. It is the job of the peer reviewers to decide if the reported experiments were conducted well and if the results of the experiments support the conclusions drawn by the authors. The same is required for research primarily conducted using computer simulations rather than physical experiments, and even for purely theoretical research.
WHAT ARE THE STEPS IN THE PEER REVIEW PROCESS?
The peer review process begins with the drafting and submission of a manuscript to a scientific journal. The authors of the manuscript will curate months or, more likely, years of results to tell a story that fits within the scope and style of a particular journal and that will be interesting to its readership. Some journals are very specific to a particular field of study, experimental technique, or even type of sample (Blood and Bone always spring to mind), and other journals are broad or interdisciplinary. Some of the latter types of journals aim to present the most important and exciting works from each field (think Nature and Science).
Once a journal receives a manuscript, it will be assigned to an editor who is either a full-time professional editor or an academic serving as a part-time editor on top of their own primary research and/or teaching role (nowadays journals may subject manuscripts to pre-review by several editor panels comprising both types of editors). The editor will decide if the manuscript is appropriate for the journal and then select scientists to serve as potential peer reviewers. Often authors are asked to suggest or exclude potential reviewers at the time of submission, but the editors are not required to follow their suggestions. The editor will then send the manuscript out to these candidates until enough of them accept the offer to review (typically two or three are required).
Peer review is normally “single-blind,” meaning the reviewers see who authored the manuscript, but the reviewers’ identities remain hidden to the authors. The anonymous reviewers are typically given between one and three weeks to send their reviews back to the editor, who will then decide whether to accept the manuscript outright (very rare), allow the authors to respond to questions and criticisms raised by the reviewers, or reject the manuscript.
Reviewers are often asked whether they do or do not recommend publication, but ultimately it is the editor who makes the decision of whether to publish. If the authors are given the chance to respond to the reviews, they are typically given one to three months, depending on the severity of the reviews and the need for either major or minor revisions to the manuscript. Minor revisions include typos or a request for an additional reference to be added to the bibliography, and major revisions can extend all the way to performing additional experiments. (Side note: In my experience, editors almost always consider revisions “major” no matter how small!)
Once the authors respond to the reviews, including making any changes to the manuscript, the editor may accept the manuscript or send the revised manuscript and author responses back to the reviewers for verification. It is possible for a manuscript to go through multiple rounds of review and for the editor to request additional reviewers. However long it is, after the review process is complete, either the manuscript is accepted, and the authors work with the editorial team to finalize the publication—or it is rejected, and the authors must work to reformat the manuscript for submission to a new journal. In either event, the text of the reviews and of the author responses normally remains confidential and is not released to the public with the published article.
HOW DO REVIEWERS AND EDITORS IMPACT THE PROCESS?
From the above description, you may have already concluded that the peer review system is not foolproof. For one, it relies on the subjective opinions of a small number of reviewers, who do not always meet the ideal criteria noted above. For example, as research becomes increasingly interdisciplinary and niche, it is harder to find qualified reviewers who know enough about the subject to conduct an accurate review but who are not somehow linked to the authors, due to the networking and collaborative work that plays a large role in scientific fields. Additionally, since the authors’ identities are known to the reviewers, it is also possible for a reviewer to discriminate against an author, be it due to professional competition or personal dislike, or for more sinister reasons of geographical, racial, or gender-based prejudice.
However, even assuming an impartial reviewer can be found, it is possible for different reviewers to have wildly contrasting impressions of a manuscript. Taking a personal example, on more than one occasion I have had a manuscript rejected after extremely critical reviews, only to have it accepted after receiving glowing reviews at another journal!
Another factor gaining importance in peer review is not the peer at all: It is the editor or editorial board. In addition to having the authority to accept or reject a manuscript, they decide whether to send a manuscript out for review at all. In the past, more senior colleagues have told me, this was almost guaranteed. However, nowadays editors regularly reject articles without sending them out for review if they do not think the manuscript is “high impact” enough.
This qualitative metric has less to do with the quality or methodological rigor of the manuscript, but rather more the perceived significance of the article and the chance that it will be highly cited. This potential significance is often clouded by the zeitgeist of the moment, with editors opting to select articles in “hot” areas of research. It is even possible for an editor to reject an article after deciding to send it out and after receiving positive reviews.
But it’s not easy for editors either! They receive more submissions than ever before and are having a harder and harder time finding peers willing to serve as reviewers. Scientists are neither paid by journals nor their employing institutions (universities, government labs, etc.) for reviewing work, so they serve as peer reviewers almost entirely based on a sense of duty to science. However, taking the time to give a quality review of another’s manuscript is something you can only do so often. A good rule of thumb is to accept to review the same number of manuscripts that you submit (or publish) per year, but it is an honor system that breaks apart if not everyone holds up their end of the bargain.
This leads us to the quality of the reviews. If peer reviewers do not have enough time to properly review an article or are not qualified to review an article, they may fail to understand it. This can lead to bad articles slipping through or to good articles being rejected. In fact, almost any scientist you talk to in any field will complain about the quality of new peer-reviewed articles they see compared to their own research. Yet those very same authors will often be disappointed by the difficulty they have in publishing their own manuscripts.
Every scientist has what is called a “Reviewer 2” story. Typically, manuscripts have two reviewers and a third reviewer if the first two reviewers do not agree on the quality and/or impact of the paper. (Some journals, though, may have three reviewers, and extremely rarely, four, as standard.) The bad review always seems to come from Reviewer 2, who often hates the paper, misunderstands it almost intentionally, makes personal attacks against the authors, and—worst of all—asks the authors to perform more experiments before they would consider the manuscript publishable. This can be extremely difficult because often the student or postdoctoral researcher who physically performed the experiments has already moved on to a new lab. The original lab may have also moved on to a new project and setting up old equipment and training a new student on a previous student’s experiment can take several months or more.
HOW DOES TIMELINE AFFECT PEER REVIEW?
The above difficulties and challenges in the peer review process make publication exceedingly slow. For example, after a study has been completed, it can take six months to a year or even longer to publish a manuscript if it goes through submission to multiple journals and multiple rounds of peer review. This long timeline has, in part, led to the increased popularity of preprint servers—especially for timely results that need to be disseminated quickly (e.g., in a public health crisis).
Preprint servers host manuscripts that have not yet been peer-reviewed, in order to enable their immediate public access. Typically, these servers are discipline-specific, such as arXiv (pronounced “archive”) for physics, ChemRxiv for chemistry, bioRxiv for biology, and medRxiv for medicine. But they can also be general, such as Research Square. While not peer-reviewed, these preprint manuscripts typically undergo some form of verification before they are posted. This includes making sure the authors are real people—and even credible in the subject area of the manuscript (as done by arXiv)—and that the manuscript appears to be an authentic scientific work.
While posting to preprint servers was originally popular with only small groups of scientists (read: physicists) and not encouraged by most journals, preprint servers are now used by many scientists across all disciplines and actively supported by journals. For example, ChemRxiv is owned and managed by some of the largest chemistry societies and publishing houses in the world. Further, many preprint servers now enable authors to directly transfer their manuscript to a journal upon uploading the preprint, and some servers even update readers with information on a preprint’s status in the peer review process (for example, Research Square). The preprint process is becoming so integral to the publication process that recently the journal eLife decided to only consider manuscripts that have first been posted on a preprint server.
PEER REVIEW: EFFECTIVE OR NOT?
While peer review is not a perfect system, there are ways it is being adapted and improved. Many changes have been proposed. Preprint servers represent just one of these changes. Another proposed improvement includes “double-blind” review, where, to prevent bias, neither the reviewers nor the authors know each other’s identities.
Conversely, some have suggested changes in the opposite direction: so-called “open review.” In the most extreme form of open review, not only are reviewers and authors known to each other, but the whole review process is conducted in public. First, the submitted manuscript is published online, then official peer reviewers and the community at large can post comments on the manuscript and publicly engage with the authors. The advantages here are that the entire review process is completely transparent, and many more voices can be considered in the evaluation of a manuscript. Journals such as Atmospheric Chemistry and Physics have implemented a full public peer review process as described above, and even journals in the prestigious Nature Portfolio have experimented with publishing the content of reviews alongside final articles. Such modifications are not without critics; however, these efforts shows that many in the scientific community realize peer review is not perfect and are committed to making it better and fairer.
So, where does that leave us? Should we trust only peer-reviewed articles and disregard preprints? Well, my advice would be to approach any scientific article with caution. We should not blindly trust a peer-reviewed article simply because it has undergone peer review, and we should not discount a preprint simply because it has not. Peer review is hardly a consistent and foolproof measure of truthfulness—it just happens to be the best system we have at the moment. Likewise, preprints were created for a reason, and this was to get important information into the hands of the public, scientists, and policy makers as quickly as possible.
Ultimately, both preprints and peer-reviewed articles are at the cutting edge of research, rather than being settled science. It is often not until hundreds of papers have been published, decades have passed, and information is included in student textbooks, that we are really sure about the truthfulness of a claim. And even then, wrong things have been published in textbooks and needed correcting. Therefore, we should all approach scientific communication with humility, from researchers and peer reviewers to the general public
As encapsulated by a quote from Albert Einstein, often shared by one of my scientific mentors: “If we knew what it was we were doing, it would not be called research, would it?”
Mark Levenstein is an American materials scientist at the Alternative Energies and Atomic Energy Commission (CEA) outside of Paris, France. A former Fulbright Scholar, he completed his master’s degree and doctorate at the University of Leeds in the United Kingdom. He then performed postdoctoral work at the University of Illinois at Urbana-Champaign on the design of new materials for coral reef restoration. His current research is focused on understanding how solid materials form in order to improve products in fields as diverse as construction, recycling, and healthcare. He is also interested in science communication, public science literacy, and the history and philosophy of science. In his free time, Mark enjoys music, art, food, and, ideally, experiencing these things in new and exciting locations.