The scientific community is experiencing a replication crisis that is drawing renewed attention to weaknesses in the peer-review process that many have acknowledged for years, Aaron Carroll, a professor of pediatrics at Indiana University School of Medicine, writes in the New York Times' "The Upshot."

Cheat sheet series: Evidence-based medicine 101

According to Carroll, people have long "been aware of problems with peer review." But in recent years the number of institutions and researchers that have been unable to replicate research findings has prompted many to question the authenticity and integrity of a number of studies.

The weaknesses of peer-review

Carroll writes, "Throwing out the system—which deems whether research is robust and worth being published—would do more harm than good. But it makes sense to be aware of peer review's potential weaknesses." He touches on four potential weaknesses with the peer-review process.

Two ways to identify clinical variation—and improve outcomes

According to Carroll, peer reviewers are scientific experts who specialize in the research topics they are reviewing, but they do not undergo specific training to conduct peer reviews. Carroll also notes that more than 2.5 million papers are submitted for peer review across the world annually. As such, it can be difficult to find enough reviewers to evaluate all the submitted research, particularly because reviewers often are not paid for their services, Carroll writes.

According to Carroll, a paper published in 1982 showed the peer-review process can be inconsistent. For the paper, two researchers replaced the names on 12 articles that already had been published in reputable journals and resubmitted the articles for review 18 to 32 months later. According to the paper, only 8% of editors and reviewers identified the submissions as duplicates, which resulted in three of the papers being pulled. The nine remaining papers continued through the peer-review process, and eight of them ultimately were rejected under a second review.

According to Carroll, a reviewer must significantly agree with a paper to accept it for publication. As a result, studies debunking previous findings or conventional wisdom might not overcome the peer-review process, because a reviewer might not agree with the new findings.

Peer reviewers also might show bias when they are reviewing submissions, Carroll writes. He notes that research has shown reviewers "preferentially accept or reject articles based on a number of demographic factors."

How to improve the peer-review process

Carroll recommends four ways to improve the peer-review process.

Get 9 tactics to improve acute care cost and quality outcomes

According to Carroll, formal training on how to conduct peer reviews could help improve the quality and pace of reviews. To address the challenges with recruiting reviewers, scientific and medical journals might consider a number of incentives, including compensating reviewers or promising to expedite the evaluation of future papers they submit, Carroll writes.

According to Carroll, several journals have implemented fully blind review processes. Under such a process, peer reviewers do not have access to the names of the authors or institutions from which the paper originated. Carroll argues this could help eliminate bias in the review process, but that it might not fully eliminate bias because papers typically describe where the research occurred and cite previous findings.

Carroll writes that allowing researchers to publish manuscripts of their studies online before they submit it for peer review could help improve the review process by allowing the public to identify problems with the research. The practice is common in certain scientific fields, but Carroll writes that "[m]edical journals would probably resist this … because it could reduce their ability to get press and attention once the research was fully published."

Carroll writes that a shift in attitudes also could benefit the peer-review process. "Too often, we think that once a paper gets through peer review, it's 'truth,'" he writes, adding, "We'd do better to accept that everything, even published research, needs to be reconsidered as new evidence comes to light, and subjected to more thorough post-publication review" (Carroll, "The Upshot," New York Times, 11/5).

Learn more about why study design matters

Been awhile since your last statistics class? It can be difficult to judge the quality of studies, the significance of data, or the importance of new findings when you don't know the basics.

Download our cheat sheets to get a quick, one-page refresher on some of the foundational components of evidence-based medicine.

• Evidence-based practice (EBP)
• Observational studies
• Randomized control trials (RCT's)
• Systematic reviews
• Meta-analyses
• Statistical significance

Get all 6 Cheat Sheets