The process of anonymous (or semi-anonymous) and rigorous review of scholarly work by peers in the same field of study is an essential component of the scientific process. After the scientific method, peer review is arguably the most important aspect of science. Peer review is the primary process for verification of the merits of scientific work prior to publication and dissemination within a field of study and to the public. Despite the importance of high-quality peer review, the process is not without complications or limitations. Journals, and organizations have attempted to provide guidelines for peer reviewers. However, the quality of peer reviews is often inconsistent between reviewers, and far too often reviewers “learn on the fly” during graduate school without undergoing formal training on how to effectively review a paper. This “on the fly” approach to training creates a lack of consistency in the quality of peer reviews which presents significant challenges to authors, editors and the scientific process overall. The lack of formal training programs for peer review may also discourage non-academics, mainly clinicians, from getting involved in the peer-review process. This lack of participation by non-academics in the peer review process presents a risk for potential biases regarding the utility and feasibility of published work, especially regarding clinically oriented research. This also limits the pool of available reviewers which may make it more difficult for editors to make timely decisions on submitted manuscripts. At times there just aren’t enough reviewers available to editors.
For this reason, Cardiopulmonary Physical Therapy Journal Blog will be releasing a series of blogs on the peer review process. The first part of this series features the perspective of three experienced and well-respected journal editors: Dr. Barbara Smith PT, PhD, Editorial Board Member of Physical Therapy Journal, Dr. Merry Lindsey PhD, Deputy Editor of The American Journal of Physiology-Heart and Circulatory Physiology, and Dr. Sean Collins PT, ScD, Editor in Chief of Cardiopulmonary Physical Therapy Journal.
Each editor was asked six key questions regarding the peer review process. The following will provide a summary of their responses to these questions. The goal of this series of blogs is to provide reviewers valuable advice on how to make their reviews more useful for both editors and authors. and to improve the quality of the peer-review process. We also hope this series provides guidance on how to get involved in the peer-review process. Thank you!
1) What are the key components to an effective and high-quality peer review?
All three editors recommended that the use of a systematic approach is essential to conducting an effective and high-quality peer review. The first step in this process recommended by each editor is to identify the purpose and hypothesis and then to appraise if both are clearly described, plausible and coherent with the available literature. This initial step is necessary since without a clearly identified purpose it would be difficult to provide an effective and thorough review of the methods, results and eventually the conclusion. Without a clearly described purpose and hypothesis, it would also be difficult to determine the novelty, utility, and impact of the study which are important factors to editors when making decisions on acceptance.
When reviewing the methods, reviewers should evaluate how the data was collected, processed, and analyzed. The methods used by authors should be consistent with the current literature investigating similar topics. When evaluating the results, reviewers should consider all potential confounders to the data provided (analysis, design, bias, etc.). When appraising the discussion and conclusion sections, reviewers should appraise the coherence of the interpretation. This means that the interpretation of the data provided by the authors should be consistent with the methods, results, purpose, hypothesis and previous literature. The authors should have also acknowledged the limitations and weaknesses of the study. In summary, strong conclusions or claims in the conclusion section should only come from strong and well-conducted studies.
For those starting out please consider using this open access guide published by Benos et al aptly titled “How to Review a paper”
2) What are some common mistakes or errors that you would like to see avoided by peer reviewers?
Dr. Lindsey: The biggest mistake is giving a few non-specific comments and then making a recommendation to reject. This is the worst mistake and not particularly helpful to the editor or author. Provide details on why a paper should be accepted or rejected. Another major mistake by reviewers would be not knowing what they are doing and attempting to sound like they do. If reviewers ask the authors to change the manuscript to something that is wrong, the authors should be able to say that’s incorrect and the editors should listen.
Dr. Collins: Not considering the limitations of the methods utilized, the impact that has on alternative explanations for the findings, and not considering the possibility of spurious results in studies with small sample sizes studies (either Type I or Type II errors). This issue is particularly true with trials that have not been pre-registered. Sometimes this is as simple as using an Equator checklist to be sure all components of a particular type of article are included.
Dr. Smith: A review reveals itself as low-quality when the commentary focuses more on copy edits than on the scientific premise of the study, the rigor of the measures and analyses, the interpretation of results, and impact on the field. As an editor, I cannot glean useful information from a review focused on minor word substitutions, punctuation, or font preferences and likely would not invite this reviewer to contribute in the future. That said, poor grammar or spelling could reflect sloppiness on the authors’ part that might also extend to their methodology. Therefore, in such situations, it is appropriate for reviewers to point out trends with spelling or wording that require correction, along with a specific example or two.
Dr. Smith also wanted to acknowledge this point regarding the importance of professionalism when conducting a peer review.
“I would also ask reviewers to be sensitive to the tone of their comments. Authors dedicate significant time and effort to their work, and nobody needs to be belittled by a nasty review. Even when the quality of the manuscript does not reach the threshold for acceptance, reviewers should take the opportunity to offer insights that can help less-experienced authors learn how to strengthen their future work. Professional courtesy and a little kindness never hurts anyone.”
I couldn’t agree more with this point raised by Dr. Smith. Rigor is not mutually exclusive to kindness. It takes a lot of work to produce even a single manuscript for publication. As much as we try to put on a façade of emotionless objectivity as scientists or researchers, it is never easy to have work that you’re passionate about criticized (this does get easier over time). By voluntarily exposing their work to criticism by anonymous peers, authors are in some respects placing their identity as a person in a fairly vulnerable position which requires a lot of bravery and trust. Perhaps the paper you’re reviewing is the first submission by an author who may be new to the field of science. A nasty comment by a reviewer could discourage that author from submitting to that journal again, submitting a paper again or even worse from even pursuing a career in science. No one benefits from any of those outcomes. Reviews should be rigorous and thorough. However, reviews should also be professional, kind, helpful and encourage trust in this essential process of science. Let’s use peer review to make science better and the scientific community stronger.
3) How would you prefer peer reviews to be formatted?
The responses for this prompt varied slightly since each journal does have its own format to follow. The general recommendations by the editors are to provide line-by-line comments/questions (i.e., line 45 page 4) to make it easier for authors to respond.
Reviewers should also organize their comments/questions by type (major and minor). Major recommendations are issues that must be addressed in order for the manuscript to be accepted for publication. Minor recommendations are issues that are not central to the scientific premise of the study such as formatting, style of writing and clarity. The major strengths should also be included which are elements that add to the merits of the paper or strength to the findings. It’s important to not get lost on what the authors did right when critiquing a manuscript. Reviewers should format their comments by section (methods, intro, results, etc.) and include the major and minor recommendations for each section.
If there are reoccurring flaws or errors within a paper, line-item comments can be combined into a summary paragraph. Either way, the most important aspects of formatting peer reviews are specificity and organization so that the comments are useful and not overwhelming for the author.
4) What do you think is a reasonable and appropriate timeline for completing a peer review request?
According to all three editors, the recommended timeline to complete a peer review is within 2 weeks. Dr. Smith added that if a manuscript is within the expertise of the reviewer, but the timing is not good, editorial board members will often provide an extension upon request. She states that as an editor she would rather wait a little longer for a thoughtful review from a content expert than receive a rapid response that lacks detail. For new reviewers, Dr. Lindsey recommends that to make a good impression with editors you should try to complete your reviews within 4-6 days. Fast turnarounds of high-quality reviews will catch the attention of an editor.
5) How much time should someone expect to dedicate to peer review a single manuscript?
The consensus among all three editors is that the time commitment to thoroughly peer review a manuscript may vary due to several factors such as the experience level of the reviewer, the reviewer’s familiarity with the topic, and the complexity of the study. For reference, Dr. Smith states that at minimum a reviewer should expect to commit about 90 minutes to thoroughly review a manuscript, however a more complex manuscript could require a few hours. All three editors acknowledge that as a reviewer becomes more experienced it should generally take less time.
6) How can people start peer-reviewing for your journal?
Dr. Smith: PTJ invites content experts in the field to review manuscripts. The Journal is currently looking at initiatives for adding and/or mentoring new reviewers. In the meantime, interested individuals should contact an Editorial Board Member directly, to discuss their area(s) of expertise and the requirements for reviewing.
Dr. Lindsey: AJP Heart has a junior reviewer category that we actively recruit new reviewers. One of the best ways is to go online and register in the submission program for that journal. The online database is where they select reviewers.
Dr. Collins: Publishing in CPTJ enters the author into the system and once published we tend to activate their accounts as possible reviewers. If someone who has not published in our journal asks to be listed, I will create an account for them to confirm after I have had a chance to review their CV, and read through some of their publications. It is also appropriate for Ph.D. supervisors to invite students and candidates into the review process. My recommendation would be to start with a very narrowly focused area of your expertise. This will allow you to focus on the other aspects of a high-quality review without having to search the literature for the current publications.
Thank you again to our excellent team of editors for their contributions to this blog! We hope this blog will serve as a valuable resource to peer reviewers both new and experienced.
Cardiopulmonary Physical Therapy Journal Blog 