Peer Review from the Other Side

Like most junior scholars, my experience with academic journal peer review has quickly migrated from the submission end to the reviewing end. It’s a big step to move into this gatekeeper role, and one with no instruction manual. Journals usually provide a vague referee report to help reviewers structure comments systematically, but for the most part our knowledge of what a review looks like comes from reviews we’ve received from own work. And I am sure my experience with the reviews I’ve received mirrors that of others: some are helpful, some are disappointing, and some are downright bizarre.

By this point, I’ve reviewed dozens of manuscripts for 15 (and counting) different journals (not to mention a smattering of conference papers). And while I’ve dutifully judged these submissions, I’ve never stopped to codify how it is I think about reviews until I was asked to answer some questions to help with a research guide. I found the exercise to be personally enlightening in galvanizing my thought process, but I also wanted to share it with others who may be on either end of peer review. The questions are not mine, so I can only be blamed for the answers. Feel free to add comments about your own experiences.

What makes you decide to accept an invitation for peer reviewing, other than your availability? Do you rely on abstracts and keywords?

As a junior scholar, I try to accommodate every review request to get involved in the field. I feel indebted to the many reviewers that I have helped me improve my own work over the years. Participating in peer reviewing is a way of giving back to the field.

What do you look for in an article, generally speaking?

All scholarly research articles are really arguments that the author has identified a useful research question that fulfills a gap somewhere in existing literature, and that it does so productively and rigorously. A submission may be soundly assembled, but have no value in advancing research. Conversely, a submission may promote an interesting new path but in such a woeful manner as to not meet the basic requirements of scholarly work. The job of a reviewer is to assess whether this submission can add something new and then to provide direction for making this happen.

What makes you reject an article? In other words, what basic criteria does an article need to reply to in order not to be rejected?

I confront every new manuscript critically yet hopeful that it will be an acceptable piece of scholarship. However, I end up rejecting the vast majority of submissions I review. Rejections occur for a variety of reasons, including a lack of depth in the writing, a lack of clarity about what the study is doing, poor use of existing theories or scholarly concepts, poorly executed methods, and lackluster findings. No one item earns a rejection, but rather I ask if problems can be fixed or if they are too severe or too numerous to be revised. When rejecting a paper, I always do two things. First, I question myself to make sure that I am not just rejecting the paper because of my own preferences; every paper should be evaluated for its own merit. Second, I make sure to write useful comments for authors so that they may find a way forward with future work. Having a journal submission rejected may be common for academics, but it is always painful and potentially deflating. As a reviewer, I try to mitigate this effect.

How do you decide between minor and major revision? 

There is no set standard, but one that I try to apply asks how clear the solution is to whatever the problem is preventing acceptance. If it is clear, from my reading anyway, that an author needs to change x, y, and z and what needs to be done is straightforward (even if time-consuming), than that is a minor revision. By contrast, a major revision occurs when I identify a problematic aspect of a manuscript–something that is not quite right–but that I am also unable to see how it gets fixed. In this case, I ask the author to go back and retry a new approach in the hopes that this will alleviate the problem. Of course, there are also major revisions that occur because of a multitude of small problems rather than any major issue.

How do you decide between major revision and reject?

Deciding between a major revision and a rejection requires a decision of whether to trust that the author is capable of making the corrections needed for acceptance. No one wants an author to waste time on a revision that would still fall far short. At the same time, if the topic is important and the paper promising, I want to give the author a chance to fix up a paper for the benefit of advancing the field. Of course, often an author confronted with a major revision that suggests new directions will simply take their submission elsewhere. As a scholar, I’ve had this happen where rather than revise a paper in an entirely new–and somewhat uncomfortable direction–I’ve gone to another journal and been accepted.

How important is grammar and writing style in your assessment? Would you reject a paper that was poorly written, even if it contained interesting information?

With english becoming the lingua franca of increasingly global academic scholarship, non-native speakers are forced to submit their work in english-only journals. I keep this in mind whenever I review an article, noting that the editor or a copy editor can help spruce up the language of a submission. I really think it is unfair to discriminate against someone’s scholarship simply because of language. That said, it can be extremely distracting to read a poorly written manuscript. Indeed, it is also difficult for the author to make sophisticated points without a mastery of the language. But nonetheless if the material is sound save for some language issues, I would not hold it against an author.

If you feel you’re not the right person to review an article, after accepting an invitation, would you still try to review it or do you pull out?

There are two reasons why I would decline to review an article: either it is a topic that I lack knowledge of or it is a method that I am unable to assess. If there is no fit on the topic, I inform the editor that I cannot do the submission justice and often make suggestions of other scholars I know who would be a better fit and ask that the editor consider me for future submissions more aligned with my own areas of interest. When it comes to a method that I cannot properly assess, I may still review a submission if I am quite comfortable with the topic. However, I would make it abundantly clear to the editor what my shortcoming is to ensure that another reviewer can properly account for these areas.

From a practical perspective, how do you assess the paper? Do you make a list of errors and corrections (including page and paragraph numbers), for example, or do you prefer giving general comments?

When peer reviewing, I try to keep track of my own thoughts by jotting them in the margins of the manuscript as I read. This way, I can quickly make note of a comment and then return to the paper with my full attention. At the end of reading a submission, I go back through and examine my comments looking for recurring criticisms in order to craft a review. When I write a review, I focus on explaining in detail general issues that have arisen, saving only a small amount of room at the end for specific errors and corrections.

When you are invited by a journal you are not familiar with, do you assess the paper primarily according to your own criteria for academic journal publishing, or do you also consider the journal’s specific aims and scopes, and if so, to what extent?

At this point, I’ve reviewed for fifteen different journals located in several different countries. For the most part, I judge each submission against my own standards for scholarly research. But I also stress specificity and clarity in my written reviews, which not only aids the author in making revisions but also helps the editor understand my qualms so that she may decide if my criteria are a good fit for the journal. I’ve actually had a paper that I recommend rejection for get accepted to a journal. I respect this decision by the editor much more than an editor merely parroting back what the reviewers said with no interest in how they arrived at their conclusions.


About Matt Carlson

Associate Professor of Communication Hubbard School of Journalism and Mass Communication University of Minnesota
This entry was posted in Uncategorized and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s