Writing paper reviews

Aug 04 2017 Published by under Uncategorized

I think the best advice I've ever given a trainee about writing reviews of other people's scientific papers is: don't take too much time on it. I know that will seem like heresy around here, and worse, feeding into reviewer 3 syndrome. BUT! if you start from the assumption of good-heart, open mind and commitment to helping others, then this advice makes sense.

My tendency and that of my bestest trainees is to spend too much time, to give too much, to bleed oneself dry. Indeed, we are more than committed, we are dedicated to helping others. And writing a good review, full of suggestions, showing where a paper works and where it doesn't is one of the best and most important ways of helping. I do not doubt that.

Nor am I suggesting that anyone should whip through a paper and pull from the folder of stock critiques ("methods are too detailed" "can't follow the discussion"). One needs to provide useful and thoughtful advice to the authors.

But one can piss one's life away helping others, writing out the details of what one would have done or said to make this better. As in all too many things there is a balance.

I just got back a paper to review for the third time. The first time I did not follow my advice. The topic and data were Important. The potential To Make A Difference was there. But, the paper was so opaque, so obscure, I could not follow what was going on. There writer was a relatively senior person and a physician and actually knows what research is. Although the reviews for this journal are double-blind, it was glaringly clear who wrote this (heck the acknowledgement of IRB approval had initials of PI attached to it).

What was wrong? New acronyms for existing things. Complex, convoluted statistics when simple ones would be appropriate. New meta-variables, for example calculating a measure of heart function based on rate and intensity and what you ate for breakfast when just testing heart-rate would have sufficed. Writing that read like it was originally in English, translated to German with Google and then back to English, so that the verbs at the end of sentences all piled up were. Figures that I coldn't see the points or determine what the variation intervals were, let alone whether the were SD, SE, IQR or something entierely different. I couldn't really tell what the conclusions were because all the other stuff got in the way. I also had problems with the scientific justification and context and, in NIH-speak, I could not figure out the premise of the work. It wasn't even that they only cited their own stuff, it was that there was no acknowledgement of other perspectives, other work that might impact on how they thought about these results.

The first review pointed out each thing, explaining why it kept me from understanding the paper. The review was too long and took too much of my time. But the letter that went out from the editor suggested that the comments were valid and that the authors needed to address them.

I got a revision back that was the height of absurdity. It said "thank you to reviewer #2 for the insightful comments. We have made the changes requested". Then it went through every comment and argued with me about it. Some of their replies made sense and some of them did not. But they changed nothing that I suggested, except redrafting the figures. That made my second review easier. I went through and pointed out the same problems. And said that since I had now read the paper several times, and was still unsure about what their specific results were, and what they thought it meant, that I had to respectfully suggest that reconsider my comments rather than just arguing with them.

The third version came back changed, with nearly everything much much better. One thing they did not do, though. My last review:

I appreciate the authors' continued engagement in my comments. The change in terminology facilitates the reading of the manuscript. The changes in analysis and presentation of results are acceptable and make the manuscript much easier to follow.

My only remaining concern concerns citations & interaction with the literature. The response to the review contains 12 citations all of which are from one group. There are MANY more perspectives as to what is a ____ and these authors seem reluctant to acknowledge the work of people outside of this one group, including the work in the [related] literature which is highly relevant to this paper, since the discussion claims implications to that field. This is not just a citation problem, it is an understanding of what other work is being done. Again, the problem is NOT so much the lack of other citations, it is the lack of understanding the literature outside a very small circle. The two groups cited here are not the only workers of ______ function.

However, this disagreement falls into the realm of scientific discourse, should not prevent publication.

I do not want to ever read this paper again.


2 responses so far

  • wally says:

    Can I ask a question related to your comment about self-citation? In the first paper out of my postdoc (which is now in press, so this isn't a current issue - I'm just curious about it), a reviewer wrote that our self-citations seemed "wired." What I took that to mean is that they felt we cited our own work far too much. The tricky thing, however, is that this was a study at the nexus of three research areas, and I had the leading experts in those three areas on the paper - so we of course had to cite their work. I cited others' work as well, but generally these are underresearched areas, so it's not like there was a bevy of research upon which to draw. Are there ways to address something like this up front so it doesn't look like you are just being self-promotional?

    • potnia theron says:

      Firstly, I think that's a weird comment. Wired can mean so many different things, why not just say exactly what you mean? Does this mean "automatic" or "reflexive"? Or wired, like hopped-up and speedy?

      I think if one's citations *are* the appropriate cites, then there is not a problem. One way to dissipate the criticism of too much self-citation is to make sure you cite others as appropriate. If, as you say, there does not exist a bevy of research to draw on, make sure you do cite what does exist. When you set up the justification and context (as opposed to background - an important distinction, particularly in grant proposal writing), as appropriate, emphasize the other work.

      Finally, congrats on that first paper.

Leave a Reply