I think the best advice I've ever given a trainee about writing reviews of other people's scientific papers is: don't take too much time on it. I know that will seem like heresy around here, and worse, feeding into reviewer 3 syndrome. BUT! if you start from the assumption of good-heart, open mind and commitment to helping others, then this advice makes sense.
My tendency and that of my bestest trainees is to spend too much time, to give too much, to bleed oneself dry. Indeed, we are more than committed, we are dedicated to helping others. And writing a good review, full of suggestions, showing where a paper works and where it doesn't is one of the best and most important ways of helping. I do not doubt that.
Nor am I suggesting that anyone should whip through a paper and pull from the folder of stock critiques ("methods are too detailed" "can't follow the discussion"). One needs to provide useful and thoughtful advice to the authors.
But one can piss one's life away helping others, writing out the details of what one would have done or said to make this better. As in all too many things there is a balance.
I just got back a paper to review for the third time. The first time I did not follow my advice. The topic and data were Important. The potential To Make A Difference was there. But, the paper was so opaque, so obscure, I could not follow what was going on. There writer was a relatively senior person and a physician and actually knows what research is. Although the reviews for this journal are double-blind, it was glaringly clear who wrote this (heck the acknowledgement of IRB approval had initials of PI attached to it).
What was wrong? New acronyms for existing things. Complex, convoluted statistics when simple ones would be appropriate. New meta-variables, for example calculating a measure of heart function based on rate and intensity and what you ate for breakfast when just testing heart-rate would have sufficed. Writing that read like it was originally in English, translated to German with Google and then back to English, so that the verbs at the end of sentences all piled up were. Figures that I coldn't see the points or determine what the variation intervals were, let alone whether the were SD, SE, IQR or something entierely different. I couldn't really tell what the conclusions were because all the other stuff got in the way. I also had problems with the scientific justification and context and, in NIH-speak, I could not figure out the premise of the work. It wasn't even that they only cited their own stuff, it was that there was no acknowledgement of other perspectives, other work that might impact on how they thought about these results.
The first review pointed out each thing, explaining why it kept me from understanding the paper. The review was too long and took too much of my time. But the letter that went out from the editor suggested that the comments were valid and that the authors needed to address them.
I got a revision back that was the height of absurdity. It said "thank you to reviewer #2 for the insightful comments. We have made the changes requested". Then it went through every comment and argued with me about it. Some of their replies made sense and some of them did not. But they changed nothing that I suggested, except redrafting the figures. That made my second review easier. I went through and pointed out the same problems. And said that since I had now read the paper several times, and was still unsure about what their specific results were, and what they thought it meant, that I had to respectfully suggest that reconsider my comments rather than just arguing with them.
The third version came back changed, with nearly everything much much better. One thing they did not do, though. My last review:
I appreciate the authors' continued engagement in my comments. The change in terminology facilitates the reading of the manuscript. The changes in analysis and presentation of results are acceptable and make the manuscript much easier to follow.
My only remaining concern concerns citations & interaction with the literature. The response to the review contains 12 citations all of which are from one group. There are MANY more perspectives as to what is a ____ and these authors seem reluctant to acknowledge the work of people outside of this one group, including the work in the [related] literature which is highly relevant to this paper, since the discussion claims implications to that field. This is not just a citation problem, it is an understanding of what other work is being done. Again, the problem is NOT so much the lack of other citations, it is the lack of understanding the literature outside a very small circle. The two groups cited here are not the only workers of ______ function.
However, this disagreement falls into the realm of scientific discourse, should not prevent publication.
I do not want to ever read this paper again.