These posts are prompted by the instructions I'm getting for being on study section. Firstly, I think the advice NIH is giving to reviewers is good. I don’t agree with all of it. You may not agree with all of it. But, having a set of instructions, a set of guidelines, makes the review process more equitable and more objective. I think the goal of SS is to be as transparent as possible, and as even handed as possible. My memories of the variation that existed when I sat on study section in 90s was that there could be that objectivity, but there was a lot of rewarding the bigdogs and hyper-criticism of the young ‘uns. So I perceive the adjusting and calibrating that is happening now, whilst I am reviewing, to be a good thing.
Secondly, here is more of the advice NIH now gives to reviewers. There wording is generally available on line. My comments are, shall we say, not.
So what is in a review? There are six sections that contribute to the score (and others, such as evaluating human subject distribution that are not). The first, as in at the top of the document, is Overall Impact, which is the summary overall assessment. I write this last, after I’ve done the sub-sections. It is the basis of the score that a reviewer gives. The other five are the five areas/criteria (in order on the review page): Significance, Investigator(s), Innovation, Approach, Environment. The links in those criteria lead to text that I will put in subsequent posts.
Today, though, I want to talk about Overall Impact. The words the reviewer writes will go into the overall assessment paragraph at the beginning of of the review you receive. You will also get the specific comments on the 5 areas, as well as scores. One point, though, is that the other members of the study section (not the ones assigned to your proposal) may not see those bits, if they chose not to. But many will read the "overall impact" part of the review.
Here are the instructions about overall impact:
Overall Impact: What is the likelihood of the research to exert a sustained, powerful influence on the research field?
Write a paragraph supporting the overall impact score that should contain the following:
- Introduce the general objective of the project in one sentence to orient reader.
- State the level of impact the application is likely to have and why (what is the major contribution/advance to be gained?).
- Identify what the major score-driving factors were for you.
- Explain how you balanced/combined/weighted the various criteria in the overall impact score.
This may be the MOST important part of your review. It comes first but is based upon all the individual pieces in your completed critique template.
Here is what NIH says is NOT a good review and their reasons why:
The proposal is overly ambitious. There are design flaws. Significance is questionable.The PI’s productivity is low.
NIH Comment on this review: Lacks detail. Hard to interpret.
More of not good review:
(1) In Aim 1, the PI plans to generate XXreagents and test them in the YYsystem. In Aim 2,XX will be usedto explorethe ZZ pathway. ThenAim 3 will examine XX as potentialtreatmentsfor ABC disease. If successful, thisresearch could significantly impact the field.
(2) Only moderate enthusiasm was generated for this application. Strengths noted were the PI and team, excellent environment, state-of-the-art methodologies, and potential importance of the work to understanding XX. Weaknesses were the over ambitious nature, lack of experimental details, some confusing preliminary data, and concern about the choice of YY to be used. Altogether, this project will have a moderate impact on the field.
NIH Comment on this review:
(1) Just a listing of strengths-weaknesses without context.
(2) Only the major score-driving concerns should be listed in the Overall Impact along with the reasons why they are major and how they drove the final score. Just a rehash of the aims. No evaluation of the impact and what the score-driving issues were.
Some of these are "stock NIH critiques". What is important is that NIH is recognizing that stock critiques are useless and trying to push reviewers to do what is right and helpful. For you, the writer, this suggests that when you read something as a critique in the Overall Impact, that is something you need to consider and work on. One of the very hardest things to do is to figure out how to address concerns in the review when you resubmit. Sometimes its just writing/clarification of what you meant that they didn't get. Sometimes, however, you need to do something more substantive (and not just more preliminary data), but rethink how you propose to test hypotheses. This section, done properly, can guide you.
Here is what NIH considers an effective review:
This proposal addresses avery significant issue in the field of XX and overall impact is high because the research is likely to provide the link between two seemingly contradictory outcomes that have stymied recent advancements in this area. The project is not technically innovative, but this is not considered a weakness because the focus on XX is important and the methods are appropriate and rigorous. The approach has some very strong aspects such as X and Y. Most of the weaknesses were minor. However, one weakness created some concern. The weakness was XX. The problem with this is that they make an assumption about ZZ that does not seem to be supported by adequate data. The investigator is well-trained in X, Y, and Z and the collaboration with Drs. A and B, who will bring strengths of C and D, increases the likelihood of a successful outcome. In conclusion, despite the weakness in the approach, the potential overall impact of this project remains high because it will advance understanding of the mechanisms underlying the relationship between XX and YY and test new methods that will be useful in both basic and clinical research areas.
Uses clear and specific language to explain points. Highlights only the main score-drivers. Any minor points are left in the criterion sections.
Indicates importance of strengths and seriousness of weaknesses when appropriate.
Explains how the strengths and weaknesses were balanced to arrive at the final score.
When I get pink sheets back for a not-close-to-fundable-proposal, it hurts. I can't always read them immediately. It helps that the scores come well before the reviews. But, a good review is valuable. It's why I am giving you all this. It may not apply directly to your writing of a proposal, although it does help clarify what the reviewers are looking for in the proposal. This will be especially clear in the next few posts, as I dissect the instructions for the five specific sections. Understanding what the reviewers are looking for is something that you can keep that in your head as you write. You can reread your proposal and evaluate what a reviewer might say. And when you get those ugly and hurtful Pink Sheets back, you can try and interpret what is being said to you.