Thoughts on reviewing NIH proposals: What is the difference between a 2.0 and 3.0 in initial score?

Feb 10 2018 Published by under grantsmanship, Uncategorized

There are lots of posts (some mine, some from others, more worthy correspondents) about the current state of NIH scoring and its relationship to funding.

The general and consistent message is that, yes, funding is hard to get. You need to put in the best possible proposal you can. You need to pay attention to grantsmanship issues, because that can help. But I want to move beyond that.

I've often commented that sometimes the difference between a discussed but not funded and a funded proposal is almost random. That this is one of the times grantsmanship matters, and what I consider one of the most important meta-rules: make the reviewer your advocate.

So study section time is coming up. I got a set of proposals to review, and I'm done. My reviews are posted, and many of colleagues have posted too. This produced a lot of reviews for me to look at, and I was seeing them in the light of recent twitter discussion.

I haven't seen a score better than 2. But I also haven't seen any 9's, or even anything >8.1. This is, in part, because we are explicitly told to balance our distribution, and make 5 the mean/median. Start with a 5, and go up or down from there.

Looking at the 2-4 range might be helpful. These are proposals that will be discussed. The difference between a 2 and a 4 is now the difference in getting funded. What are the words used that distinguish between the 2s and the 3s and 4s?

Here is something I saw more than once:

"Overall, while hopping disorders in elderly rabbits is a topic of importance, the work is viewed as incremental in nature and not that particularly innovative"

I did not review this proposal. This was a study that received a "2" in approach, but a "6" in innovation. A good reviewer may have more detail on why it is "not particularly innovative". I've seen things like:

These results have been established for adult bunnies, and the only difference in this proposal is that the study population is elderly bunnies.

The significance could be very high: we may want or need to know this for elderly bunnies. But the innovation is not. Here is another critique I've seen concerning innovation:

The proposed techniques and approaches have been used by this PI and study team for many years, thus not particularly new or innovative, other than being used for this project in elderly bunnies.

I am not sure this is particularly helpful, as most PI's use the same basic techniques. This critique produces the cry: "what am I supposed to do? this is what I do". There is the sentiment expressed that "reviewers are just looking  for problems". I do not think this is true. Instructions to reviewers are to  start with a "5" in every category and move up or down. I have read the complaints of people who say "I got a 3 or a 4 but there were no weaknesses."   A 3 or 4 may have no weaknesses, just not enough strength to boost it to a 2. So how then you ask, "am I supposed to improve this and get funded?".

To answer that, let's start with the NIH wording on innovation for R01s:

Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

And we are back to the discussion of last month, and last year, and honestly, last decade. NIH reviewers are not in the business of making suggestions on how to improve your proposal. You may want that. You may feel you are owed that. You may think that it is unfair not to give you that kind of help. And of course, you may have any feelings you might want to have. But those feelings will not change reality. The job, the role, the assignment to reviewers, from NIH, is to evaluate proposals and provide justification for the scores that they give.

The view that something isn't innovative because it does nothing beyond change the population is sufficient for NIH standards, but tremendously frustrating to the PI applicant. The reviewers are not supposed to tell you how to make it innovative, only judge whether it is or not.

Back in the olden dayes, when grants were 25 pages long, reviews unstructured and everyone wore suits to study section, there were unlimited resubmissions. One (but there were others) of the reasons NIH decided to go to first 2 resubs (3 total) and eventually 1 resubmission was the view that "reviewers were writing the proposals" and telling PI's what they needed to do.

In fact, I think not telling a  PI how to make a project innovative is A Good Thing. It opens up room for creativity and insight and change. If reviewers and study sections said "do this", well of course you would. And then the reviewers would not so much be evaluating your ideas and projects, but proposing their own.

The best place to go to figure out how to change your proposal are those questions and guidelines that NIH gives the reviewers (here!). Make it possible for the reviewers to answer yes! yes! yes! to those questions. The proposals that have turned me into their advocate are proposals that I want to make sure my reasoning about why this is A Damn Fine Proposal is clear, and upfront and persuasive as possible to my fellow reviewers.


Tags: ,

No responses yet

Leave a Reply