Looking at Pink Sheets / NIH reviews (part 1)

Nov 13 2017 Published by under Uncategorized

A former trainee, now working with in a medical center/research group/structured environment, asked me to take a look at the reviews for their recently triaged proposal. While I have permission to quote these, I've done everything I can to cover up identities.

There are a number of interesting points worth learning from. In particular, a number of things that were constant across reviews. I have put the reviewer comments in italic to differentiate from my thoughts.  I will  try and organize these into categories.

This is where this post got long, so just one category today:

Category: Obvious things that you should pay attention to ... or ... the reviewers are saying exactly what they mean. This is reviewer 1:

  • Major weaknesses are the dependence of Aim 3 on success in Aims 1 and 2: This is obvious. If you need the results from aim 1 to know what you will do in aim 2, what happens if aim 1 fails? This is a difficult thing, as one is usually trying to build a series of steps. And frequently, what we find in the beginning changes what we find later on. But Aims need to be independent. One way to cope with this is to rearrange what is in each aim so that something you might vary within an aim, becomes the difference between the aims.
  • [major weakness includes]: a lack of scientific rigor. Guys, NIH is serious about this premise and rigor stuff. We were told before and at study section to explicitly mention these words in our reviews. See here. and here. Many of the specific comments in this review are about rigor.
  • Although the clinical need is clear ... the weaknesses outweigh the strengths of the proposal. Ouch. Being clinically relevant is not enough. Being translational is not enough. The proposal must stand on its research merits.
  • For Personnel, I've seen (and written) versions of the following comments over and over. These are all about picking a project that is feasible. Feasible is not an explicit criterion, but it comes up in study section discussion all the time:
    • This team does not have a long history of collaboration
    • The ability of the PI to direct such a large project is not proven.
    • The proposal would be improved by an explicit plan for regular communication and coordination of the efforts in the diverse labs involved.

Here is a list of simple things that should be in every proposal. These are all about rigor of the proposed project. This is reviewer 2:

  • What are controls?
  • Expected results and interpretation are not provided in each aim.
  • Sample sizes are very small. And  the numbers in the human subjects table do not match either those in the research design for Aim 1 or Aim 2. 
  • Sex as a biological variable is not considered.

Which brings us to this reviewer's conclusion for Approach Weaknesses:

  • Thus, scientific rigor and reproducibility are a concern.

Reviewer three continues with this theme. When all three reviewers hit the same point, in this case lack of rigor, one must stop and reconsider the proposal.

The proposal suffers from numerous problems ranging from a lack of clarity, superficial background, and contradictory statements. Details of the actual research are lacking as is identification of who will actually carry out the research

At the end of one of the reviews are the following famous words. Some people see this as a Stock Critique. It can be. But in this case it was the reviewer summarizing why this proposal got "7's" in Approach.

The weaknesses reduce the likelihood that the project will have a sustained, powerful influence in the field.

No responses yet

Leave a Reply