Reviewing Grants: Intro on what the heck *happens* to my grant

Sep 27 2017 Published by under Uncategorized

NIH really really tries for transparency. Almost everything I'm saying exists somewhere on the NIH website. The issue is the "where" part of that.

I've got lots of posts elsewhere on the writing part. This is what happens, briefly after submitting. Firstly, there are two different routes: going to the CSR -Center for Scientific Review, vs. responding to a particular request for grants. The latter means it is designated to an initial IC (this jargon confusing? See here and here). I'm going to talk, for now, about going to CSR.

At CSR it gets assigned to a IRG (Initial review group) or Study Section. It also gets assigned to an IC. You can request, and NIH tries to honor requests, but there is a balancing act. The head of the IRG, who is a SRO (Science  Review Officer) gets some say in what is assigned to his/her sie's group.

Then the SRO assigns proposals to the people who sit (and seldom stand) on the IRG. The SRO usually has another list of ad hoc reviewers. I've been on IRGs where more than 50% ere ad hoc, which to me indicates the SRO was at least trying to get appropriate expertise.

At this point, the reviewers, the study section members, get a list of proposals and all of the significant personnel you listed. This is because NIH takes Conflict of Interest very seriously. At this point, proposals are assigned to reviewers. And the barrage of information that I'm putting in other posts starts coming to us. There are 2-4 reviewers, but usually 3 per proposal. [see here for info about serving on a study section]

There is a due date for reviews, usually about 5 days before SS meets. When (and not before) you have uploaded your reviews, you get to see what the other reviewers have written.

Some brief calculations for you: The Harsh Reality of study section. There are 15-30 people, all of whom are overworked, and reviewing because they think it’s the Right Thing To Do. This group may have 60-100 proposals assigned by CSR to this meeting of the SS. Three reviewers per grant = 180 to 300 grant-reviewer assignments. Which means that everyone has 6-10 grants to review. It is a lot of work.

We will look at the other reviews before we leave town for the meeting. In fact, we are encouraged to, and to think about what they other reviewer has said. Do I (or almost anyone) do this for all 60-100 proposals? Nope. I do it for the 10 I have reviewed. I do read while I am in study section, and listen to the reviewers, but that's yet another post.

When we get to study section, the chair will ask for “preliminary scores” and we (three or four reviewers) will give those overall scores. We can say something different if one of the other reviewers has persuaded us in their written & uploaded review.

This is the "total impact score". There are five sub-areas. More about "total impact" vs. five areas in next post, but also see here and here. But what is relevant here, if I’ve given you a “2” on Investigators and a “6” on innovation no one may actually know those values. But! After giving the scores, we start with the primary reviewer explaining what the proposal is about, and why they gave the scores they did, and what aspects of the five areas informed that score. Then the second reviewer will say something like “I agree with Dr. HuffnPuff, except that I think the significance was higher for these 16 reasons.”

The review you get (Pink Sheets) will include everything that was written by the reviewers. Reviewers get a chance post-meeting to edit their reviews (and are encouraged to do so). Someone, usually the primary reviewer, writes up something that captures what happened in the discussion.  You will get the specific comments on the 5 areas, identified by area, as well as scores. The important fact here, though, is that the study section may not see them, if they chose not to.

Why wouldn’t they? See above about amount of work that goes into the review. I don't know exactly how much I put into each review. And it varies with proposal. But its more than 4-5 hrs per. Times 10. At least a week of work.

With this introduction, next posts will be back to the instructions NIH is giving to its reviewers.

 

 

8 responses so far

  • girlparts says:

    Any suggestions on how to target your grant to the study section you would like? On my last submission, I named a particular study section in the new form, and wrote in expertise areas that fit that study sections..... and they told me to stick it. In this case, both study sections had expertise in the area we were studying, but one is basic scientist dominated, and the other is clinical science dominated.

    • potnia theron says:

      Sorry about that. There are lots of human-based decisions that go on here, and they are hardly perfect. Sometimes one study section has too many proposals and they assign the ones on the margin to another. Things that you can do is make sure your title contains the right words, maybe even "clinical" (if you want the clinical section). In the past you could contact the CSR SRO (assuming it is a CSR study section) and ask for help. I would suggest emailing the SRO and seeing if you can get it changed. If it is for the upcoming cycle (ie submitted in June/July for Sept/Oct review) its probably too late to change. But please let me know what happens.

      Here is a link that might be useful: https://nexus.od.nih.gov/all/2016/04/30/use-the-assignment-request-form-in-forms-d-to-provide-nih-peer-review-referral-information/

      • girlparts says:

        I did contact the SRO afterwards, and they refused to move it. Officially, the basic section works on stage 1 of bunny hopping , and the clinical section works on later stages. We were working on stage 1, but claiming implications for a disease that occurs in stage 2. The SRO argued back that it was clearly about the disease and therefore belonged in the clinical section. I was surprised that they felt so strongly about it; I had assumed it wasn't that hard to request a study section. Now that I'm trying to re-target a completely different proposal, I'm wondering how to do better.

        • potnia theron says:

          A couple of things: you can talk to SRO's before you submit. Ask: is this something that would fall in your area? You can put reduce the clinical emphasis in the abstract (but may be not in significance).

          Your situation is the sword that many basic scientists dance upon: it needs to be clinically relevant, but not so much that it goes to clinicians for review. If there was pressure to get things into the clinical section (ie relative number of proposals), anything that can be moved, would.

  • Luminiferous aether says:

    If a member and applicant have a well-established COI, does the conflicted member still get to see the applicant's proposal, scores and uploaded reviews in the CSR system (I know that they have to step out during discussion of the proposal at SS).

    • potnia theron says:

      No. We are asked about COI, and those applications are not longer accessible electronically - proposal, reviews, scores, the whole thing. In fact, they don't make the proposals accessible to reviewers online until AFTER the COI forms are processed. Because currently, most study sections review in score order, best first, you can guess how a proposal stands overall by the order. But that's it.

Leave a Reply