Small part of reviewing grants: seeing all the scores

May 29 2018 Published by under Uncategorized

This does not reach the level of hitting refresh to see your score, but it is in the same phylum. When you submit scores for NIH reviews, you post them on the Commons website. There is a time for submitting and then comes a time for reading (the reviews). And in this time for reading one finds out how close are all the scores for each proposal.

When I was a sprout, and there was no intertubz, one had to wait for the "reading of the scores" in study section, where grants were reviewed in alphabetical order. Now there's an unintentional bias:  last names that start with X or Y or Z. or even W or T. Only then one discovered whether the reviewers reached consensus (which at the time was something greatly desired by NIH, now not so much).

One of the best things about being older is that I now have more confidence in my reviews. But as I told my new postdoc, yes, I still at my age have some imposter syndrome. And one of the worse exposés of one's IS is when you feel that you've done the review wrong wrong wrong relative to the Big Dogs on study section. Being too low (good score) means missing some critical flaw that perhaps one was just not smart enough to see. Being too high (bad score) means having no appreciation for what is important in the field (or in the olden dayes, why this Other Big Dog should be funded despite writing a dreadful proposal).

So, indeed, this morning, which opened read phase for study section that meets next week, I did open commons first thing and read the other reviews next thing. I will admit to some small relief that there is only one proposal with wildly disparate scores, and most are at least in the same family if not genus.

Of course consensus may reflect bias all around.

 

update: lively discussion at the tweets on this

8 responses so far

  • A Salty Scientist says:

    Tangentially related to this post, but has been bothering me re:bias on the Twitters: for those who have served on study section, approximately what percentage of grants are from PIs who you know professionally (e.g. are familiar with their work).

    • potnia theron says:

      I think this is a function of age and willingness to go to meetings. I know, have met, have shook hands with 2 of the 10 I got to review this time. But there have been times when its been 0/6 but also as high as 4/8.

    • Dnaman says:

      Highly dependent on the study section.

      Some are very narrow, where all the applicants and reviewers go to the same conferences.

      Some are very broad, where none of the reviewers ever heard of one another before the meeting.

  • DrugMonkey says:

    A lot? I would hesitate to apply a specific number.

  • Dnaman says:

    Highly dependent on the study section.

    Some are very narrow, where all the applicants and reviewers go to the same conferences.

    Some are very broad, where none of the reviewers ever heard of one another before the meeting.

  • A Salty Scientist says:

    My guess was that it would be rare for reviewers to not know at least some subset of applicants during *normal* NIH review. This could be a source of either implicit or explicit bias (positive or negative). Persons from underrepresented groups often have a harder time networking, which could be a net disadvantage during peer review (e.g. unknown applicants may have a higher bar to clear on preliminary data regarding feasibility). If this is a source of bias, it would not be captured using a review process consisting of *fake* PI names.

  • Another Anon says:

    Potnia! Your link to twitter profile doesn't link to your twitter (in contrast to your rss feed button, which does).

Leave a Reply