A tweet from @doc_becca set things off last night:
All I am asking for is that there is a mechanism in place to keep scores from getting worse. Such a colossal waste of time.
— Dr Becca, PhD (@doc_becca) June 21, 2017
This is going to have to be a quick post. I've got too many urgent things piling up around me. This post isn't urgent, but it is important. There will, without question, be mistakes. I trust my loyal readers to point out, preferably with glee and schadenfreude where I'm wrong.
Another caveat: This is not about Doc Becca. I count her as a friend. I am upset about her situation and where she is and the stupidity of her university. I've been following her for years, since she was stumbling towards the tenure track. I love her writing, and from what I know of her career IRL, she deserves tenure, based on her science and productivity.
Again, this is not about her. This is about all the things that got said about study section last night. Some of this is my opinion, but lots of it is about how study sections (SS) work.
So on to content, and not in a necessarily good order:
Firstly, remember reviews are two steps: Study Section and Council. Study section reviews proposals for scientific merit and assigns a score. This score, later on, by NIH staff, will be normalized into a percentage based on all the scores from the current and the last two iterations of this section. At SS, the proposal may be discussed (if it s in roughly top 50%) or not (triaged, in bottom 50%). Reviewers, any reviewer on the SS, may ask for a triaged proposal to be discussed, before SS, or even at SS. I have done this. It is not common, but not rare, either.
SS does not decide funding. Council does, with much input from NIH staff. The person who runs SS (a Scientific Review Officer) is not part of the team that decides funding or ranking of grants to be funded. SRO's usually work for OER (office extramural research), whereas staff PO's (Program Officers) work for individual IC's. IC's decide funding. There is variation in the score that is funded, and there is variation in who, at what score, gets funded. IC's do this because they have programmatic priorities. If Bunny Hopping is in this year, a worse percentile may get picked up. Something that is outside the "main mission" of an IC may not get funded.
Secondly, (and we are now on our third espresso of the morning): the way that SSs work is not a big mystery. Junior people can do a round on SS to learn how. I've got some posts on that, and will try to dig them up for you. NIH has a program for this and it is very valuable. The most grantsmanship I learned was when I sat on SS.. So now, stuff about how SS work:
Who decides who reviews which proposals? The SRO does. They have a miserable job here, and no time to worry about screwing you. Truly. They have a list of reviewers, which may not include sufficient expertise. They have to go begging for reviewers. Outside reviewers. I've been to reviews where I've been an outside person and there have been more outside people then standing members, because of the range of proposals that come in.
Sometimes outside reviewers don't go to the meeting, and call in. They haven't heard the other reviews, and they are not "calibrated" to the section. Most outside people are aware of this and defer to the ones who are there. But not always. This is another source of variation. But not everyone can drop everything for 2-3 days and go to Washington. You can say "no phone reviews" but that may mean worse reviewers. What's worse? Someone more removed from the area of the proposal, who has no appreciation of either the premise or the design.
People serve on SS for 3 or 4 years, and technically are supposed to be at all of them. Some SS have a program where you do an extra year and come 2 meetings of 3 each year. I was told that option was so over-subscribed at the one I'm joining, that its not an option for now. But, people join, people leave. Maybe the proposal had an ad hoc to start with, and that ad hoc can't do it again. You may not get the same reviewers, you may have 2 of the same, or 1 of the same, or none. There is not a set rule here. BUT! when a reviewer gets a new to them proposal that has been reviewed before, they get the entire summary sheets that the PI received. IME, reviewers read these and consider them. (but more on re-review below).
(BTW: how does your proposal go to a particular section? Thats another post, but in short: you can request, there are key words that help determine, and people who do this).
SRO's have to get expertise, but they also cannot give any reviewer significantly more than others. What is more? Varies from SS to SS. For some, its 4 or 5. But, I've been on ones where I've got 8 or 9 and thats standard for the sitting or standing members. (I sit, because I'm old, and standing hurts my back. This is a joke, do not read anything into it). It's a lot of work for everyone, and you get paid squat for doing it, and it's one of those things you do. I don't think anyone relishes the power involved. Ok, maybe there are a few antediluvian bigdogs who do. I don't. I just try to do the best damn job I can.
IN MY EXPERIENCE: SS members care. They work hard. They are obsessively concerned with being fair, and just and right. They are sensitive to the PI, and take "Investigator" criteria seriously. No one is out to screw you. But, of course, they are human, and have biases and have a lot to do. Sometimes they think proposals are bad. Sometimes they get irritated with a proposal partway through (Make the reviewer your ally, your advocate). But for the most part, the reviews that I've seen, even the ones that don't get discussed, reviewers are capable of partitioning their perspectives. They can find both good and bad in a proposal. They try to balance those things, and realistically evaluate their relative importance.
Finally, a bit about some of things that got said last night.
Scores going up and down? As much as I want Doc Becca to get funded, I do not see how one can be protected against a dropping score. The reviewers have a different proposal. Maybe the proposal is worse, to one of the ones reading. I've received conflicting advice on a first submission before: add human subjects, do not add human subjects. Take out Aim2, expand Aim2. And, yes, conflicting advice is horrible to deal with, and you can't know if the person who gave that advice is going to review again. NIH has tried to circumscribe that kind of advice with "review the proposal in front of you, do not write a new version for the PI". But even just saying "this is good" or "this is weak" can show up in the same review.
If I was told that my review had to be limited by the previous reviews, it would make reviewing very hard for me. I read each proposal. I try to give each proposal my very best thoughts. If I see something glaringly bad, that got missed (as far as I can tell) in the previous review, I am not going to give it a pass because some other reviewer didn't see the problem. But, if I think something is very very good, significant, innovative, and the previous review said "meh", I am also going to point that out, and advocate for it.
People last night said that this is a problem with reviewing, if this happens all the time. I don't know if it happens all the time. I dont have statistics. It has certainly happened to me, more than once, and I've been putting in proposals for a very long time.
You may agree, or not with the idea that scores can drop. But, if you believe that having external reviewers, peer reviewers, reviewers from the larger community assess proposals, is a good thing, and that if proposal can get worse. that you must admit that a score may drop.
If you want to limit scores, then the system will need to change. Maybe that would be a good thing. I tend to think not, as things that would limit, in general, reviewers will not improve the system. In the end, I suspect that those limits would be co-opted by those in power, those with the most grants, and the most time and resources to submit.
Is the system broken? Once again, I say no. It is not perfect. There are problems, and individuals who get lost or hurt or destroyed in the grinding of the gears. But, the alternatives to people like me reviewing grants is letting the PO's at NIH make all the decisions. Even if they could, which they can't, physically, they just don't have time, this would not be a good thing. Right now there is some flexibility there, and as is true of everything else, those people are human beings with all the attendant flaws of human beings.
I've not edited this way I usually do, because its late and I want to get it out. There's a lot more to be said, so likely another post on this.
Mike Lauer did an analysis, and said >75% of proposals improved in prctile. Median improvement was ~10%. https://t.co/GthwImV8N3
— Alex Kwan (@kwanalexc) June 21, 2017