Archive for: February, 2018

Jumping off into the unknown

Feb 14 2018 Published by under Uncategorized

I have a good friend who is a much better human being than I am. She is generous, with her time and her affection, her intellect and her street smarts about life. Let’s call her Susan.

Susan did a PhD in a prestigious place, and a postdoc even more so. But not in a particular sexy and shiny subfield. It involved Live Animals that were not mice or rats, but answered questions that crossed fields. She managed to get the respect of both places, and was funded by both subdiscplines, but much more NSF than NIH. She is respected and has done good work. She got married to another scientist, had kids, and made a bunch of compromises. She’s in her mid 50s now and what’s important, was a woman at a time and in a place where there weren’t many. She was an adjunct before the word and concept had evolved.

In fact, she’s been an adjunct for 20-ish years. And she’s just tired of it. She’s retiring, which as she tells it, is a bit fiscally risky. But she’s tired. She has had 5 year contracts, which while more than many of the current adjuncts in social sciences or humanities get, but its taken a toll. She’s been funded, which has been life and death for her.

I can’t say that I think much of her husband, but then no one has asked me to think much of him, let alone my opinion. She’s a better scientist. Her kids are more or less launched in life. She can point to a body of work that is good, and important, and made a difference. He’s still a jerk, in my book. He has set down some rules about where they might go. I am pretty sure these are not consistent with what Susan might want. But again, no one is asking me my opinion.

What amazes me is that she doesn’t know what comes next for her. I’m actively thinking about retirement, but one of the things that keeps pulling at me is that there is nothing I love as much as I love doing science. She and I had a long skype the other day, and I wish I had taken notes. We are different, but she still had much insight for me. There was lots of wisdom floating in the air. And I, alas, did not capture it.

Susan is leaping off into space. Yes, it is a leap that has a good safety net. She’s not going to starve. She’s not going to be on the street, sleeping in a sleeping bag under the underpass. But she doesn't know what else comes next. And maybe you could fault her for "running from" as opposed to "running to". Yet, I admire her. To know what you want, and what you don't want, really know, is a high achievement.


One response so far

Thoughts on reviewing NIH proposals: What is the difference between a 2.0 and 3.0 in initial score?

Feb 10 2018 Published by under grantsmanship, Uncategorized

There are lots of posts (some mine, some from others, more worthy correspondents) about the current state of NIH scoring and its relationship to funding.

The general and consistent message is that, yes, funding is hard to get. You need to put in the best possible proposal you can. You need to pay attention to grantsmanship issues, because that can help. But I want to move beyond that.

I've often commented that sometimes the difference between a discussed but not funded and a funded proposal is almost random. That this is one of the times grantsmanship matters, and what I consider one of the most important meta-rules: make the reviewer your advocate.

So study section time is coming up. I got a set of proposals to review, and I'm done. My reviews are posted, and many of colleagues have posted too. This produced a lot of reviews for me to look at, and I was seeing them in the light of recent twitter discussion.

I haven't seen a score better than 2. But I also haven't seen any 9's, or even anything >8.1. This is, in part, because we are explicitly told to balance our distribution, and make 5 the mean/median. Start with a 5, and go up or down from there.

Looking at the 2-4 range might be helpful. These are proposals that will be discussed. The difference between a 2 and a 4 is now the difference in getting funded. What are the words used that distinguish between the 2s and the 3s and 4s?

Here is something I saw more than once:

"Overall, while hopping disorders in elderly rabbits is a topic of importance, the work is viewed as incremental in nature and not that particularly innovative"

I did not review this proposal. This was a study that received a "2" in approach, but a "6" in innovation. A good reviewer may have more detail on why it is "not particularly innovative". I've seen things like:

These results have been established for adult bunnies, and the only difference in this proposal is that the study population is elderly bunnies.

The significance could be very high: we may want or need to know this for elderly bunnies. But the innovation is not. Here is another critique I've seen concerning innovation:

The proposed techniques and approaches have been used by this PI and study team for many years, thus not particularly new or innovative, other than being used for this project in elderly bunnies.

I am not sure this is particularly helpful, as most PI's use the same basic techniques. This critique produces the cry: "what am I supposed to do? this is what I do". There is the sentiment expressed that "reviewers are just looking  for problems". I do not think this is true. Instructions to reviewers are to  start with a "5" in every category and move up or down. I have read the complaints of people who say "I got a 3 or a 4 but there were no weaknesses."   A 3 or 4 may have no weaknesses, just not enough strength to boost it to a 2. So how then you ask, "am I supposed to improve this and get funded?".

To answer that, let's start with the NIH wording on innovation for R01s:

Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

And we are back to the discussion of last month, and last year, and honestly, last decade. NIH reviewers are not in the business of making suggestions on how to improve your proposal. You may want that. You may feel you are owed that. You may think that it is unfair not to give you that kind of help. And of course, you may have any feelings you might want to have. But those feelings will not change reality. The job, the role, the assignment to reviewers, from NIH, is to evaluate proposals and provide justification for the scores that they give.

The view that something isn't innovative because it does nothing beyond change the population is sufficient for NIH standards, but tremendously frustrating to the PI applicant. The reviewers are not supposed to tell you how to make it innovative, only judge whether it is or not.

Back in the olden dayes, when grants were 25 pages long, reviews unstructured and everyone wore suits to study section, there were unlimited resubmissions. One (but there were others) of the reasons NIH decided to go to first 2 resubs (3 total) and eventually 1 resubmission was the view that "reviewers were writing the proposals" and telling PI's what they needed to do.

In fact, I think not telling a  PI how to make a project innovative is A Good Thing. It opens up room for creativity and insight and change. If reviewers and study sections said "do this", well of course you would. And then the reviewers would not so much be evaluating your ideas and projects, but proposing their own.

The best place to go to figure out how to change your proposal are those questions and guidelines that NIH gives the reviewers (here!). Make it possible for the reviewers to answer yes! yes! yes! to those questions. The proposals that have turned me into their advocate are proposals that I want to make sure my reasoning about why this is A Damn Fine Proposal is clear, and upfront and persuasive as possible to my fellow reviewers.


No responses yet

Professionalism and Integrity on Study Section

Feb 05 2018 Published by under Uncategorized

There was some discussion on the tweets lately about What Happens At Study Section. Given that NIH triages 50% of the proposals, that is does not discuss, making that cut is important. Part of the discussion concerned whether one of three reviewers could "sink" a proposal.


A procedural note (which was cleared up on the tweets, but I can't find it and it's worth repeating here): if a proposal is triaged, there is no "voting outside the range". Voting outside the range is something that happens after a discussion and after final scores are given by the reviewer. If a proposal is triaged, no one votes and there is no range to be outside of.

In the study sections on which I have sat, since triage has become a thing, there is a list of triaged proposals that is circulated prior to review. ANYONE, not just the reviewers of a particular proposal, can call for review of a proposal (before the meeting) and move it to the discuss group. Moreover, there is another chance to do this during study section meeting.  I've seen moving from triage to discussion at nearly every study section, even from the non-reviewers. If someone feels strongly about a proposal, they can force the discussion.

Discussions tend to be complex things. If someone feels very strongly about a proposal, they can try and drive the discussion. This tends not to happen. I have seldom seen anyone be irrationally negative about a proposal. And almost always, everyone who says something negative, tries and balance it with what they do perceive as positive about the proposal. I have seen more strong very positive reviews.

"Voting out of range", for those who don't know, happens at the end of the discussion. The three (or sometimes four) reviewers each give a final score, and indicate how they have changed from their initial score, based on the discussion. Then the chair asks if anyone is voting out of the range of the reviewers. This is for the non-reviewers, as the reviewers set the range.   It happens. As DM indicated, it's often more than one person, and more often towards a worse score. My sense is that it frequently occurs when non-reviewers think that a problem raised in review is more serious than the reviewer thinks it is. They often explicitly indicate this verbally (which is how I have come to think this).

In general, IME, preliminary scores tend to have a wider range than final scores before voting. Most reviewers are not only listening, but actively engaged in discussing the proposals. Although I have heard the words "I am excited by this proposal" and "I am disappointed in this proposal", by and large, reviewers are not irrational or overly emotional about proposals. They tend to base their reviews on points of substance, and follow the NIH guidelines on reviews.

If you haven't seen the (very extensive) guidelines for reviewers, it is well worth looking at before you submit.  This link is a real rabbit hole, but one worth pursuing. For example, the guidelines for an R01 include:

How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

And I have read proposals that include the text:

Successful completion of these aims will change the treatments available for dysfunctional bunny hopping.

There is absolutely nothing wrong, and quite a lot right, with telling the reviewers what they are looking for in your proposal.

It is easy and often emotionally satisfying to be angry at Study Section and especially, IME, Reviewer #2. They Don't Get It. They are prejudiced against bunny hopping studies. There was one Reviewer who sank my study.  These things are not impossible. They are just not likely.


3 responses so far