Archive for: May, 2018

(one of the) Biggest Concerns/Problem in Grants I Have Reviewed (This Time)

May 31 2018 Published by under Uncategorized

I hesitate to say "the biggest" because I am sure that there are other ones.

However...

I felt that half of the proposals I read were just not clear. I know that's a vague requirement, and yes, clear to me is not clear to someone else.

But, nevertheless, I had to work to figure out what were these people proposing to do. Part of the problem in the writing, and perhaps the design/thinking was mixing up significance with design. Part of it was way too much Significance (3-5 pages of 12). Part of it was too much preliminary data (results all over the place, in every section, and repeated). I get that you have prelim data. But tell me what it means, succinctly and then move on.

After the big question (will bunnies hop?), what are the next level questions. This is what specific aims are supposed to do. But in some proposals they didn't.

Then, how are you going to do this?

What was the sample being used? What were the experimental treatments? What were the independent/carrier variables, the response/dependent variables? What other non-scientifically-relevant, but confounding variables were in the analysis? How did the results, in your response variables, test the hypothesis (as in "if we see X then we can conclude Y"). Heck, what are the specific hypotheses (or milestones) you propose to test. This is linking significance to design/approach.

One of the hardest things in writing a proposal is figuring out how much and what detail to include of the research design. I reviewed proposals that were making type I and type II errors on this: including stuff I didn't need, and failing to include stuff I really wanted or needed to know. I was wading through the wrong detail to find the right.

[oh, I appreciate that you have a complex outline scheme that goes down to III.b.1(a). But, those numbers are useless without some headings. I am not going to be deeply understanding of your particular and idiosyncratic numbering scheme, so that when I get to III.C.1.(a), I have no clue as to why it is in a new section. Short headings  like "variables measured" and "statistical analysis" are very nice.]

But one signal (meta criterion) that these proposals had problems were that it was clear they were busting at the seams: small (5mm) margins, no extra space (even 6pt) btwn para, let alone sections, no indents for para, 7pt superscript citations, small figs & captions. It made me think that the PI didn't really know what to include and what to exclude and was trying to err on having everything there.

There are no easy rule of thumbs for "being clear in your writing".  I read many proposals that were clear with all the things laid out and I could follow them.  I could concentrate on reviewing the proposal for content, not on figuring out what that content is.

Here are a few ideas. NIH has tons of info for reviewers (I reviewed this a while back, and will put an update in another post). One thing to do give the proposal to a colleague who is not in the same lab, and ask them. And, are there  sentences that says "The significance of this proposed project..." and "The innovation in this work lies in ..." and "If this project is successfully completed, we will change the world by making it safe for bunnies everywhere...".

 

 

 

No responses yet

Small part of reviewing grants: seeing all the scores

May 29 2018 Published by under Uncategorized

This does not reach the level of hitting refresh to see your score, but it is in the same phylum. When you submit scores for NIH reviews, you post them on the Commons website. There is a time for submitting and then comes a time for reading (the reviews). And in this time for reading one finds out how close are all the scores for each proposal.

When I was a sprout, and there was no intertubz, one had to wait for the "reading of the scores" in study section, where grants were reviewed in alphabetical order. Now there's an unintentional bias:  last names that start with X or Y or Z. or even W or T. Only then one discovered whether the reviewers reached consensus (which at the time was something greatly desired by NIH, now not so much).

One of the best things about being older is that I now have more confidence in my reviews. But as I told my new postdoc, yes, I still at my age have some imposter syndrome. And one of the worse exposés of one's IS is when you feel that you've done the review wrong wrong wrong relative to the Big Dogs on study section. Being too low (good score) means missing some critical flaw that perhaps one was just not smart enough to see. Being too high (bad score) means having no appreciation for what is important in the field (or in the olden dayes, why this Other Big Dog should be funded despite writing a dreadful proposal).

So, indeed, this morning, which opened read phase for study section that meets next week, I did open commons first thing and read the other reviews next thing. I will admit to some small relief that there is only one proposal with wildly disparate scores, and most are at least in the same family if not genus.

Of course consensus may reflect bias all around.

 

update: lively discussion at the tweets on this

8 responses so far

Grant Writing Tips

May 25 2018 Published by under Uncategorized

NIH has made sections for a reason. There in fact may be many reasons. One of them is to make it easier for you, young Padwan, to convey your ideas to the reviewers and to the people Making Decisions about funding. In fact the sections labeled "Innovation" and "Significance" in the proposal correspond to sections that are criteria for assessing a proposal. (Remember, five sections: Significance, Investigators, Innovation, Approach, Environment). Remember the instructions NIH gives me, the reviewer are available to you the writer. Read them.

Pro Tip: in your section titled "INNOVATION" have a single, summary sentence that starts "This proposal [or the project proposed here] is innovative because_________.

Corollary to said tip: In your "SIGNIFICANCE" section have a sentence that starts "The significance of this project lies in ___________. "

If you make me look for it, I may not find it. Possibly one of the most damning things in a review: It is difficult to determine the innovation in this proposal, because X (intervention proposed) is not novel, nor is the use of Y (method for assessing intervention) for disease QQ. Except of course, you had something else in mind as the innovation, and I could not find it. Some reviewers will look, and others will not. Tell them what you want them to know about what you are doing.

 

3 responses so far

Phillip Roth has died

May 23 2018 Published by under Uncategorized

Phillip Roth has died.

Philip Roth - 1973.jpgPhillip Roth was controversial from the start. He wrote about sex before one did. He wrote about being a modern Jew before one did. He was sexist (in my view) but honest (to his vision) and wrote with a clarity that moved me when I read his first works in high school. His Plot Against America seems prescient now.

You may not like Phillip Roth, and you may not have read anything he wrote. Yet that he influenced American culture is without question.

 

No responses yet

QOTD: Generational dystopia

May 23 2018 Published by under Uncategorized

Each generation imagines itself to be more intelligent than the one that went before it, and wiser than the one that comes after it. George Orwell

No responses yet

Multi-PI proposals (part 1)

May 22 2018 Published by under Uncategorized

NIH added a new category a while back: multi-PI proposals. NIH is making the distinction between roughly co-equals (ie co-PI) and folks who are senior and contribute to the work but are not taking a lead role (called co-I's). (My understanding is that co-PI counts towards tenure/promotion and co-I does not. YMMV).

The overview from NIH:

The multi-PD/PI option presents an important opportunity for investigators seeking support for projects or activities that require a team science approach. This option is targeted specifically to those projects that do not fit the single-PD/PI model, and therefore is intended to supplement and not replace the traditional single PD/PI model. The overarching goal is to maximize the potential of team science efforts in order to be responsive to the challenges and opportunities of the 21st century.

Here is the link to Policy notices, this year and last,  (1) and (2).

There is a lot of information here, with many sublinks.

For review criteria NIH says: "Standard NIH review criteria accommodate both single PD/PI and multiple PD/PI applications", there are a few additional requirements that feed into the review.  One of these is the Project Leadership Plans for Multiple PI Grant Applications. And, of course, there is lots of information and even some good examples.

Which brings us to a comment about writing length from girlparts was:

I never used to have this problem (being too long) -throughout school, early years of grant writing - page limits were never an issue. But lately, with the rise of the multi-PI proposal, it has become a problem for me. The scope of the science is bigger with two-three labs involved. If they are also cross-disciplinary proposals, they seem to require more background information. You might have one reviewer in your field, who wants all the nitty gritty experimental details, and then someone in the other PI's field who has no idea what you are talking about, or its significance without an overview of your whole field.

and

I wish I knew how to make it work. I'm thinking that I should take a similar approach to that of cutting down from 25 to 12 pages - mostly less technical detail?I'll try to step back a few levels and state the general goal of the experiment, and not so much of the how, with references to our previous studies. It's a multi-submission approach: if the reviewers come back with specific technical questions, they are the easiest kind to answer. Fingers crossed.

I think she hits it on the head. You cannot stint on the significance and innovation. And part of the significance is why this is a multi-PI proposal. I agree that reducing the technical detail to include more on the justification is probably the better road to take.

Yet, one important additional consideration is that if the project is too big, even with/despite multiple PIs, you can still be dinged for "over-ambitious" . It may be time to think about other mechanisms, such as program project grants (P-grants) or even just more than one R01. If you can't include the necessary information in the 12 pages, then either 1) you are stuffing in too much detail (and indeed, page lengths are your friend) or 2) the project is too big and you are stuffing in too much big picture.

One way to tell that your multi-PI proposal is Too Damn Big is if you can't figure out how to reduce it from 5 aims to 3. There is a lot of information/thought/chatter  on how many Aims is the Right Number of Aims. Make sure to read the comments, there is some very subtle NIH zen in there about substance versus organization of substance.

 

 

 

One response so far

Repost: Type II errors

May 21 2018 Published by under Uncategorized

Second part of what are errors. Other reason for repost: I find the idea of Type I and Type II to be a useful construct for many other things in life. There are lots of ways to make mistakes, and one can just say "ah a mistake". But understanding the implications of errors calls for a finer taxonomy of what is wrong.


 

 

Type 2 errors are a bit harder to wrap one’s mind around. Around which to wrap one’s mind? Whatever.

Type 1 errors are when you reject the null hypothesis (you think: the red fish are swimming at a different speed than the blue fish) and you are wrong (in reality: the red fish swim at the same speed as the blue fish). The p-value tells you how likely you are to be wrong (5% of the time at p=.05). Also know as a false positive.

Type 2 errors are often called “the sad error” because its “nope, can’t reject the null” when in fact, the reality is that those damn red fish are swimming at different speed. Sad because you in fact would really like to find a difference. Also know as false negative results.

[This of course raises a whole ‘nuther specter of what does it mean to “really like” a result. And, if you listen to the grant mavens, you should never have a hypothesis for which there is a sad outcome – ie accepting a boring null. Having a question that right or wrong gives you something that is very interesting, if not spectacular, is one hallmark of good science].

Anyway, type 2 errors are related to power. If α is the p-value (how likely are you to be wrong in rejecting the null), then β is how likely you are to be wrong in accepting the null. 1- β is termed The Power of the result. This is the power to which your IRB or IACUC or study section refers when they say “please do a power analysis to justify your sample size”.

α, β, and sample size are linked, together with “effect size”, and a measure of often referred to as δ (delta) in one lovely relationship, so that if you know any three, you can calculate the fourth. This is the heart of power analysis. You pick values for α and β (which itself can be fraught with difficulties), often .05 and .80 although there is nothing magical, special or blessed about these levels. 

The next step is to calculate the effect size, another bag of worms that is worthy of its own post. The effect size is the difference you expect to see, the slope you expect to see, the whatever value you calculate that would reach significance in your eyes. But, I hear you cry, how the hell do I know this, if that is the whole damn point of the experiment/project? Aha. You use preliminary data. You can use data from a similar experiment. You can even use data from the literature. You can guess. For our fish: we expect a different of at least 1cm/sec between the reds and the blues. Else it doesn’t matter. Now, the raw effect size is usually not a very good measure, because you haven’t included a sense of the variation. Thus if a difference of 2.0 cm/sec means one thing if the standard deviation in each group is about .005 cm/sec, then 2cm/sec is a honking big difference, but if the sd is about 4cm/sec, than a difference of 2cm sec is not worth thinking about. To calculate sample size, some programs ask for the variations or SE or sd as well as effect size, others just ask for a scaled effect (diff/sd usually).

Then, you take all these numbers push them through a program or your favorite stats whiz, and get out The Sample Size (which can be plopped into your grant or protocol).

The bottom line here: this is not magic. It is not words from the Gods. This is not beyond the ability of any (and I do mean any) scientist to think about. It’s just one way, a very good way, of thinking about your data.

No responses yet

Repost: Type I and Type II errors: p-values

May 21 2018 Published by under Uncategorized

But really only about type I. I know most everyone knows this. But I remember when I didn't and had to learn it. Maybe this will help.


Type 1 and type 2 are lousy names. They convey no information about what they are. Once, I asked a very good friend who was finding new species and naming them to name if he would be willing to name one after me (I was younger and brasher). He said “hell, no, that name will convey nothing about the species” as in H. longifolia or H. bigantleria

Anyway type 1 error is the type you are always seeing associated with p=.05. It is the probability of being wrong if you reject the null hypothesis. That is, if you say that red fish are bigger than blue fish, based on your samples of each, you could see this much difference in size between them just by chance. You roll a single die several times and get the following distributions – these are the number of times you rolled each number.

Die 1:

density x plot

This is about what you would expect for 100 roles – about equal numbers of each. Now, here is what your labmate (the one who is stealing your data, you think) got, with hir own die:

density y plot

This is NOT what you expect. All those 6’s? nearly 50% 6’s? Now, of course it is possible that someone would roll a die 100 times and get a 6 47 times. But how likely??? That is the question that p-values answer.

In the olden days, when I was young, you had to calculate this crap by hand, or look it up in a table that never ever made sense. But now, with the advent of modern society, decafe coffee, no smoking in restaurants, it is possible to have it computed for you. That is the p-value. What is the probability of seeing this by chance?  p=0.05 tells you how much chance. Or 1 in 20 times you would see this much difference by chance.

And yes, someone is stealing your data (p = 0.00000012 in the above case).

No responses yet

Word Limits, Page Limits, and my secret super power

May 18 2018 Published by under Uncategorized

My secret super power is making things short. It is enhanced by my second-in-line-power which is to write short in the first place. When I finished my thesis, it held the record for the shortest approved thesis in my department. It may still be, but I don't have lots of data to support that statement.

I have often said, here, in the tweets, to all my trainees till their eyes roll back in their heads and they fall out of their chairs in boredom:

Page limits are your friends. They tell how much the reviewer/grant-giver/journal/whatever wants to read. If you are having trouble "fitting it in", it is not the fault of the limits. You are saying too much. They don't want that much.

If you are a full page over, you don't need to go back and edit out single words. You need to go back and figure out which paragraphs can be reduced to a single sentence. If you are multiple pages too long, then you need to figure out what sections can be reduced to a single sentence, or even are totally necessary.

The ability to know what to put in a proposal, and not make type I errors (leaving out stuff that should be there) or type II errors (putting in stuff that isn't necessary), is a skill worth honing.

 

5 responses so far

Repost: A few thoughts on abstracts for NIH grants

May 18 2018 Published by under Uncategorized

Again, an old one that still applies.


Abstracts, which are now called project summaries, are critical to your grant. Sometimes, that is all that someone who gets to score you will ever read (i.e., the other people on study section who are not your reviewers). Writing the abstract after the rest is done allows you to take the best sentences from elsewhere in your grant combine them, smooth them, adjust them. This should not be too hard to write, after you have written your kick ass grant proposal. That doesn’t mean you shouldn’t take the time to make sure its good.

As an aside – if you get funded, the abstract is what is publicly available to anyone with a computer and an internet connection. If you are applying for jobs, ten to one someone on the search committee will check you out in RePORTER. If you are doing anything within the academic community, someone will check you out. It is very good to have those abstracts readable for all those people who will be checking you out.

I am busy reviewing grants now for an October study section. I have just read five abstracts and it is hard for me when I see how bad some of them are. I want to support these people, I want to advocate for them. I want to see them funded. These are all young investigators (this is a mechanism, in this institute, designed for young investigators, which a separate pot of money set aside for them). But these abstracts makes my job harder. Not that you should care about how hard the reviewer’s job is. BUT… making things easy for the reviewer ups your chance of being funded. I’d like to put 40 hours into each grant and figure out exactly what the PI means. I do not have time for that (nor does NIH expect me to have time for that). But if I understand your grant, it is much more likely I will perceive what is good about it.

These abstracts are not bad in the sense of poorly written and sentences that don’t make sense (though there is some of that). There are problems in the sense of poor choices about what to include, poor choices about what to emphasize and poor choices in opening & ending sentences.

Problems (things not to do):

1. Having divided your summary into categories (not necessarily bad, but certainly uncommon), starting with “Personnel” and discussing the PI in the third person (She will do this…). Did someone else write this grant for you and therefore is talking about you? But more to the point – as a reviewer I am less interested in what you will do, or why you are so lovely, than in why is this project interesting, important, and most of all, significant.

2. Spend 2/3 of the (single) paragraph on the justification for the work. This is too much. I want to know less about the problem, and more about what you are doing for the problem. A good strategy (for SA’s too), with each of these points being 1-2 sentences, max:

  1. What is the overall problem:  Bunny hopping is critical for a happy life.
  2. What is known: The disease XYZ is responsible for all deaths due to bad bunny hopping.
  3. What is not known: XYZ has two routes in which it has the potential to impact bunny hopping, but it is unknown which, or both, are responsible for the failures.
  4. What you are going to do: We propose to test these two models of XYZ and measure their impact on bunny hopping.

3. This issue exists independently, but also as a function of 2. No information on what you propose to do. No hint of what kind of methodology (whole animal physiology, genotyping) you are proposing. What kinds of things will you measure? This again can be 1-2 sentences, but needs to tell me what I am going to find in the approach you propose.

4.Jargon in the abstract that is not defined. This one (or two, or in this case three) word(s)  may be a common word in your sub-specialty, but I don’t know what they mean. Yes, I can look it up, but I’m actually reading your abstract during a boring meeting, prior to reading the whole thing later on, so I can’t. I stop reading yours, and move on to the next one. Of course, when I go back and write my review, I’ll look it up. But in my time-deprived, sleep-deprived world, it means you get one less reading than the others. And I’m irritated. One does not want irritated reviewers.

5. No kick-ass final sentence. Please give me something to work with in my review: The successful completion of this project will make bunny hopping possible for patients who can’t get off the ground. In general, if you can include, in the abstract, sentences like (and no, you don’t have to bold or underline these), you make life easy for the reviewer:

  • The significance of this work is:
  • The innovation of this work is:
  • The results from this project (NOT “if this project succeeds”) will change the world in the following ways…

As I keep emphasizing in the debates about font, white space, etc, that it doesn’t matter what you like, it matters what the reviewers like. This is part of grantsmanship. Do it right and you don’t just pass go and collect $200, you get hotels on Park Place and Broadway and get to get out of the game, and do the science.

 

 

No responses yet

Older posts »