Although I like the Chirstopher O'Riley version better. This will do.
Archive for the 'Uncategorized' category
(note: experiments not going as well as one might have hoped)
I hesitate to say "the biggest" because I am sure that there are other ones.
I felt that half of the proposals I read were just not clear. I know that's a vague requirement, and yes, clear to me is not clear to someone else.
But, nevertheless, I had to work to figure out what were these people proposing to do. Part of the problem in the writing, and perhaps the design/thinking was mixing up significance with design. Part of it was way too much Significance (3-5 pages of 12). Part of it was too much preliminary data (results all over the place, in every section, and repeated). I get that you have prelim data. But tell me what it means, succinctly and then move on.
After the big question (will bunnies hop?), what are the next level questions. This is what specific aims are supposed to do. But in some proposals they didn't.
Then, how are you going to do this?
What was the sample being used? What were the experimental treatments? What were the independent/carrier variables, the response/dependent variables? What other non-scientifically-relevant, but confounding variables were in the analysis? How did the results, in your response variables, test the hypothesis (as in "if we see X then we can conclude Y"). Heck, what are the specific hypotheses (or milestones) you propose to test. This is linking significance to design/approach.
One of the hardest things in writing a proposal is figuring out how much and what detail to include of the research design. I reviewed proposals that were making type I and type II errors on this: including stuff I didn't need, and failing to include stuff I really wanted or needed to know. I was wading through the wrong detail to find the right.
[oh, I appreciate that you have a complex outline scheme that goes down to III.b.1(a). But, those numbers are useless without some headings. I am not going to be deeply understanding of your particular and idiosyncratic numbering scheme, so that when I get to III.C.1.(a), I have no clue as to why it is in a new section. Short headings like "variables measured" and "statistical analysis" are very nice.]
But one signal (meta criterion) that these proposals had problems were that it was clear they were busting at the seams: small (5mm) margins, no extra space (even 6pt) btwn para, let alone sections, no indents for para, 7pt superscript citations, small figs & captions. It made me think that the PI didn't really know what to include and what to exclude and was trying to err on having everything there.
There are no easy rule of thumbs for "being clear in your writing". I read many proposals that were clear with all the things laid out and I could follow them. I could concentrate on reviewing the proposal for content, not on figuring out what that content is.
Here are a few ideas. NIH has tons of info for reviewers (I reviewed this a while back, and will put an update in another post). One thing to do give the proposal to a colleague who is not in the same lab, and ask them. And, are there sentences that says "The significance of this proposed project..." and "The innovation in this work lies in ..." and "If this project is successfully completed, we will change the world by making it safe for bunnies everywhere...".
This does not reach the level of hitting refresh to see your score, but it is in the same phylum. When you submit scores for NIH reviews, you post them on the Commons website. There is a time for submitting and then comes a time for reading (the reviews). And in this time for reading one finds out how close are all the scores for each proposal.
When I was a sprout, and there was no intertubz, one had to wait for the "reading of the scores" in study section, where grants were reviewed in alphabetical order. Now there's an unintentional bias: last names that start with X or Y or Z. or even W or T. Only then one discovered whether the reviewers reached consensus (which at the time was something greatly desired by NIH, now not so much).
One of the best things about being older is that I now have more confidence in my reviews. But as I told my new postdoc, yes, I still at my age have some imposter syndrome. And one of the worse exposés of one's IS is when you feel that you've done the review wrong wrong wrong relative to the Big Dogs on study section. Being too low (good score) means missing some critical flaw that perhaps one was just not smart enough to see. Being too high (bad score) means having no appreciation for what is important in the field (or in the olden dayes, why this Other Big Dog should be funded despite writing a dreadful proposal).
So, indeed, this morning, which opened read phase for study section that meets next week, I did open commons first thing and read the other reviews next thing. I will admit to some small relief that there is only one proposal with wildly disparate scores, and most are at least in the same family if not genus.
Of course consensus may reflect bias all around.
update: lively discussion at the tweets on this
NIH has made sections for a reason. There in fact may be many reasons. One of them is to make it easier for you, young Padwan, to convey your ideas to the reviewers and to the people Making Decisions about funding. In fact the sections labeled "Innovation" and "Significance" in the proposal correspond to sections that are criteria for assessing a proposal. (Remember, five sections: Significance, Investigators, Innovation, Approach, Environment). Remember the instructions NIH gives me, the reviewer are available to you the writer. Read them.
Pro Tip: in your section titled "INNOVATION" have a single, summary sentence that starts "This proposal [or the project proposed here] is innovative because_________.
Corollary to said tip: In your "SIGNIFICANCE" section have a sentence that starts "The significance of this project lies in ___________. "
If you make me look for it, I may not find it. Possibly one of the most damning things in a review: It is difficult to determine the innovation in this proposal, because X (intervention proposed) is not novel, nor is the use of Y (method for assessing intervention) for disease QQ. Except of course, you had something else in mind as the innovation, and I could not find it. Some reviewers will look, and others will not. Tell them what you want them to know about what you are doing.
Phillip Roth was controversial from the start. He wrote about sex before one did. He wrote about being a modern Jew before one did. He was sexist (in my view) but honest (to his vision) and wrote with a clarity that moved me when I read his first works in high school. His Plot Against America seems prescient now.
You may not like Phillip Roth, and you may not have read anything he wrote. Yet that he influenced American culture is without question.
Each generation imagines itself to be more intelligent than the one that went before it, and wiser than the one that comes after it. George Orwell
NIH added a new category a while back: multi-PI proposals. NIH is making the distinction between roughly co-equals (ie co-PI) and folks who are senior and contribute to the work but are not taking a lead role (called co-I's). (My understanding is that co-PI counts towards tenure/promotion and co-I does not. YMMV).
The overview from NIH:
The multi-PD/PI option presents an important opportunity for investigators seeking support for projects or activities that require a team science approach. This option is targeted specifically to those projects that do not fit the single-PD/PI model, and therefore is intended to supplement and not replace the traditional single PD/PI model. The overarching goal is to maximize the potential of team science efforts in order to be responsive to the challenges and opportunities of the 21st century.
There is a lot of information here, with many sublinks.
For review criteria NIH says: "Standard NIH review criteria accommodate both single PD/PI and multiple PD/PI applications", there are a few additional requirements that feed into the review. One of these is the Project Leadership Plans for Multiple PI Grant Applications. And, of course, there is lots of information and even some good examples.
Which brings us to a comment about writing length from girlparts was:
I never used to have this problem (being too long) -throughout school, early years of grant writing - page limits were never an issue. But lately, with the rise of the multi-PI proposal, it has become a problem for me. The scope of the science is bigger with two-three labs involved. If they are also cross-disciplinary proposals, they seem to require more background information. You might have one reviewer in your field, who wants all the nitty gritty experimental details, and then someone in the other PI's field who has no idea what you are talking about, or its significance without an overview of your whole field.
I wish I knew how to make it work. I'm thinking that I should take a similar approach to that of cutting down from 25 to 12 pages - mostly less technical detail?I'll try to step back a few levels and state the general goal of the experiment, and not so much of the how, with references to our previous studies. It's a multi-submission approach: if the reviewers come back with specific technical questions, they are the easiest kind to answer. Fingers crossed.
I think she hits it on the head. You cannot stint on the significance and innovation. And part of the significance is why this is a multi-PI proposal. I agree that reducing the technical detail to include more on the justification is probably the better road to take.
Yet, one important additional consideration is that if the project is too big, even with/despite multiple PIs, you can still be dinged for "over-ambitious" . It may be time to think about other mechanisms, such as program project grants (P-grants) or even just more than one R01. If you can't include the necessary information in the 12 pages, then either 1) you are stuffing in too much detail (and indeed, page lengths are your friend) or 2) the project is too big and you are stuffing in too much big picture.
One way to tell that your multi-PI proposal is Too Damn Big is if you can't figure out how to reduce it from 5 aims to 3. There is a lot of information/thought/chatter on how many Aims is the Right Number of Aims. Make sure to read the comments, there is some very subtle NIH zen in there about substance versus organization of substance.
Second part of what are errors. Other reason for repost: I find the idea of Type I and Type II to be a useful construct for many other things in life. There are lots of ways to make mistakes, and one can just say "ah a mistake". But understanding the implications of errors calls for a finer taxonomy of what is wrong.
Type 2 errors are a bit harder to wrap one’s mind around. Around which to wrap one’s mind? Whatever.
Type 1 errors are when you reject the null hypothesis (you think: the red fish are swimming at a different speed than the blue fish) and you are wrong (in reality: the red fish swim at the same speed as the blue fish). The p-value tells you how likely you are to be wrong (5% of the time at p=.05). Also know as a false positive.
Type 2 errors are often called “the sad error” because its “nope, can’t reject the null” when in fact, the reality is that those damn red fish are swimming at different speed. Sad because you in fact would really like to find a difference. Also know as false negative results.
[This of course raises a whole ‘nuther specter of what does it mean to “really like” a result. And, if you listen to the grant mavens, you should never have a hypothesis for which there is a sad outcome – ie accepting a boring null. Having a question that right or wrong gives you something that is very interesting, if not spectacular, is one hallmark of good science].
Anyway, type 2 errors are related to power. If α is the p-value (how likely are you to be wrong in rejecting the null), then β is how likely you are to be wrong in accepting the null. 1- β is termed The Power of the result. This is the power to which your IRB or IACUC or study section refers when they say “please do a power analysis to justify your sample size”.
α, β, and sample size are linked, together with “effect size”, and a measure of often referred to as δ (delta) in one lovely relationship, so that if you know any three, you can calculate the fourth. This is the heart of power analysis. You pick values for α and β (which itself can be fraught with difficulties), often .05 and .80 although there is nothing magical, special or blessed about these levels.
The next step is to calculate the effect size, another bag of worms that is worthy of its own post. The effect size is the difference you expect to see, the slope you expect to see, the whatever value you calculate that would reach significance in your eyes. But, I hear you cry, how the hell do I know this, if that is the whole damn point of the experiment/project? Aha. You use preliminary data. You can use data from a similar experiment. You can even use data from the literature. You can guess. For our fish: we expect a different of at least 1cm/sec between the reds and the blues. Else it doesn’t matter. Now, the raw effect size is usually not a very good measure, because you haven’t included a sense of the variation. Thus if a difference of 2.0 cm/sec means one thing if the standard deviation in each group is about .005 cm/sec, then 2cm/sec is a honking big difference, but if the sd is about 4cm/sec, than a difference of 2cm sec is not worth thinking about. To calculate sample size, some programs ask for the variations or SE or sd as well as effect size, others just ask for a scaled effect (diff/sd usually).
Then, you take all these numbers push them through a program or your favorite stats whiz, and get out The Sample Size (which can be plopped into your grant or protocol).
The bottom line here: this is not magic. It is not words from the Gods. This is not beyond the ability of any (and I do mean any) scientist to think about. It’s just one way, a very good way, of thinking about your data.
But really only about type I. I know most everyone knows this. But I remember when I didn't and had to learn it. Maybe this will help.
Type 1 and type 2 are lousy names. They convey no information about what they are. Once, I asked a very good friend who was finding new species and naming them to name if he would be willing to name one after me (I was younger and brasher). He said “hell, no, that name will convey nothing about the species” as in H. longifolia or H. bigantleria
Anyway type 1 error is the type you are always seeing associated with p=.05. It is the probability of being wrong if you reject the null hypothesis. That is, if you say that red fish are bigger than blue fish, based on your samples of each, you could see this much difference in size between them just by chance. You roll a single die several times and get the following distributions – these are the number of times you rolled each number.
This is about what you would expect for 100 roles – about equal numbers of each. Now, here is what your labmate (the one who is stealing your data, you think) got, with hir own die:
This is NOT what you expect. All those 6’s? nearly 50% 6’s? Now, of course it is possible that someone would roll a die 100 times and get a 6 47 times. But how likely??? That is the question that p-values answer.
In the olden days, when I was young, you had to calculate this crap by hand, or look it up in a table that never ever made sense. But now, with the advent of modern society, decafe coffee, no smoking in restaurants, it is possible to have it computed for you. That is the p-value. What is the probability of seeing this by chance? p=0.05 tells you how much chance. Or 1 in 20 times you would see this much difference by chance.
And yes, someone is stealing your data (p = 0.00000012 in the above case).