Archive for: May, 2016

Rigor & Reproducibility: Scientific Rigor (3/N)

May 25 2016 Published by under Uncategorized

As I said earlier (link to my intro post):

NIH is changing has changed proposal guidelines rules again. Here is a link to what they have to say about the new parameters: Rigor  & Reproducibility.

In 2/N I wrote about "Scientific Premise", or at least what I can discern about it. It's probably worth at least glancing at those before diving deep into this. The next concept with which we must (and we must) deal with is Scientific Rigor. We all like rigor. We all strive for rigor. I've never met anyone who says (seriously) "my science isn't particularly rigorous, but I'm okay with that".

Scientific rigor is defined, by NIH, as:

the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings.

Reading through the various bits of  verbiage posted by various NIH-niks, it seems that being a rigorous scientist is mostly what we'd all like to be: a good scientist by first principles. This smacks a bit of circularity and its  not much help to me (or you) in terms of actually writing the damn proposal. I do think NIH has tried, in many places to be as explicit as possible in defining these terms, with the caveat that different subdisciplines will have different standards or emphasis. In fact they say so explicitly in a couple of places. So...

One word that comes up is in the discussions of rigor is transparency. My perception is that transparency means an explicit description of what you are going to do experimentally or analytically. In the examples NIH published (you need to scroll down, or search on "examples" on this page), one major part of rigor seem to be a relatively well-written paragraph in the Research Design section. These examples hit on the points and use the words NIH thinks indicate transparency. I've deconstructed much of one example here:

Male and female mice will be randomly allocated to experimental groups at age 3 months. [my highlights]

Are samples randomly chosen and assigned to groups? If you have animals (like mice), the choice may be automatically random, and the assignment to groups important. If you are doing a clinical or epidemiologic study, then how you find & chose individuals becomes more important. For a study that involves development, or a particular disease/physiologic entity with a time course, the when of assigning to groups is important. However the exact mechanism of choosing random is not (flipping a coin, or a table or a program, it matters not). This points out one of the difficult things for interpreting these instructions: what does need to be included and what does not. The next sentence in their example gives the why of this age (what happens).

At this age the accumulation of CUG repeat RNA, sequestration of MBNL1, splicing defects, and myotonia [the criterion that is relevant to the project succinctly stated] are fully developed. [my highlights]

Then comes 1-2 sentences on what is done, including explicit dosage of the treatment being given. Short, simple but containing enough detail that someone else could do it:

The compound will be administered at 3 doses (25%, 50%, and 100% of the MTD) for 4 weeks, compared to vehicle-treated controls. IP administration will be used unless biodistribution studies indicate a clear preference for the IV route.

This is followed by another sentence that probably has been in your proposal, but is one of the cornerstones of rigor - how did you determine the sample?

A group size of n = 10 (5 males, 5 females) will provide 90% power to detect a 22% reduction of the CUG repeat RNA in quadriceps muscle by qRT-PCR (ANOVA, α set at 0.05).[my highlights]

Group size, power stats and effect size, together with statistical method, all in one compact short sentence. You do not need more than one sentence. You do not need more than what is in this sentence, even if you have spent three months learning how to do what is in this sentence. But this single sentence indicates that the person writing actually understands what is necessary for a power calculation. In my experience, effect size is over looked in the brouhaha concerning p-statistics and p-hacking and whatnot. But that's another post. This sentence also contains the criterion for determining what the effect is ("CUG repeat..."). That is an alpha (α), btw, in the phrase about how power was calculated. This is a marvelous sentence, and I do not even come close to doing this kind of science, yet as a reviewer I can appreciate what this sentence signals about the proposer. It would be very easy to take this sentence structure and put your own science into it:

A group size of n = 16 (8 males, 8 females) will provide 90% power to detect a 10% reduction in the efficiency of hindfoot flexion measured through frame-by-frame analysis of high-speed cinematography (ANOVA, α set at 0.05). [my highlights]

I do not think this is plagiarism. A time honored way to learn how to write is to take other people's excellent sentences and rewrite them with new content. Another a time honored way to avoid plagiarism is to take notes whilst writing, and then write a para from your notes. Further, there are a number of grant-writing workshops that have their own explicit sentence-level templates. When reviewing, I can almost always tell which workshop the PI attended. I do not mind the use of templates when reviewing grants. In fact, it usually makes it easier to read the grant and understand the content and meaning. Remember, you do not want to make the reviewer work. Your goal is to enlist them as an ally in promoting your proposal to the study section.

Back to the example, more on methodology including blinding of scientists (and which are blinded of a large group of workers):

The treatment assignment will be blinded to investigators who participate in drug administration and endpoint analyses.

This is another sentence which you can use as a template for your proposal. And this example ends with a statement about goodness-of-lab:

This laboratory has previous experience with randomized allocation and blinded analysis using this mouse model [refs]. Their results showed good reproducibility when replicated by investigators in the pharmaceutical industry [ref].

This is more along the lines of documenting what a lab/ PI can do. I am not sure both of these sentences are necessary, and certainly a well-published PI could put references at the end of these statements documenting this.  But, pointing out that one can do the work necessary for the project is an important part of any proposal.

There are other examples (again scroll down to example #2 &3), that are also worth reading, sentence by sentence. Read each sentence in their examples and ask: what is the purpose of this sentence? What is being conveyed about doing the research as well as the specifics of the project being proposed? From example #3:

 Random Forest [a machine learning approach described in the sentence before] uses a bootstrap method to assess test error, ideal in our situation of small sample size (n=18). For diversity and load measures, significance between groups will be assessed using non-parametric Wilcoxon rank-sum tests.   

This acknowledges the small sample size and proposes a valid methodology. It might have been improved with a citation where the program and method had been successfully published, even if not in the same situation.

In Mike Lauer's (who writes in the NIH extramural blog, Nexus) post on Rigor he makes a couple of points worth repeating about what is rigor, comparing grant applications to papers (here is an update about Rigor from Mike):

In published papers, full transparency in reporting experimental details is crucial for others to assess, reproduce, and extend the findings.

He points out that signaling [my choice of word] of rigor in a proposal would include both experimental aspects (what you are going to do) and analysis aspects (both how you decide what to do, and also what you are going to do with the data you get). These things include sample size considerations, determination of what constitutes scientific control in experiments, replications, and avoid bias. Bias and Robust are two more words that show up frequently. Luckily NIH tries to define everything:

What is meant by "robust" and "unbiased"?

Robust results are obtained with solid, well-controlled experiments capable of being reproduced under well-controlled conditions using reported experimental details. Applicants should consider methods to reduce bias, such as having multiple individuals recording assessments, defining terminology in advance, using independent, blinded assessors, etc. "Robust" and "unbiased" results are goals, not absolute standards to be met, and may vary across scientific disciplines.

Note room for variation in the application of these ideas. The reviewer instructions (download the pdf here) contain these bon mots:

Scientific Rigor: The strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results.

There was some back & forth in the blog comments about "strict application" and innovative/exploratory work. This is indeed a whole separate post, but what exactly a PI believes, nay, knows to be "exciting, exploratory" and my favorite "paradigm changing" (something NIH allegedly wants), is often perceived by reviewers as "lacking rigor" or "sloppy reasoning". Framing your exciting new stuff in the context of the scientific method, and explaining why the results are paradigm changing is a more successful route to take in proposal writing, with respect to what study sections perceive.

The reviewer instructions continue with the following points:

Whereas scientific premise pertains to supporting data, scientific rigor pertains to the proposed research (statistical procedures, data analysis, precision, subject inclusions and exclusion criteria, etc.).


Scientific premise <--> supporting data (what is known)

Scientific rigor <--> proposed research (what you are going to do)

So, first off, I am always suspicious of any writing that starts with "whereas". We are not lawyers or politicians.  But the upshot of this is: the premise (see 2/N) is in the significance, background and the justification/support. The rigor is in the design and approach. And the important caveat:

Different research fields may have different standards or best practices for scientific rigor.

I suppose this seems to go without saying, but its nice to see it like this. I remember when power stats were first being demanded in IACUC reviews of sample size justification. It was difficult to persuade people that there were a variety of legitimate sources that a PI could use in the calculation of either the variation or the effect size parameters.

Rigor will be assessed in peer review as part of the Approach criterion for research grant applications and as part of the Research Plan criterion for mentored career development award applications.

There is a lovely table in the reviewer guide, as to where this information goes, but the gist of which is:

Scientific Premise: pertains to Supporting Data and the review criterion is Significance

Scientific Rigor: pertains to Proposed Research and the review criterion is Research Plan

At some point in the future, I will talk about the two more parts (four in total) to the guidelines: (3) Consideration of Relevant Biological Variables (which includes sex as a biological variable (SABV); and (4) Plan for Resource Authentication. In some ways these are easier to understand, if harder to implement.




2 responses so far

What I did in New York City

May 24 2016 Published by under Uncategorized



My sweetie and I went to NYC for what has now become our twice a year trip. We did many exciting things include meeting  2016-05-14 12.41.54@schoppik and family. The restaurant we & schoppikniks went to was wonderful Asian. Asian condiments make very very happy 2016-05-14 11.44.05







We stayed near this place (Nolita) and ate many other lovely things. They also had my two (current) favorite teas from Harney & sons: Paris & Hot Cinnamon Spice.


2016-05-13 20.53.332016-05-13 20.50.542016-05-13 20.49.44

We ate at weird places 2016-05-13 20.43.32(dead branches seem to be A Thing. Even in the middle-of-nowhere) and not so-weird classic French places.2016-05-15 18.36.09The Butcher's Daugher 2016-05-14 09.18.18is a vegetarian/mostly vegan restaurant near where we stayed. I thought it was great. Others did not.





Of course we saw buildings that were formerly Important Places in the History of Jazz. Did you know that every third building in Manhattan was famous & important to Jazz at some point? This modern building in mid-town (part of CCNY, I believe) used to be a recording studio where many Important and Famous Songs were recorded.

2016-05-16 10.55.41







We also heard Vince Giordano & the Night Hawks: 2016-05-16 19.43.36 They play early (20s/30s) Jazz and are the house band for Boardwalk Empire, and many Woody Allen films. They play Mon & Tues night at the Iguana (upstairs) where you can get good Mex food, and marvelous drinks. This is NOT their best song, but one that might appeal to a larger group.

The music is marvelous and lively and there are always people who can seriously dance.  He called out my sweetie by name (famous visiting author from the Midwest! - which is at least 2/3 true), and was the second most wonderful thing that happened. The most of course was seeing the picture below. Some people follow an actor, some follow a music group or an artist. My sweetie follows a building:

2016-05-16 10.50.45

Finally here is a picture that I have no idea what it is or why I took it. I'm not even sure what it is, other than dirt and maybe grubs?


2016-05-14 11.40.50


One response so far

A public service announcement

May 24 2016 Published by under Uncategorized

A public service announcement. There is great hoof-ha about postdocs (ah-gain) and technicians (I've added to the hubbub) and lots of people saying lots of things, some by Important People. One could spend much time reading and thinking and arguing about this. It is an important issue. One could also read what one needs to know in a short time and return to the (only occasionally, it often seems) glorious business of actually writing grants. Let alone the world of doing science. Just saying.

No responses yet

Quote of the Day

May 23 2016 Published by under Uncategorized

“People think dreams aren't real just because they aren't made of matter, of particles. Dreams are real. But they are made of viewpoints, of images, of memories and puns and lost hopes.” ― Neil Gaiman

My dreams quite frequently involve Those must be the puns & memories. Not lost hopes.

No responses yet

How I think of my parents (and a hat tip to Bernard Herrmann)

May 23 2016 Published by under Uncategorized

The movie "The Ghost and Mrs. Muir" is a wonderful movie that has stood quite the test of time. The sound track is marvelous  ( composed by Bernard Herrmann and conducted by Elmer Bernstein).  Here is an homage I found on the web. The movie moves through the decades of the life of Lucy/Lucia Muir. But the last scene, when Gene Tierney dies, and her ghost leaves her body and joins Rex Harrison, she is once again dressed as a woman in the early 20th century, long skirts and all. It's at the end of this clip.

My mother is not lovely now. Her dementia is written all over her face and she in fact looks a little scary. But, when I think of her and my father, I think of them as they were in this picture, in the late 40s. I do remember her wearing big-skirted dresses and the two of them dancing in the living room when I was very little. Through my child-eyes, they were so very handsome.


No responses yet

quote of the day

May 20 2016 Published by under Uncategorized

In this age, which believes that there is a short cut to everything, the greatest lesson to be learned is that the most difficult way is, in the long run, the easiest. --Henry Miller

No responses yet

Technicians are not grad students in training

May 20 2016 Published by under Uncategorized

Over at Psychosimian, in the discussion about the PD/40 hr/OT discussion, someone commented:

Beyond post docs, this will have a huge effect on lab techs! All of my techs work more than 40 hours a week…

with the justification:

The techs are mini grad students. First or second year out of college, with their own projects, working for a while before applying to grad school.

drugmonkey says in response: So what?

To which I add: are you kidding me? Techs are not mini-anything. They are techs. They may have more or less responsibility. They may have their own projects or not. They may want to go to grad school or med school or just pay back educational debt. And, while it is always best to work on the basis that other people are honest and forthright, it is not always so. A tech, who wants a job with you, may say what they think you want to hear. I know lots of basic science types who are not enthusiastic about pre-meds, and most pre-meds know this. You are not going to be privy to what they may truly think and want. They may not know what they think and want.

While working for you, your technicians are not getting course credit for what they do in your lab. They may be learning skills, but the ones you want and need and chose for them. You are paying fringe benefits not for a student, but for an employee, and these benefits are different at most places. These techs have a job to do. Most PI's I know would not tolerate in a tech what they do in a student. Most PI's have very different expectations of a tech than a student. You may say "this is not me, of course I treat everyone in my lab equally", and I will laugh.  The people who are the most unequal in treatment, in my experience, are the ones who claim equality.

But the bottom-most line of the bottom lines is that they are human beings, working a job. One could argue that lower-level managers at fast food places want to move up. They are training to become something more than a low level manager. The whole point of the law is that it doesn't matter what you are or are not training for. Someone who is working a job that pays less than $47.4K gets overtime.

Personally, I dislike the exemptions. But a line needs to be drawn somewhere. There is a distinction between students and techs (leaving aside the postdoc issues, for now).  If you have one tech who is "mini-training" and another who is not, does the one who is not training get overtime pay, and the one who is training does not?  Your perception of their future is not where the line is drawn between exempt and non-exempt. The line drawn is between students who are registered and technicians who are not.

Ultimately, this tech may apply, and even go, to grad school. Maybe they won't.  Maybe their experience with you will change their mind about future goals. But at this moment, their plans, whatever they are and whether you know them or not, do not change the contractual nature of the relationship between you and them. And that contract is governed by certain very specific laws in the United States.



6 responses so far

Rigor & Reproducibility: Scientific Premise (2/N)

May 11 2016 Published by under Uncategorized

We're back thinking about Rigor & Reproducibility, as perceived by the NIH. Firstly, there are lots of online resources from the NIH,  including the reviewer guide (a pdf download) for these issues. It is always worth taking a look at what NIH tells its reviewers. Much is a repeat of the information that they have on the various web pages, but some of it is not. When writing a proposal, taking time to try and think in terms of a critical reviewer can make a difference.

So here's new concern number 1: The scientific premise of the proposed research. Premise, in logic, is the stuff that comes before, the basis of what comes next. The next, in this case, are the hypotheses one proposes to test, or the questions one proposes to answer. It is the basis of the argument, of the undertaking. Here's the NIH wording on this:

The scientific premise for an application is the research that is used to form the basis for the proposed research question(s). NIH expects applicants to describe the general strengths and weaknesses of the prior research being cited by the applicant as crucial to support the application. It is expected that this consideration of general strengths and weaknesses could include attention to the rigor of the previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources. See related FAQs, blog post

In the reviewer guidance it starts with this:

Scientific Premise: The key data introduced by the applicant to justify the project.

This is useful because it is a little bit different from the definition on the main page. To reviewers, premise is explicitly defined as "the key data".  Basically, what in the past (I'm trying to avoid using 'why' here) gives you justification for doing what you propose to do?  The FAQ has more insight for us:

Scientific premise concerns the quality and strength of the research used to form the basis for the proposed research question. NIH expects applicants to describe the general strengths and weaknesses of the prior research being cited by the applicant as crucial to support the application.

Now, where should this go in a proposal? When skimming all of the information (and there is a lot of it), it seemed easy to get confused about what goes where. In the web page explication of premise, Mike Lauer explicitly says: a part of the Significance section of the Research Strategy, ...

Just as a reminder:

 SignificanceR01, R03, R21, R34. Does the project address an important problem or a critical barrier to progress in the field? [more... ]

Significance is demonstrating the "importance" of the project. So, how do Significance and Premise differ and interact, beyond being in the same place in the proposal? Significance explicitly asks: how will the field change if this research is done, how will the world in general, and a number of specific things, including knowledge, etc, be improved as a result of the proposed work. The FAQ says [my emphasis]:

Scientific premise includes a retrospective consideration of the foundation for the application, rather than a prospective analysis should the aims be achieved.

This is one of the main clues for us. Premise includes: what in the past (the literature, other results) suggests that this is important work to do? General significance is: what will happen in the future if this project is carried out.

For example, the significance of a study of age of onset of  type 2 diabetes in children, comparing males and females, urban and suburban, socioeconomic class, could be that type 2 pediatric diabetes is a precursor of other pediatric concerns, as well as a predictor of adult obesity. In asking for the premise, NIH wants an evaluation of the data that links pediatric type 2 diabetes to (1) other pediatric concerns and (2) adult obesity. The use of the word "precursor" as I did here, suggests a need for data that demonstrates a causal link. "Predictor" implies correlation and less likely causation. You may wish to argue about the specific implications of these words. But, the words linking the underlying background to the proposed work should chosen with care, as they can be replete with implications.  I chose these with those specific implications in mind.  The larger significance in this example would be determining if the premise holds for both sexes/genders, and equally for other parameters. This might include the idea that finding any differences would be a suggestion of where to look next for the underlying causes of the relationships in the premise.

As a closing note, the reviewer guidance includes the following bon mots addressing the reviewer, "you" in the quote [my emphasis, again]

You should factor a weak premise or the failure to address scientific premise adequately, into your criterion score and overall impact score. The page limit is not an acceptable excuse for an applicant to not address scientific premise.


One response so far

grant writing as haiku

May 10 2016 Published by under Uncategorized

the format is specified. the number of aims is rigid.

but what is said, what is proposed, that is you.

No responses yet

Rigor & Reproducibility in NIH grant proposals (1 of N, where is likely to be large and unpleasant)

May 10 2016 Published by under Uncategorized

NIH is changing has changed proposal guidelines rules again. Here is a link to what they have to say about the new parameters: Rigor  & Reproducibility.

How do I know I'm getting old? I look at this page and I sigh deeply and wonder how much longer I want to do this. Then I start thinking about ideas for projects and I get charged up again. So, I'm gonna dive into this one, and see what makes sense to me. I've not read anything else about this yet, so this is mostly my own take on it.

First off, it would be very easy to do a cynical parody page of this introduction:

Scientific rigor and transparency in conducting biomedical research is key to the successful application of knowledge toward improving health outcomes. The information provided on this website is designed to assist the extramural community in addressing rigor and transparency in NIH grant applications and progress reports.  

I mean, for pete's sake, who of us doesn't want to be rigorous and transparent? But there are goals, blah blah blah, NIH has the highest standards, blah blah blah. That's not what we need. We need: what is changed in the proposals? What kind of wording needs to be added? Of course, its wording, because of course, we have always been rigorous and transparent. We now just need to make sure that the reviewers perceive our inherent rigor and transparency. This. Is. Not. A. Parody. I really do want to figure it out and write something meaningful for you, (and learn it for myself). Right now this applies to research grants (R's)and mentored career development (most Ks)awards. Institutional Training Grants (T32), Institutional Career Development (K12/KL2), and Individual Fellowships (F-awards) will come "no sooner than 2017".

There are four main areas or foci:

  1. The scientific premise of the proposed research
  2. Rigorous experimental design for robust and unbiased results
  3. Consideration of relevant biological variables
  4. Authentication of key biological and/or chemical resources

Here's the pretty picture: (and a link to the better resolution pdf):

Infographic courtesy of Ms. Nichole Swan, Dr. Shana Spindler, and Dr. Yvette Pittman of the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD).

The thing I'm left with, tonight, as I write this post is that NIH wanted to add these concepts explicitly. But, unlike the changes in page length, or adding Innovation and Significance sections, these are not straightforward in terms of the writing of the proposal. The exception is the 4th, authentication. This is explicitly included in an extra attachment of no more than one page. From the FAQ:

Authentication of key resources will be addressed in a separate attachment and will not be scored.

Not being scored is important, as it means do not put extra science in this attachment. Doing so will only irritate the reviewer. Remember, the primary meta-rule of grantsmanship is: make the reviewer your ally and champion.

Reading through this, I found dealing with the other foci less transparent. To start, one of the questions in the FAQ is where to address these focus areas:

Where in grant applications should applicants address the four focus areas of the NIH policy on rigor and transparency?

Scientific premise, scientific rigor, and relevant biological variables such as sex should be addressed within the Research Strategy of research applications, as these elements are integral to the research plan. Since scientific premise will be reviewed and scored as part of the Significance review criterion for research grant applications, applicants should address premise as part of their corresponding Significance section in the Research Strategy. Scientific rigor and relevant biological variables will be reviewed and scored as part of the Approach review criterion.

For mentored career development award applications, all three areas (scientific premise, scientific rigor, and relevant biological variables such as sex) should be addressed in the Research Strategy and all three areas will be reviewed as part of the Research Plan.

This seems a bit vague, but basically you've got to include it everywhere.

My second favorite question is always this. My perception of "how am I supposed to do this in the page limits" is that it is a newbie question. I stand by my earlier advice about page limits.

Are the current page limits sufficient to describe rigor and transparency?

The NIH expects that rigor and transparency can be included within the existing page limits for the Research Strategy. ... [more language on if you screw with page limits, we will not review you, and you will not get funded and you will die a thousand little deaths].

Ok, it's late, I'm tired, and tomorrow is going to be a long day. I'll come back and think about the foci, and try & write about how to cope. For Sex as a Biological Variable (SABV) go see the DM.

3 responses so far

Older posts »