Predicting Scientific Success and Dubious Logic

Jun 16 2017 Published by under Uncategorized

In Scienc (in an article about GREs as a less than useful tool for predicting success) we have:

can objective measures such as numbers of publications do any better at spotting true intellectual promise among faculty candidates? Not according to physicist Peter Higgs, whose work on subatomic particles in the 1960s inspired the long but ultimately successful hunt for the eponymous Higgs boson. As he told The Guardian in 2013, while traveling to Stockholm to receive the Nobel Prize in Physics, for years he had been “an embarrassment to [his] department when they did research assessment exercises.” With fewer than 10 papers published since this 1964 breakthrough, he often responded to departmental requests for lists of recent publications with a simple reply: “None.” Given today’s requirement to publish frequently, he added, “It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964. … Today I wouldn't get an academic job. It's as simple as that. I don't think I would be regarded as productive enough.”

Then there’s mathematician Yitang “Tom” Zhang, who was completely unknown—as in zero peer-reviewed publications and an adjunct teaching job—when, in 2013, at the age of 57 and 12 years out from receiving his Ph.D., he submitted a paper that astounded the mathematical world by solving a long-standing problem in number theory. Now hailed as a “genius” and a “celebrity,” he has since that triumph received numerous major prizes and appointments to two professorships, first at the University of New Hampshire and then UC Santa Barbara.

So we've got two guys who didn't publish a lot, and were eventually considered "geniuses". Does this mean that all people who don't publish a lot are geniuses? Of course not. This only falsifies the statement that "if you don't publish, you are not (or cannot be) good". Nor does it have anything to say about people who do publish a lot. We are back to type I and type II errors, and what do departments and search committees and tenure committees want to guard against.

If, of low-publishing people, 1 in 100 is a Nobel laureate material, and the other 99 are pikers, is it worth hiring or tenuring one person who hasn't who doesn't publish in the hopes that they solve The Problem? In medical terminology, know the existence of one zebra when you hear hoof beats, does that mean that you should expect the next 10 or so to be zebras, or just plain horses? What is the risk in making the wrong decision? Is the 1 in 10000 chance of losing the genius worth the 99.99% chance that you are tenuring something who will not carry their weight in the department?

I remember when I was younger, back in the mid-Oligocene, that we had lots of "dead weight" in the department. These were people who fit the early/external pattern of Prof Higgs. They were tenured, they had had one grant in the Jurassic, and came in to teach one class a term, whether they had to or not. They hadn't written a grant proposal in years, and maybe churned out one paper every other year. I was furious because they sat in judgment of the junior faculty, assessing the quality ("not quite there") of research they couldn't be bothered to do. I remember what it was like to have people who thought they might, someday, become Dr. Zhang, but just couldn't be bothered right now.

I go back and read the quotes from Prof Higgs and I am struck by the sheer arrogance of his position. The entitlement that permits him to think that  a department with standards, standards to which he might be held, would somehow inhibit his creativity.

I won't defend the current pace, nor the current obsession with funding. I've seen too many good people, working people, people who are Good Scientists (is the Nobel how we want to judge science, anyway?), chewed up by the system and spat out.

But as always, there is something between an endless, highspeed treadmill and waiting 20 years to publish. There needs to be room, in our science departments, our research communities for all kinds of people. But those people also need to understand that tenure isn't a ticket to endless coffee breaks, either.

 

 

27 responses so far

  • Microscientist says:

    To comment on the original article about predictors of success. You can't predict because, even in science, there is a certain amount of luck involved. We hate to think of it that way because science is supposed to be objective all the time. But the doing of scientific research is a combination of training, skill, art, and luck.
    This luck can be the right person reviewing and being a champion for your grant or paper, or you just stumbling on the right set of conditions for that Oh, wow experiment. To add also to your chorus, we all need more free time to just read than think about our science, something that is becoming more and more difficult these days. I am not capable of doing good science at a high rate of speed, but by response is always that I do careful, precise science. Most (including my tenure committee), seem ok with that response.

    • potnia theron says:

      glad that you have a tenure committee who sees that. Other important visual vistas for T&P committees: that not everyone can get funded in this day and age, and that there need to be other indicators of success.

  • qaz says:

    And my understanding is that Higgs hasn't actually done anything since that 1964 paper. (Certainly, his annual reports suggest that.) Basically, his department kept him on as dead weight so that they would get credit when he won the Nobel for that 1964 paper. Other people actually went and tested it. Other people followed up on what it means. Other people did all the work that actually made that 1964 paper important.

    • potnia theron says:

      Good points. And you know, and I know, and we all know, folks who revel in the prizes and glory, and are more interested in being Scientists than in Doing Science.

  • Arlenna says:

    I think the focus on "geniuses" and the publication metrics in that article was kind of unrelated to the main point, and it's annoying to me that so much page real estate was given to that argument.

    The other aspects of it, like the lack of correlation between GPAs and GREs and future scientific success is more important. Giving people grad school opportunities, hiring people as faculty, and giving them tenure doesn't need to have anything to do with whether they are "geniuses" or not. I think the important thing is the entry point, grad school admission, has been using GPAs and GREs as gates and they shouldn't be.

    • qaz says:

      It is very important to realize that there is the supposed evidence for a lack of correlation between GPA/GRE and future success in graduate school from bad data that should never have passed peer review. It is all based on uncontrolled, observational experiments with a deeply confounding problem - these are all people who were ALREADY admitted to graduate school. This doesn't mean that GPA/GRE do predict - just that we don't have any data on this. The correct experiment is to admit a random selection of GRE scores and see how they do. Of course, that's not going to happen, but most graduate schools have decided to no longer include GRE in their decisions, so we will have a better prediction test. Hopefully these schools will keep track of the GREs of the students they admit and we will see if GRE does predict success when it is not part of the equation.

      Nevertheless, I strongly agree with your main point - we need to give more people opportunities because we need to give more people the opportunity to step up and to recover from mistakes.

  • Rheophile says:

    THANK YOU!

    Higgs, in particular, rubs me the wrong way here. Perhaps there are a few people who have such singular and important insights that they need to be protected from the necessity of actually telling people what work they are doing (Andrew Wiles, Ken Wilson). But Higgs is a particularly bad example - since about five people published the same idea at the same time as him. (https://en.wikipedia.org/wiki/Higgs_mechanism)

    Those people, incidentally, all then continued to publish lots of other interesting papers, and some of them won the Nobel Prize for other topics!

  • Anon says:

    "I go back and read the quotes from Prof Higgs and I am struck by the sheer arrogance of his position. The entitlement that permits him to think that a department with standards, standards to which he might be held, would somehow inhibit his creativity."

    I don't understand what "arrogance" you are referring to. This is what Higgs is quoted as saying:

    "It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964. … Today I wouldn't get an academic job. It's as simple as that. I don't think I would be regarded as productive enough."

    And I think he's absolutely right! Incidentally, you could say the same about a number of papers written in the early 20th C. and before -- papers that pushed science forward a hell of a lot more than the incremental drivel that populates 99.9% of journals today. The way we do science and who is able to do it has changed radically. Some of that change is good (e.g., more inclusivity), but a lot of it is not. It's not arrogant to point that out to the general public.

    • The pace of science has accelerated as the tools have gotten better. For example, take crystallography. Researchers used to spend a whole PhD getting one crystal structure. Now for many things, it can be done in a day. Why wouldn't time to publication be faster with such changes? I'd be really surprised if the fraction of truly "important" (whatever that means) papers has changed much with time...

      • qaz says:

        The pace of technology has changed, but that is the nature of science. Science is always working at the edge of technology. In 1600, simply measuring a ball rolling down an inclined plane was a decade's work by the top laboratory in the world (Galileo). In my field, the undergrads do what the top labs did 30 years ago. To move the field forward, you need to do more than that. I don't think this is what is producing the increase in publications. (In your field, is the structure of a single crystal structure [one day's work] publishable?)

        What has changed the pace of science is the ease of publication. In 1900, a paper had to be written out by hand, transferred to type and then printed in a journal. In 1950, it had to be written on a typewriter (which was faster). By 1980, we had printers, but still sent to the journal in hardcopy. Now, of course, nothing ever has to go to the printer, and you can go from typing the vomit draft to making your results available to the public as fast as your little electrons can carry you. That has increased the number of publications. I would argue that it has not changed the distribution of quality of what is written. So you get more of the top papers, but also more of the mediocre ones. To quote Sturgeon's law, 95 percent of everything is cr*p. You can see this in the explosion of other fields that depend on printing (such as literature).

        • Of course, the tools getting better can also mean the tools used for paper prep. Few people write papers on one structure anymore (unless it is something really important--this isn't my field, but I can remember a few recent single structure papers for targets people had been after for years), but the improvement in tools means people can do comparative studies that yield useful information in a relatively short time now. Certainly less time than in getting one structure in 1940.

          To move the field forward, you need to either develop new tools or you need to apply existing tools in a new way or to a new system. No change in that since the early 20th century. It is just much faster now to go from exotic new tool to undergrads can do it.

          I agree with you on paper quality (see my last comment): the fraction of "important" papers has not changed with time. There definitely are way more papers, which means more bad, but also more good.

        • Anon says:

          I disagree that the fraction of important papers has not changed over time. Of course, no one will have the data to say one way or the other until much later, but I think that the increased pressure to publish more papers -- and more often -- impacts the kinds of problems that people are willing to take on. I think it’s in part what Higgs alludes to when he says, “It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964.”

          Let’s be honest, folks. If your job is on the line, if the future of your lab is on the line, are you really going to sweat the details of your LPU before sending it out? This attitude filters down to the trainees. Let’s look like we’re doing more by spending just enough time to make it “good enough” for publication … with that bar sinking lower and lower as reviewers barely read what they’re supposed to review because there are too many papers to be reviewed. And in the meantime, the important things to be learned from spending a lot of time thinking about and examining your data are lost in the shuffle. Simulation and more data are no substitute for thinking, and no one has found a way to accelerate that yet. How can that not impact the quality of most papers?

    • potnia theron says:

      Are you *sure*, absolutely *sure* that you know the difference between drivel & field changing? It's not always immediately obvious.

      • Anon says:

        Oh, please! If you can't tell the difference between drivel and field-changing most of the time, you have no business reviewing anything.

        • Rheophile says:

          I can beat "most of the time" easily! I have a 98% accurate method of telling if something is field changing. I say "this is not field changing."

          This is why I am skeptical about claims that there was a better ratio of "important" science back in the day. Which is more likely: A) 5% of papers used to be brilliant, and now it is 1%, or B) maybe there is a slight bias due to hindsight? Broad, qualitative claims about the past have a really terrible signal-to-systematic-error ratio.

        • drugmonkey says:

          If you believe with high confidence that you can tell the difference between drivel and field-changing with high and lasting fidelity then you have no business reviewing anything.

  • What a strange article. The part about predicting success was pretty interesting, and in line with my experience. I don't know of anyone who actually takes GRE scores seriously. We don't even require them. I think most of the people I've served with on the admissions committee thought that GPA was a poor predictor of success once above a certain minimum level (we need people to be able to pass our classes to succeed in grad school, but GPA and research skill don't always go hand in hand).

    The part about Higgs and Zhang was super-bizarre. Why point out the success of one highly unusual "genius" in Zhang? I don't even know how to characterize Higgs, as he was one of several people in that time frame to have a similar idea. Either way, how is Higgs an expert on predicting success? Why quote him at all? He won a Nobel, and is therefore an expert in everything? Kind of the opposite of the idea of the prize, which is supposed to be for a single achievement. Aside from that, his productivity since 1964 is besides the point, since he'd already made his breakthrough achievement by then, and would not be considered an unknown talent.

    It's all a part of the cult of "genius" in science, which lets some truly unpleasant people (for example, James Watson and Geoff Marcy) get away with doing terrible things to people for quite some time because "genius".

    • qaz says:

      SAT scores definitely predict college grades. (There's lots of very strong data on that.) I bet GRE scores predict graduate school success as well. There are three very important issues here.

      1. All of this is probabilities and correlations. If you have a low SAT or GRE score are you doomed? If you have a high SAT or GRE score are you guaranteed? Of course not! But that doesn't mean that someone with a high SAT or GRE score isn't more likely to succeed than someone with a low score.

      2. Both SAT and GRE scores are correlated with lots of other things that predict success. Some of which we want to include (actually being smart, having grit to study, being able to perform under pressure) and some of which we don't (having come from a high-income family). The SAT and GRE were always just markers for other factors.

      3. Most of the studies on this are looking at the difference between people who scored 90 percent and 95 percent on the GREs. I bet there is a real and significant difference between the average person who scores 20 percent on the GREs and the average person who scores 99 percent.

      Finally, if we don't use GREs, then we need some other measure. Do we have any other measure that is more fair? "I met them and they seemed smart" or "They know my friend person X and X says they're a good choice" (i.e. rec letters) seems like a recipe for implicit bias and the troubles that the old-boys-network generated. Is there any evidence that GPA is any better? Not given the grade inflation that I've seen, particularly coming from rich kids' schools.

      I would argue that what we really need to do is to give more people the opportunity to try (which means more funding for grad students, and people shutting up about training too many students) and making it OK to try and do something else, either by not finishing grad school or by not entering academia. It is particularly important to say "It's OK to to come to grad school, spend a couple of years at it and decide it's not for you" because if that's not OK then PhD programs can not dare risk taking on students that don't have all the markers for success.

      • Yeah, we use GPA to decide if incoming students can do well (or well enough) in our courses. Our main issue with the GRE is that the math section is high school level math (not helpful for grad level work) and the language section discriminates heavily against people for whom English is not their first language. It doesn't give more/better information than GPA, and students must pay to take it. International students have to give TOEFL scores, so GRE doesn't give more information than that either.

        We require research experience, and most students who have that get a letter from their research supervisor. We use letters, student statement (a writing sample) and GPA for admissions. In the current funding situation, we can't really take risks on students. No one has the money to burn on a student who will leave after a year or two.

        I actually agree with you--I don't think we train too many students (at least in my field, where most students don't want to be professors). I do think we don't have enough support for training students to enable us to take risks and admit people who don't have many markers of probable sucess after undergrad, which is a shame.

        • David says:

          To add on to the idea of 1-2 years and more funding, it would be great if companies would support master's level training. Seems like STEM companies have a real use for folks with some graduate level education and a master's is a good way to get it (from a time and commitment standpoint). If companies paid for it and convinced the university's administrators that it was worthwhile (and therefore the professors got credit for training those students) I think the workforce would be greatly improved.

      • drugmonkey says:

        You are right qaz, but you will make no headway whatsoever. Because this is about the 70%ile standardized scorers' bad feels about themselves. No more, no less.

  • Lemon Lemon says:

    In my opinion, one reason why the GRE is not useful as a predictor of graduate school success in many fields is that it doesn't test for/predict laboratory/research skills. One can be book-smart and not know even the basics of research methods, the "practice" of science. I have seen cases of students admitted to graduate programs with high GPAs and high GRE scores who were top students in organized classes, but were completely clueless in terms of planning a research project, what would be involved and what would be needed, the basics of time frame, and/or actually carrying out an experiment in a laboratory.
    A GRE score could be a reasonable requirement for a coursework-only Master's program, for example, therefore: and/or a GRE Math score could be a good measure of the mathematical chops required for many STEM programs, because that's a crucial element. But otherwise, I think not so much.

  • Anon says:

    "...were completely clueless in terms of planning a research project, what would be involved and what would be needed, the basics of time frame, and/or actually carrying out an experiment in a laboratory."

    Gee, call me crazy, but doing research is a *learned* skill. Like, what you're supposed to learn from your mentor in grad school.

    "a GRE Math score could be a good measure of the mathematical chops required for many STEM programs"

    Are you serious? GRE Math is a joke! Nothing more than high school math on there.

  • Paleogould says:

    It also very clearly reflects a flawed understanding of our best ideas of how scientific progress works. There are always multiple people working at the edges of a problem, pursuing different avenues, some which will prove fruitful some which will not. And indeed there are cases were the first solution proposed is later inspired/modified by other lines of thought. What, in this more collective concept, is the worth of the individual like Higgs?
    The best example of this is the two generation team of scientists responsible for the modern synthesis. Some were important titans of their field, some were reclusive, truculent mathematicians, some were reasonably ordinary scientists who were just interested enough in that problem as it applied to their respective fields. And some who were never included in the synthesis were working independently on ideas that would prove to be useful 30 years later.

  • Anon says:

    "It also very clearly reflects a flawed understanding of our best ideas of how scientific progress works."

    The best idea of how scientific progress works is that it works in many different ways. There are lone wolfs and huge collaborations, and both are valid and necessary ways of advancing science. You may wish to educate the public more about the importance of the collective effort as a whole, but that in no way diminishes the "worth of the individual like Higgs."

  • drugmonkey says:

    This is a fantastic post, Potnia. A system devoted to making sure the rarest eccentric genius is never missed is insanity.

  • […] the post on "genius" and publication rate, someone remarked on the "the incremental drivel that populates 99.9% of journals […]

Leave a Reply