There are tons of examples of bad science getting good press and of good science getting exaggerated press that makes you think Cancer Has Been Cured or Disabled Children Can Not Only Walk but Walk on The Moon.
This morning on the tweets I engaged one Christian Jarrett
@Psych_Writer Editor of @ResearchDigest. Neuro blogger for @WIRED; writer on creativity/productivity @99U. Next book: Great Myths of the Brain.
He tweeted about an article he wrote:
— Christian Jarrett (@Psych_Writer) July 31, 2014
And was answered by
My reply and his reply to me:
The original article is Differences in voice-hearing experiences of people with psychosis in the USA, India and Ghana: interview-based study by T. M. Luhrmann, R. Padmavati, H. Tharoor and A. Osei. The published study had a sample of 20 people from the USA, India and Ghana, who heard voices were diagnosed as schizophrenic, and were interviewed. The conclusions the authors reached was:
These observations suggest that the voice-hearing experiences of people with serious psychotic disorder are shaped by local culture. These differences may have clinical implications.
I've gone back and read both the original science and the report on it. Jarrett does say that more research is necessary to "replicate these findings". The point I picked up on from Keith Laws was that the sample may be so biased (because it was not controlled) as to make the results meaningless.
The real meaning of a random sample is that each and every member of the population has an equal chance of being picked for the sample. Of course, this is usually impossible, both practically and philosophically. If you are testing a new screening method for blood sugar, you want it not just to work for people right now, but for people who would need screening in the future. If your population is to extend into the future (i.e., the population is all people who have had diabetes, and all people who will have diabetes), then there is no way your sample can be truly random, because you can't include the future people. Practically what most people settle for is that there is no bias in your sample. That some particular group/characteristic is not more or less likely to get picked. Sometimes stratification can help with this (ie analyze the men differently from women, include age as a covariate).
This is not necessarily easy.
The worst bias is the stuff you don't know about. Or can't know about. Or haven't thought about it. This is one of the reasons that the NIH is asking animal researchers to include both sexes in their study. Bias is nasty, often insidious, and can wreck the best laid designs of mice and men and other non-exempt USDA species. Bias means that results you think are attributable to your main factor (sex, country, age) are in fact because there is another, unseen factor that is in your design. For example, if all the females were older, and all the males younger (animals, peoples, whatevers), and you found a sex effect, it might be due to age. That one's obvious. But there are other more subtle ones that creep in. Often in animal studies it's easier to control, things like uniform genetic backgrounds, controlled diets, controlled environments. My concern about the voice hearing study are some of the factors that were not controlled. Some of the other tweets raised concerns about the consistency of the diagnosis
— Ivana Fulli (@DrFulli) July 31, 2014
So, the question is: are the remotely sampling for the same population? A qualitative study can lay groundwork. But a biased qualitative study will only muddy the waters. Are people with SZ as likely to end up in the same place/facility in these countries? What about different socio-economic-religious groups? The point about "friendly" voices is very sexy. Very re-tweetable. But will anyone care when the study is repeated with an unbiased sample and finds no difference?
I do agree with the sentiment that cross-cultural research in mental health is to be encouraged. But not at the cost of publishing stuff that has a high potential to be wrong, and will then enter the greater knowledge-sphere of "true facts".* * Of course this depends on what % of your brain you use.