Tuesday, February 9, 2016

Why does anybody mess with their data?

A few weeks ago, I was listening to a bit of point/counterpoint on the Mother Jones Inquiring Minds podcast. On one episode, Brad Bushman gave an interview about the causes of gun violence, emphasizing the Weapons Priming Effect and the effects of violent video games. (Apparently he and his co-authors have a new meta-analysis of the Weapons Priming Effect; I can't read it because it's still under revision and the authors have not sent me a copy.)

On the other, Inquiring Minds invited violent-media-effect skeptic Chris Ferguson, perhaps one of Bushman's most persistent detractors. Ferguson recounted all the reasons he has for skepticism of violent-game effects, some reasonable, some less reasonable. One of his more reasonable criticisms is that he's concerned about publication bias and p-hacking in the literature. Perhaps researchers are running several studies and only reporting the ones that find significance, or maybe researchers take their null results and wiggle them around until they reach significance. (I think this is happening to some degree in this literature.)

Surprisingly, this was the criticism that drew the most scoffing from the hosts. University scientists don't earn anything, they argued, so who in their right mind would go into science and twist their results in hope of grant funding? Anyone wanting to make money would have an easier time of it staying far away from academia and going into something more lucrative, like dog walking.

Clearly, the hosts are mistaken, because we know that research fraud happens, publication bias happens, and p-hacking happens. Andrew Gelman's blog today suggests that these things happen when researchers find themselves chasing null hypotheses: due to publish-or-perish pressures, researchers have to find statistical significance. But why does anybody bother?

If the choice is between publishing nonsense and "perishing" (e.g., leaving academia to take a significant pay raise at a real job), why don't we see more researchers choosing to perish?

1 comment:

  1. "If the choice is between publishing nonsense and "perishing" (e.g., leaving academia to take a significant pay raise at a real job), why don't we see more researchers choosing to perish?"

    I have done a research master in BS ("behavioural science" or "bullsh#t" depending on how you want to look at it) and i am still thinking about what i can do with it. The only option seems to be to enter academia (try and find a PhD) which i don't want to do anymore (i've lost nearly all faith in academia).

    Perhaps "perishing" means flipping burgers somewhere. I fail to see all the options for a "real job" (let alone combined with a significant pay raise) with my kind of diploma. As far as i can see, it's all been a giant waste of time and energy. But maybe that's just me.

    On a more serious, and hopefully useful, note: i think, and hope, that better research practices (i.e. pre-registration, high power, etc.) will finally make clear which effects are likely to be real and which are not. I think the researchers in the kind of fields Gelman talks about will then *have* to switch over to other, hopefully more genuine, effects and more fruitful topics of interest.

    So my question is not "why don't we see more researchers choosing to perish?" but "how do we get to see more researchers switching research topics/ leaving fields in which they are basically chasing noise".