Sick and Tired of Bias-Correction

Rick Deckard, wishing he could've stuck to experimental research. (Blade Runner, 1982)

At present, a tremendous amount of work is going into the development, refinement, and application of tools for the detection of, and correction for, research bias. We now have funnel plots, meta-regression, trim-and-fill, p-curve, p-uniform, selection models, Bayesian selection models, R-index, the Test for Excess Significance, and the Test for Insufficient Variance, to name only a few.

I like these tests because they've helped me to question some research findings that I don't think are quite right. But at the same time, the thought of having to spend the rest of my life performing meta-analyses and using these tools is deeply exhausting to me. Dr. R seems to enjoy what he's doing, but the idea of churning out test after test after test for all eternity does not seem very pleasant.

It's tedious because these tests are all imperfect. Each has its own assumptions of how the data-gathering process works, and those assumptions are often woefully mistaken. Each performs well in a sample of about, oh, 500 perfectly homogeneous studies, which is a problem when most social psychology literatures can muster about 30 studies of questionable homogeneity.

These tests' results are helpful, but they'll never recover the actual raw data. No Egger test can reveal that someone's dissertation started out as N = 410 in a 2 × 2 × 4 design with two outcomes and worked its way down to N = 140 in a 2 × 3 design with one outcome. You'll never recover the actual research findings.

It's for this reason I think that scientific reform is far, far more important than any amount of meta-analytic data-sleuthing. Data-sleuthing lets you rescue hypotheses when you're already 10 or 20 years deep in a research program, maybe improving your estimates by 25-50% or so. But transparent and principled science is the only way to get it right the first time.

So if you ask me, I'd rather everybody played fair from the start. Playing detective is a lousy way to spend one's time. Let's please publish our null results, all our outcomes, our full datasets. Let's abstain from the opportunism of advocacy or the craven greed of hunting for effects to claim as our own. Let's instead assume the impartial and disinterested stance expected of scientists. Papers are so much more fun to write when they're honest, and so much more fun to read, too.

In the meantime, the meta-analyses will probably have to continue until everybody is sufficiently embarrassed that we agree to embrace reform. I hope it won't take much longer.