Null Results Aren't Necessarily Void

It's clear to see a study’s impact when its results are significant or represent a clear departure from conventional wisdom.
Null Results Aren't Necessarily Void

But what if researchers find nothing instead? Often, those results are discarded.

When faculty don’t publish studies with null or insignificant results, it’s bad for scholarship, say three researchers from Stanford University in California. They include Neil Malhotra, professor of political economy at the Stanford Graduate School of Business, as well as lead author Annie Franco and Gabor Simonovits, both of Stanford University’s department of political science.

The group analyzed 249 studies conducted between 2002 and 2012 as part of the Time-sharing Experiments for the Social Sciences (TESS) program. They disqualified incomplete studies and those that had appeared as book chapters. The remaining 221 studies were placed into three categories: strong, with supported hypotheses; null, with little or no support; and mixed. Sixty percent of “strong” studies and 50 percent of “mixed” studies were published, compared to only 20 percent of “null” studies.

The researchers see three problems with this outcome. First, if studies with null results aren’t published, many other scholars waste time repeating unsuccessful experiments. Second, if other researchers conduct the same studies but have statistically significant results, the discrepancy between their studies and the unsuccessful studies is never known. Finally, without knowledge of null studies, researchers might lose objectivity and push for results that aren’t there.

The authors call for the creation of more “high-status publication outlets” for studies with null results and a requirement for researchers to pre-register studies. They argue that academia needs to do more to convince scholars “not to bury insignificant results in file drawers.”

“Publication bias in the social sciences: Unlocking the file drawer” was published in the September 19, 2014, issue of Science.