the purpose of this brief editorial comment is to highlight the importance of what is regrettably termed “negative data.” The definition of negative data is straightforward: the failure to reject the null hypothesis. But is such an outcome any less important than the alternative? If not, then why are negative findings so difficult to publish? The answer to the first question, in our view, is a resounding no, as explained below. The answer to the second question is complex and elusive.
For many years, scientific journals have been judged by their impact factor, a simple metric that favors citations that occur in the first 2 years after publication. This leads journals to favor articles deemed “exciting” and “high profile”, with the hope that these papers will be cited soon after publication and frequently. Unfortunately, the holy grail of “high impact” has, at times, favored glitz over scientific rigor. Indeed, having the results of your study broadcast on NPR has become, for many, the ultimate scientific status symbol. The unfortunate side effect of this practice is that scientists have been raised to believe that the only work worthy of doing must be completely novel and glamorous. This, in turn, leads to additional problems that are damaging to the scientific enterprise. For one, studies designed to reproduce published findings are extremely rare, despite a growing consensus that replication is important. As a result, it is likely that scientific journals are filled with false-positive results, as addressed in an editorial in Nature Reviews Drug Discovery (2). In addition, the quest for novelty and glitz often biases experiments toward sexy techniques, rather than important questions (in this context, we note that Albert Einstein did rather well with paper and a chalkboard). The third problem is that other scientists often pursue areas of inquiry that have been prejudged as “exciting”, potentially leading to a waste of scarce resources. And at its worst, the quest for positive results leads to abhorrent behaviors, such as outright manipulation of data, deletion of outliers, and the use of inappropriate statistical models.
With this as a backdrop, we wish to highlight a recent paper published in the Journal of Applied Physiology (1). The authors showed that identical gains in muscular strength and size could be achieved by lifting either very light or very heavy weights under conditions where subjects in both groups performed each exercise to the point of volitional muscle fatigue. The study also failed to find any significant relations between postexercise changes in anabolic hormones. In fact, the only difference was a slightly larger improvement in bench press strength in the subjects that lifted lighter weights. These results, based almost entirely on negative findings, are important for many reasons. For example, lifting heavy weights can be dangerous, especially in older people who may have connective tissue pathology or a lack of coordination. In addition, resistance exercise training is widely believed to be an important component of overall physical fitness, but lifting heavy weights can be intimidating if not painful, preventing many people from initiating this type of exercise.
Fortunately, the importance of this “negative study” was quickly unearthed by the popular press, with summaries of the work published in many news outlets, including the New York Times. If this study had been rejected as “negative” or “unexciting”, it is possible that these important observations would not have been disseminated.
Interestingly, the exposure that this study has generated begs the following question: Will other investigators try to reproduce these “negative” findings? In our view, such a study is crucial, because identifying a false negative is, of course, just as important as uncovering a false positive. And, if the negative results are subsequently confirmed, will journals publish the work? The key point is that dissemination of data from well-designed experiments is the lifeblood of good science, whether the results retain or reject the hypothesis. Moreover, replication of both positive and negative results is equally important. Indeed, these best practices can lead to excitement in the scientific community, because disagreement between laboratories is eventually settled by rigorous peer review that is laser focused on establishing the truth.
The negative studies that we avoid publishing are those where the statistical power is insufficient to distinguish a falsely negative outcome from a truly negative outcome. In fact, for us, an indeterminate conclusion is the proper definition of a negative study, and we hope the scientific community abandons the term “negative” for an adequately powered study that shows no group differences in their outcome variables.
No conflicts of interest, financial or otherwise, are declared by the author(s).
R.F.F. drafted manuscript; R.F.F. and P.D.W. edited and revised manuscript; R.F.F. and P.D.W. approved final version of manuscript.
- Copyright © 2016 the American Physiological Society