Journal of Obesity & Eating Disorders

Editorial

You may have seen (and ignored) reports in the popular press regarding an article purporting to discuss the relationship between the upcoming American election and strategies in a well-known card game [1], proving that Poe’s law is alive and well. However, this work by Jonathan Falk and Andrew Gelman is a satire of the biases and framings found within certain types of offending research that still make it into the body scientific literature, despite the presence of peer review. If you missed it, it is well worth a read, along with their accompanying interview with the folks at Retraction Watch [2]. No, really, go and read it. I’ll wait.

If you’re like me, this satirical work evoked more than few laughs, but mainly a big ball of distress, lying heavy in the gut like an undigested all-you-can-eat buffet. While pushed to ridiculous extremes for comedic effect, the themes and points made by Gelman and Falk are becoming uncomfortably common. As a biostatistician specializing in nutrition and obesity research and research reporting, I have seen first-hand the prevalence of unfounded conclusions and generalizations, improper design and a startling lack of reproducibility (most recently and prominently in the field of psychology [3], but coming soon to a discipline near you) in recent publications. So, as a means of introducing myself to the readership of the Journal of Obesity and Eating Disorders, I’d like to comment on the concerns I always have in mind when reading a new paper or reviewing a manuscript for publication:

Know the limitations of your research population

The goal of research is generalizability, to use the observation in the few to extrapolate to the behavior, characteristics and maladies of the many. So it is important to know and acknowledge how the study population differs from the broader population of interest. It does no one any good to assume that your subjects, like Falk and Gelman’s elite bridge players, are “otherwise completely typical”.

Know the limitations of your research design

While randomized controlled trials are considered the gold standard for individual studies, they are not the be all and end all of scientific research and indeed may not be feasible when conducting research on topics such as eating disorders, for both practical and ethical reasons. But when conducting observational research, it is important to know and acknowledge the limitations of your chosen design: Case studies and one-sample pre-post designs cannot compare their results to those that would be observed in a comparable control group, so how ‘real’ the observed effect is faces stronger scrutiny. Case-control studies address this issue, but can only show association, not correlation. Openly addressing these limitations can help turn a weakness into strength.

Systematic reviews and meta-analyses are only as good as their constituent studies.

This is related to the previous item, but it deserves a point of emphasis. The ‘systematic’ in a systematic review and metaanalysis is what separates it from the literature search that goes into any paper’s introduction: It must be well-defined and exhaustive in scope. Even then, the review and its analysis will still reflect any biases inherent in the body of the published literature, such as white-hat bias.

Missing data are data pre-processing decisions are important.

The reasons underlying a lack of reproducibility are only rarely falsification or fabrication. More commonly, decisions regarding how the collected data should be processed before analysis, in particular the choices made regarding chosen endpoints and the handling of outliers and missing data can drastically change the conclusions reached after analysis. Thus is it important to be aware of what pre-processing decisions were made and report them in the methods section, or perhaps as part of an appendix or other supplementary materials. For a deeper discussion of the more unintuitive aspects of this problem, the reader might consider reading another narrative authored by Gelman and Loken [4].

Consult with a statistician early and often

While statisticians are (unfortunately) usually only consulted at the analysis portion of the research process, we can also help with the design and execution stages, so that the research is well-guided and powered for its intended task. This goes a long way towards avoiding unnecessary hardship after weeks, months or even years of laborious data collection. As Sir R. A. Fisher once put it, “To consult a statistician after an experiment is finished is often merely to ask him to conduct a post-mortem examination” [5]. We want the best possible science to survive through to publication, too.

In summary, we should all be focused on doing what we can to improve the state of our scientific fields, be that as a researcher, a reviewer or as an informed citizen. Keeping the aforementioned issues in mind is a simple, but important, first step towards that goal.

References

1. Falk J, Gelman A (2016) NO TRUMP! A statistical exercise in priming. 33rd International Conference on Machine Learning, Workshop on Data-Efficient Machine Learning, New York City. https://drive.google.com/file/d/0B0VXvxUNyiVSWjhFTXRZQkVLY1E/view?pref=2&pli=1
2. McCook A (2016) Trump vs. trump: Does the candidate affect the use of trump cards in Bridge? Retraction Watch. https://retractionwatch.com/2016/06/24/trump-vs-trump-does-the-candidate-affect-the-use-of-trump-cards-in-bridge/
3. Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349: 6251.
4. GelmanA, Loken E (2014) The Statistical Crisis in Science.American Scientist102: 460.
5. Ronald A, Fisher (1938) Presidential Address to the First Indian Statistical Congress.