John Carlisle, a consultant anesthetist at Torbay Hospital in the UK, used statistical tools to conduct a review of thousands of papers published in leading medical journals. While a vast majority of the clinical trials he reviewed were accurate, 90 of the 5,067 published trials had underlying patterns that were unlikely to appear by chance in a credible dataset.
While some of these errors may be the result of “misinterpretation, statistical error, or plain simple mistakes,” Carlisle says in a press release emailed to The Scientist, it’s likely that some of the research in question “may have been deliberately falsified.” He published his results on June 5 in the medical journal Anaesthesia.
The tool works by comparing the baseline data, such as the height, sex, weight and blood pressure of trial participants, to known distributions of these variables in a random sample of the populations.
If the baseline data differs significantly from expectation, this could be a sign of errors or data tampering on the part of the researcher, since if datasets have been fabricated they are unlikely to have the right pattern of random variation.
“This study is about working to correct the scientific record,” Andrew Klein, the editor-in-chief of the journal, said in a press release. “The new Carlisle screening tool has been developed here in the U.K. and it is now clear that it should be used by medical publications around the world. The integrity of medical science demands that we do everything we can to ensure complete accuracy in the publication of research.”
In the case of Japanese scientist, Yoshitaka Fuji, the detection of such anomalies triggered an investigation that concluded more than 100 of his papers had been entirely fabricated.
The latest study identified 90 trials that had skewed baseline statistics, 43 of which with measurements that had about a one in a quadrillion probability of occurring by chance.
The published review includes a full list of the trials in question, allowing Carlisle’s methods to be checked but also potentially exposing the authors to criticism. Previous large scale studies of erroneous results have avoided singling out authors.
Relevant journal editors were informed last month, and the editors of the six anesthesiology journals named in the study said they plan to approach the authors of the trials in question, and raised the prospect of triggering in-depth investigations in cases that could not be explained.
The flagged studies came from eight journals, including the New England Journal of Medicine, Anaesthesia, and JAMA. As Klein tells Retraction Watch, all the journals responded “swiftly and positively” to Carlisle, who reached out with the findings of his study. Six of the journals in the new analysis are in anesthesiology, where Carlisle has focused his efforts for several years, refining the method since 2012.
That earlier work led to the revelations that much of Yoshitaka Fujii’s research was fraudulent, and Fujii now has made 183 retractions of his voluminous “research”. The method has also been used by others to identify issues in more than 30 papers by bone researcher Yoshihiro Sato.