SCIENCE NEWS – After eight years, a project that tried to reproduce the results of key cancer biology studies has finally concluded. And its findings suggest that like research in the social sciences, cancer research has a replication problem.
Researchers with the Reproducibility Project: Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012.
But only a quarter of those experiments were able to be reproduced, the team reports in two papers published December 7 in eLife.
The researchers couldn’t complete the majority of experiments because the team couldn’t gather enough information from the original papers or their authors about methods used, or obtain the necessary materials needed to attempt replication.
What’s more, of the 50 experiments from 23 papers that were reproduced, effect sizes were, on average, 85 percent lower than those reported in the original experiments. Effect sizes indicate how big the effect found in a study is.
“If it turns out that the science on which a drug is based is not reliable, ‘it means that patients are needlessly exposed to drugs that are unsafe and that really don’t even have a shot at making an impact on cancer.’”
For example, two studies might find that a certain chemical kills cancer cells, but the chemical kills 30 percent of cells in one experiment and 80 percent of cells in a different experiment. The first experiment has less than half the effect size seen in the second one.
The team also measured if a replication was successful using five criteria. Four focused on effect sizes, and the fifth looked at whether both the original and replicated experiments had similarly positive or negative results, and if both sets of results were statistically significant.
The researchers were able to apply those criteria to 112 tested effects from the experiments they could reproduce.
Ultimately, just 46 percent, or 51, met more criteria than they failed, the researchers report.
“The report tells us a lot about the culture and realities of the way cancer biology works, and it’s not a flattering picture at all … ”
