1
$\begingroup$

Unofficially, biology researchers always complain how frequently fail to replicate results others have achieved. But of course, such failed experiments are underreported.

Is there data, or at least well-reasoned guesses, on what is the success rate of experiment replication in preclinical models? I don't mean translational research and the impossibility to replicate a mouse result in a human. My question is about failing to reproduce the results within the same model system, replicated at the detail level available in the original publication (e.g. same animal strain, same drug dosages, etc.)

|improve this question
$\endgroup$
  • $\begingroup$ Have you checked out plos.org/wp-content/uploads/2015/06/journal.pbio_.1002165.pdf yet? It may be a good place to start looking. $\endgroup$ – Ben Sheller Sep 23 '15 at 5:45
  • $\begingroup$ I searched for maybe an hour before posting, and while I found some papers declaring that reproducibility is a problem, none went into details. I'll look closer into the one you posted @BenS. $\endgroup$ – Rumi P. Sep 23 '15 at 10:54
1
$\begingroup$

The Cochrane Collaboration has a great deal of this type of analysis. One of my favourite features of a Cochrane Review is the routine use of Funnel Plots, where there are sufficient data to produce them. Sensible interpretation of funnel plots can give some fairly strong hints about the reproducibility of published literature.

Funnel plots show datapoints of individual published studies which investigated the effectiveness of a technique when used for a specific purpose. The datapoints are plotted by estimated effect size (on the x axis) and an assessment of the quality of the trial, incorporating things like sample size and experimental design (on the y axis). A study with a small sample size and/or poor design (low on the y axis) would be expected to provide a relatively inaccurate estimate of the method's true effect size. As sample sizes increase and experimental methodology gets better (travelling up the y axis), you expect the studies to zero in on the true effect size, giving a pyramid or 'funnel' shape to the overall plot.

The reproducibility of trials affects where the points are in the pyramid in a couple of ways.

If there were a bunch of studies which looked for an effect of the method, and didn't find one, they might not be published because null results are considered boring by journal editors - the so-called 'file-drawer effect'. This would show up in a funnel plot as a pyramid that has a suspiciously low number of datapoints on the side of the pyramid that is close to the zero effect-size. The 'missing' datapoints are an indicator that this technique probably did not always reproduce an effect size as strong as the published effect sizes.

The other way that funnel plots can inform on the reproducibility of published studies is simply by looking at the width of the base of the pyramid. If it's wide, then with the experimental design methods of the studies at the bottom, the results aren't highly reproducible - different studies give very different results when looking at the same question.

It's a big job to synthesise this into a global assessment of 'the reproducibility of pre-clinical trials', but if you're interested, I'd recommend you spend some time trawling the Cochrane Library to get a hang of the sorts of variance you're dealing with.

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ PS: the fact that synthesising reproduction efforts into a combined analysis is a big job has not prevented it happening for high-profile psychology journals recently - with a fairly alarmingly low reproducibility rate. sciencemag.org/content/349/6251/aac4716 $\endgroup$ – bshane Sep 24 '15 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.