Reeve-Irvine Research Center Replication and Reproducibility in Spinal Cord Injury Research.

Reeve-Irvine Research Center

Replication and Reproducibility in Spinal

Cord Injury Research


Take home

There has been a preponderance of failures to There has been a preponderance of failures to replicate.replicate.

Lack of replication is not a bad thing.Lack of replication is not a bad thing. It can lead to critical adjustments in approach It can lead to critical adjustments in approach

and save the field huge amounts of money and save the field huge amounts of money pursuing false leads.pursuing false leads.

Non-replication of early findings is part of the Non-replication of early findings is part of the natural history of discovery (Kevin Staley).natural history of discovery (Kevin Staley).


The Problem:

Many reports of treatments that improve Many reports of treatments that improve outcome after SCI; No translation. Why?outcome after SCI; No translation. Why?

Rumors: “We repeated that experiment and it Rumors: “We repeated that experiment and it didn’t work”.didn’t work”.

Failure of clinical trials for a variety of Failure of clinical trials for a variety of disorders including stroke and TBI.disorders including stroke and TBI.


Program Officer, Naomi Kleitman, NINDSProgram Officer, Naomi Kleitman, NINDS Contract Officer, Laurie Leonard, NINDSContract Officer, Laurie Leonard, NINDS

PI’s and SitesPI’s and Sites Oswald Steward (UC Irvine, 2003, 2008)Oswald Steward (UC Irvine, 2003, 2008) Dalton Dietrich (U. Miami, 2003)Dalton Dietrich (U. Miami, 2003) Philip Popovich (Ohio State U, 2008)Philip Popovich (Ohio State U, 2008)

In recognition of the problem, NINDS launched In recognition of the problem, NINDS launched the FACILITIES OF RESEARCH the FACILITIES OF RESEARCH EXCELLENCE IN SCI (FORE-SCI) contracts, EXCELLENCE IN SCI (FORE-SCI) contracts,


NIH buys a service / deliverableNIH buys a service / deliverable

NIH stipulates the scope and desired productNIH stipulates the scope and desired product Faithful replication of published studiesFaithful replication of published studies Facilities provide, in one location, resources, capabilities Facilities provide, in one location, resources, capabilities

and expertise in SCI researchand expertise in SCI research Activities are defined; conduct of additional studies is Activities are defined; conduct of additional studies is

limitedlimited

Advisory Committees advise PI and NINDS about studies Advisory Committees advise PI and NINDS about studies chosen for replicationchosen for replication

Slide from Naomi KleitmanSlide from Naomi Kleitman

Contracts are different from grantsContracts are different from grants


SSpecific performance goals of the Contracts:pecific performance goals of the Contracts:

Try to replicate promising, preclinical studies relating to therapies that Try to replicate promising, preclinical studies relating to therapies that could lead to effective treatments for human SCI,could lead to effective treatments for human SCI,

Compare the efficacy of treatments in a standardized environment with Compare the efficacy of treatments in a standardized environment with a minimum of variability in surgery, animal care, outcome evaluation a minimum of variability in surgery, animal care, outcome evaluation and cellular analyses,and cellular analyses,

Promptly report the methodology and results.Promptly report the methodology and results.

The desired result is that, if proven to provide reliable and robust benefit, The desired result is that, if proven to provide reliable and robust benefit, these promising strategies would be appropriate to move to the next level of these promising strategies would be appropriate to move to the next level of translation or, if appropriate, clinical testing.translation or, if appropriate, clinical testing.

If studies are NOT reproducible, this could save $millions that would If studies are NOT reproducible, this could save $millions that would otherwise be spent on dead ends otherwise be spent on dead ends and failed clinical trialsand failed clinical trials..


FORE-SCI – Replication studiesFORE-SCI – Replication studies


Clinically relevant endpoints (usually means sparing or Clinically relevant endpoints (usually means sparing or recovery of function).recovery of function). Is treatment potentially translatable to the clinic?Is treatment potentially translatable to the clinic? Some were already in or on the way to clinical trialsSome were already in or on the way to clinical trials

Degree of improvement (effect size)Degree of improvement (effect size)

Scientific merit of the publicationScientific merit of the publication

General strengths and weaknessesGeneral strengths and weaknesses


Criteria for study selectionCriteria for study selection


Surprising preponderance of failures to replicate (1/12) Surprising preponderance of failures to replicate (1/12) What does it mean to the field?What does it mean to the field?

Methods sections are often incomplete or misleadingMethods sections are often incomplete or misleading Randomization is rarely explained and often is NOT DONE.Randomization is rarely explained and often is NOT DONE. Communication with original authors is essential, but often Communication with original authors is essential, but often

reveals that the experiment was NOT done as the Methods reveals that the experiment was NOT done as the Methods imply.imply.

Significant technological hurdlesSignificant technological hurdles Reproducibility of SCI models; control deficit levels. Reproducibility of SCI models; control deficit levels.

Publishing negative results is doable and generally well-Publishing negative results is doable and generally well-received by the field.received by the field.


Findings to date:Findings to date:


Important methodological issues:

Many papers describe work carried out over a period of Many papers describe work carried out over a period of several years. Groups were not run simultaneously. several years. Groups were not run simultaneously. There is no description of this in Methods. There is no description of this in Methods.

This is true of most SCI experiments, and is This is true of most SCI experiments, and is always always true true when there are multiple groups involving many when there are multiple groups involving many animals.animals.

Batching of animals/non-simultaneity of group assessment is almost never explained.


Peng et al., PNAS, 2009

It is sometimes impossible to remain blind. Here, treatment turns rats blue!


The file drawer problem. Studies that “work” are published; The file drawer problem. Studies that “work” are published; studies that don’t aren’t. studies that don’t aren’t.

Type I statistical error. Multiple comparisons, only one of which is Type I statistical error. Multiple comparisons, only one of which is significant. significant.

Methodological details that are not reported (non- simultaneous Methodological details that are not reported (non- simultaneous group assessment). group assessment).

Effects are not robust.Effects are not robust. Inadvertent bias: Inadvertent bias:

Unrecognized tendency to be more careful with the Unrecognized tendency to be more careful with the “experimental” group during the surgery for example.“experimental” group during the surgery for example.

Non-random order of surgery/treatment/testing.Non-random order of surgery/treatment/testing. Important or difficult procures may be done first. Important or difficult procures may be done first. Post-operative care is a treatment variable. Post-operative care is a treatment variable.

Why is there a failure to replicate?Why is there a failure to replicate?


PreclinicalPreclinical is any study that tests a biological concept in an animal is any study that tests a biological concept in an animal model of disease.model of disease.

A large percentage of preclinical studies by the above definition are A large percentage of preclinical studies by the above definition are R01-funded.R01-funded.

R01 review does not currently emphasize the importance of replication, R01 review does not currently emphasize the importance of replication, optimization, etc.optimization, etc.

Blinding and randomization requires a larger staff than most R01 grants Blinding and randomization requires a larger staff than most R01 grants can support.can support.

Replication and optimization studies are not career-buildersReplication and optimization studies are not career-builders..

Some pointsSome points

How do we think about failures to replicate?

Does a failure to replicate mean that the basic biology is invalid? Or does it simply mean that the effect depends on experimental details that are not easily recognized? Either way, the important conclusion is that the effects are not robust.

Treatments that do not produce robust effects are unlikely to be translatable.


If it’s too good to be true, it’s probably not true. Extraordinary claims require extraordinary Extraordinary claims require extraordinary

documentation.documentation.

The level of documentation for regeneration after The level of documentation for regeneration after spinal cord injury is difficult to compress into the spinal cord injury is difficult to compress into the space allowed by high profile journals.space allowed by high profile journals.

So, maybe studies reporting regeneration should not So, maybe studies reporting regeneration should not be published in high profile journals.be published in high profile journals. (Except for my studies of course).(Except for my studies of course).


Roadblocks to solutions

NIH review criteria? Optimization and NIH review criteria? Optimization and replication are not “innovative”.replication are not “innovative”.

Academia: Adjust reward structures?Academia: Adjust reward structures?


IACUC issues

Minimizing animal use (thus reducing Minimizing animal use (thus reducing “n”) vs. ensuring sufficient power.“n”) vs. ensuring sufficient power.

IACUC requirements to avoid IACUC requirements to avoid duplication. Replication is by definition duplication. Replication is by definition duplication.duplication.


Some fallacies:

It’s hard to publish negative results.It’s hard to publish negative results.

FALSE: FALSE: Reviewers have been very positive. Reviewers have been very positive. Every replication paper has been Every replication paper has been

accepted. accepted.


Some fallacies:

Repeating an experiment is not interesting, Repeating an experiment is not interesting, especially if the results are negative.especially if the results are negative.

FALSE: FALSE: There is increasing recognition that There is increasing recognition that

reporting negative results is important and reporting negative results is important and interesting.interesting.

And there have been unexpected findings And there have been unexpected findings that add value.that add value.


Some fallacies:

You’ll make enemies.You’ll make enemies.

Hmm; well maybe this isn’t a fallacy.Hmm; well maybe this isn’t a fallacy.

Reeve-Irvine Research Center Replication and Reproducibility in Spinal Cord Injury Research.

Documents

Transcript of Reeve-Irvine Research Center Replication and Reproducibility in Spinal Cord Injury Research.