Immune assay harmonization and external quality assurance ......Immune assay harmonization and...
Transcript of Immune assay harmonization and external quality assurance ......Immune assay harmonization and...
Immune assay harmonization and external quality assurance:
Results of the ongoing Proficiency PanelProgram of the Cancer Vaccine Consortium
Sylvia Janetzki M DSylvia Janetzki, M.D.ZellNet Consulting, Inc.
Introduction – CVC (1)Introduction CVC (1)
I th it id b hi i ti• Immunotherapy community-wide membership organization• Run under Sabin Vaccine Institute umbrella (2002-2007) • Now program of the Cancer Research Institute (CRI)
VisionTo cure and prevent cancer throughTo cure and prevent cancer through the advancement of cancer vaccines and immunotherapeuticsand immunotherapeutics.
Cancer Vaccine Consortium - History
2002 2003 2004 2005 2006 2007
Immuno-Assessment
FDA2nd ElispotCVC 1st Elispot
Satellite Symposium
Colloquium and Annual Meeting
FDA Workshop
2 Elispot Panel
3rd Elispot,
CVCFounded
1 Elispot Panel
Potency Clinical Trials p ,1st ICS, CFSE, and Multimer
yTestingWorkshop
Commercialization
Clinical TrialsWorking Group
Strategy Meeting
Phase 3 Clinical TrialsWorkshop
Proficiency Panel Program ObjectivesProficiency Panel - Program Objectives
• To offer an external assay validation program• To enhance assay harmonizationy
Goal:Goal:
Establish immune assays as standardEstablish immune assays as standard monitoring tools in clinical trials and clinical
ti ttiroutine settings
R i d R f h it tiReminder: Reasons for hesitationmeasuring immune endpoints
1 L k li it d k l d b t1. Lack or very limited knowledge about correlation with clinical outcome
2. Lack of confidence in data obtained from bioassays Validity of data
1. Correlates of protection: there are examples
Dudley: Science 2002; Melanoma (repopulation with TILs – tumor regression)
Morgan: Science 2006; Melanoma (transfer of genetically modified lymphocytes – cancer regression)modified lymphocytes cancer regression)
Todryk: PLos 2008, Correlation of memory T cell response and protection against Malaria
Forrest: Clin Vaccine Immunology 2008; CorrelationForrest: Clin Vaccine Immunology 2008; Correlation vaccination – cellular immune response (Elispot) – protectionagainst Influenzaagainst Influenza
Challenges• Lack of consistent correlation (countless publications)
g
• Limited knowledge about Tcell mediated efficacy (Merck STEP trial)
• Rather qualitative than quantitative monitoring (Appay)
• Polyfunctional effector cells (Darrah, Roederer)y ( , )
• Migration of effector cells to tissue (DeVries/tetramers)
Maximum recovery of relevant immunological data
Reliable immune monitoring techniques
2 C fid i d t bt i d f bi2. Confidence in data obtained from bio-assays
Negative example: Lack of acknowledgment of immune monitoring data from Sipuleucel-T trial (phase 3; Small 2006) d it i ifi t l ti ith i ti t tdespite significant correlation with vaccination status Reason: lack of bioassay validation (proliferation assay)
Do YOU believe immune monitoring data published?Always?Always?
Data need to be reliable, repeatable, plausible= VALID= VALID
Validity of DataValidity of Data: Ensure that the results of a study involvingGLP y g
human subjects areReliable– Reliable
– Repeatable– Auditable– RecognizablegGLP includes training, SOP, Reagents and instruments audits and external qualityinstruments , audits and external quality assurance.
= proficiency panel testing
Program ObjectivesProgram ObjectivesCVC Proficiency Panel
• To offer an external assay validation program• To enhance assay harmonizationTo enhance assay harmonization
Goal:
Establish immune assays as standard monitoring tools in clinical trials and clinicalmonitoring tools in clinical trials and clinical routine settings
Establish immune assays as a platform for biomarker discovery and developmentdiscovery and development
Assay Harmonization
T ll bilit f lt l b
• Identification of crucial protocol variables that
To allow comparability of results across labs
pinfluence assay outcome
• Formulation of acceptable and feasibleFormulation of acceptable and feasible recommendations to the field
• Enhancement of their implementation and• Enhancement of their implementation and confirmation of their usefulnessA li ti f h i d i li i l t i l• Application of harmonized assays in clinical trials
The CVC Proficiency panel program
The CVC Proficiency Panel Program2005: 1st Elispot panel (36 labs, 10 countries)
The CVC Proficiency Panel Program
2006: 2nd Elispot panel (29 labs, 7 countries)2007: 3rd Elispot panel (35 labs)
1st Multimer panel (29 labs)1st ICS panel (28 labs)
10 countries
1st CFSE panel (21 labs)
Central lab: PBMC (pretested), antigen, shippingScientific panel leadersCVC office for central administrationIndependent statistician
Panelists 2005 - 2007Panelists 2005 - 2007
Total: 63 entities (some with multiple labs participating)Total: 63 entities (some with multiple labs participating)
1. Academia/hospitals/non-profit: 332. Biotech/biopharma/services: 193. Big Pharma: 43. Big Pharma: 44. Government / Armed forces: 3(5 Supplies/central lab: 4 )(5. Supplies/central lab: 4 )
Cancer: 46/59 = 78% Infectious diseases: 11/59 = 19% (HIV, TB, other)Other: 2/59 = 3%
Elispot panel: ProtocolElispot panel: Protocol
• 4 pre-tested PBMC from same lot (3 runs, 2 protocols)p ( , p )• Pre-tested CEF and CMV peptide pool from same lot• PBS/DMSO from same lotPBS/DMSO from same lot• Same plate layout (6 replicates; 24 medium alone)• Same number of cells and amount of reagentsSame number of cells and amount of reagents• IFN- Elispot, no pre-stimulation
• Lab-specific protocol, materials and equipment(based on premise that each participating lab has a SOP)(based on premise that each participating lab has a SOP)
• No PHA or PMA/Iono (confluence and counting issues)
Response DefinitionResponse Definition
None: < 10 spots/200,000 PBMCLow: 10 – 50 spots/200,000 PBMCp ,Medium: 51 – 100 spots/200,000 PBMCHigh: >100 spots/200 000 PBMCHigh: >100 spots/200,000 PBMC
Response detection (yes/no):
10 spots/200,000 PBMC 3x background (PBMC alone) 3x background (PBMC alone)
EPP-1EPP 1
F2 all participants (medium strong response – CEF)
4/36 labs outliers = missed to detect > 50% of responses17/36 labs missed to detect low responder against CMV17/36 labs missed to detect low responder against CMV13/36 labs missed to detect low responder against CEF
Challenges to detect crucial protocol variablesChallenges to detect crucial protocol variables
St d d i ith t l h i• Study design with open protocol choices
• Wide range in protocol and reagent choicesg p g
(e.g. 7 ab sources, 10 enzyme sources, 7 readers, cell
i i b 2 d 20 h iresting times between 2 and 20 hours, unique
serum choices for each lab etc.))
• Dispersion of protocol choices small sample sizes
• No adequately powered statistical analysis possible
EPP2Panel participation available for all CVC memberswith sufficient experience
EPP2
with sufficient experience
F2
No outliers4 labs missed to detect the low responder 47% in EPP1 14% in EPP2(+ 5 marginal)
Different protocols – same results
D1
Lab 1 Lab 2 Lab 3Plate PVDF MCE PVDFAntibodies Mabtech Mabtech Own KitDevelopment HRP/AEC AP/BCIP HRP/NovaredDevelopment HRP/AEC AP/BCIP HRP/NovaredSerum Human AB FCS FBSDNase No Yes YesResting No Yes/ON Yes/ONResting No Yes/ON Yes/ONCell counting Hemacyto Hemacyto Automated/GuavaReader Zeiss AID CTL
Similar protocols – different resultsp
D3
Lab 4 Lab 5
Plate MCE MCEAntibodies Mabtech MabtechDevelopment HRP/AEC HRP/AECSerum Human AB AIM-VDNase No NoResting No NoCell counting Hemacytometer HemacytometerCell counting Hemacytometer HemacytometerReader Zeiss Zeiss
Influence of automated cell counter use
1000
100
s pe
r wel
l • No outlier• 2 labs missed
10
Spot
cou
nt • 2 labs missed weak responder
D4-med
D4-CEF
D4-CMV
D3-med
D3-CEF
D3-CMV
D2-med
D2-CEF
D2-CMV
D1-med
D1-CEF
D1-CMV
1Donor & Reagent
DDDDDDDD
Mean cell viabilityGuava users Non-Guava users
Donor 1 80.8 86.7
Donor 2 78.4 85.6p<0 01
Apoptosis!
Donor 3 84.0 89.8
Donor 4 79.2 85.5
p<0.01
Advantage of overnight resting before platingAdvantage of overnight resting before plating
900
700800900
per
wel
l 2 hours rest 20 hours rest
400500600
ot c
ount
p
100200300
Mea
n sp
o
0D1-
CMVD1-CEF
D2-CMV
D2-CEF
D3-CMV
D3-CEF
D4-CMV
D4-CEF
Donor and Reagent
Background reactivity: 0-5 spots
Plate evaluation is operator- and SOP-dependent
Counts Lab X Lab Y Lab Z
Plate evaluation is operator and SOP dependent
Own Central Own Central Own Central
Mean 81.5 3.3 16.7 4.8 9.7 1.7
SD 104.7 1.8 25.9 2.9 5.3 1.6
Median 34.5 3.5 6 5.5 10 1.5
Mi i 3 1 3 1 2 0Minimum 3 1 3 1 2 0
Maximum 270 6 69 9 18 4
Background counts (PBMC alone)
Initial Elispot Harmonization Guidelines
A. Establish lab Elispot SOP for:A1. Counting method for apoptotic cells in order to
determine adequate cell dilution for platingA2. Overnight resting of cells prior to plating
B. Use only pretested serum with optimal signal:noise ratioB. Use only pretested serum with optimal signal:noise ratioC. Establish SOP for plate reading, including:C1 Human auditing during reading processC1. Human auditing during reading processC2. Adequate adjustment for technical artifacts
D. Only let well trained personnel conduct assay
Janetzki et al., CII. 2008 Mar;57(3):303-15
EPP 3 (35): new PBMC, 13 labs new( ) ,(Recommendation to implement harmonization guidelines)
3/22 repeaters missedresponse = 14%p
No outliers
4/13 new labs missed response = 31%
EPP3: Validation PhaseEPP3: Validation Phase
20 labs said that they did comply with all20 labs said that they did comply with all Harmonization guidelines (57%).9 of these labs were “top” performers (9/9 =100%)9 of these labs were top performers (9/9 =100%)
15 labs said they did not comply with all15 labs said they did not comply with all Harmonization guidelines (43%).5 of those labs missed (a) response(s) (=33%).5 of those labs missed (a) response(s) ( 33%).
13 labs introduced changes to their protocol in g presponse to the harmonization guidelines (37%).
- 7 added overnight restingg g- 2 changed medium/serum
Overnight rest of cellsOvernight rest of cells
EPP1 EPP3
T t l # f l b 36 35Total # of labs 36 35
Yes 17 (47%) 29 (83%)Yes 17 (47%) 29 (83%)No 19 (53%) 6 (17%)
>8hrs 10 (59%) 22 (76%)<8 hrs 7 (41%) 7 (24%)
Automated cell counter useAutomated cell counter useApoptosis assessment
EPP1 EPP3
Total lab# 36 35
Yes 7 (19%) 14 (40%)No 29 (81%) 21 (60%)
Yes- but missedResponse(s) 2 (29%) 0 (0%)Response(s) 2 (29%) 0 (0%)
Confirmation of guidelinesConfirmation of guidelines
• 7/13 responses missed due to serum choice(high background reactivity)( g g y)
• 4/13 responses missed due to plate reading strategiesstrategies
Next round (EPP4):Next round (EPP4):• Tackle of serum issue • Integration of guidelines
Serum TaskSerum TaskGoal: to identify at least 1 serum-free medium thatperforms at least equally well as lab’s medium/serumWho: “Top” performersWho: Top performersWhat: Own medium/serum, 3 serum free media( C )(ExVivo, AIM-V, CTL)How: EPP pretested PBMC and antigenSame setup for all 4 platesSame protocolSame protocolSame experiment
CVC Multimer panel 1CVC Multimer panel 1Scientific leaders: Pedor Romero, LICR/Lausanne
& Cedrik Britten, Leiden
• 27 labs from 7 countriesPBMC f 5 t t d d ith i ti it
,
• PBMCs from 5 pre-tested donors with various reactivity against HLA-A2-restricted
Influenza M1 58 66 (GILGFVFTL)- Influenza-M1 58-66 (GILGFVFTL)- Melan A/Mart-1 26-35 (ELAGIGILTV)
St d d lti ( ti ) id d• Standard multimers (antigens) were provided:• Tetramers from Beckman Coulter
P t f P I• Pentamers from ProImmune• Labs used their own staining protocol
MPP1 OutcomeMPP1 - OutcomeThree clear recommendations could be deducedThree clear recommendations could be deduced(Britten, Romero; manuscript in preparation)
Next steps:• Integration of new recommendations
• Discussion about background controlDiscussion about background control
• Systematic examination of protocol variablesSystematic examination of protocol variables identified which might have influence on final resultsresults
ICS Panel 1: Rude awakening
Panelist received:
Scientific leader: Holden Maecker & Maria Jaimes, BD
Panelist received:1) PBCMs 3 donors with different CEF and CMVpp65 reactivity 2) Lyoplate containing antigens for stimulation of cells (triplicates)2) Lyoplate containing antigens for stimulation of cells (triplicates)3) Lyoplate containing staining reagents:a) CD4 FITC / IFNg+IL-2 PE / CD8 PerCP-Cy5.5 / CD3 APC b) Comp Control beads
and instructions for acquisition and analysis using those beads4) EDTA FACS Lysing solution FACS Permeabilizing Solution24) EDTA, FACS Lysing solution, FACS Permeabilizing Solution2
Highly standardized protocol and reagentsNote: protocol and reagents validated at BD and via NIH ICS panel
ICS panel 1ICS panel 1“Gold standard”
Example1Example1
Example2CD4/LP13
Next Steps ICSNext Steps - ICSThe 1st ICS panel has shown: it is not a matter of using “someone's”p gvalidated protocol and reagents - own validation, training per ownSOP, experience and own standards are key!
• Standardized protocol
Own protocol
Fi t d ti• First recommendations:
Compensation and gating strategies p g g gBackground control issue Experience? SOPs?
CFSE panelp
Example: CD8 - all
Scientific leader: Jean Boyer, Dan Schullery, UPenn
8090
LP7 LP19 LP27(negative)p
4050607080
0102030
0.89 0.97 0.040 0.76 0.51 0.003 0.64 0.58 0.011pp(Relative to DMSO)(Relative to DMSO)
17 labs: 3 donors against CMV pp65 and CEFProtocol recommendation, but only few strictrequirements
Challenges for CFSE panelChallenges for CFSE panel
• Huge variability• Huge variability• Huge variability• Very few acceptable responses observed• Huge variability• Very few acceptable responses observed• How much do we know about the assay and
its applicability in immune monitoring of • How much do we know about the assay and
its applicability in immune monitoring of cancer vaccines?
• Diverse sera sources (serum known to be of cancer vaccines?
• Diverse sera sources (serum known to be of (huge influence in CFSE assay – NIAID panel)
• Vague gating strategies
(huge influence in CFSE assay – NIAID panel)
• Vague gating strategiesVague gating strategiesVague gating strategies
Discontinued due to questionable positionin cancer field
SummarySummary
• Proficiency panel program has shown to be of• Proficiency panel program has shown to be of high value for external quality assurance and for assay harmonizationassay harmonization
• First harmonization guidelines for Elispot• First recommendations for Multimer and ICS• First recommendations for Multimer and ICS• CFSE with questionable current position in
cancer vaccine monitoringcancer vaccine monitoring• Further systematic examination of crucial
protocol variables ongoingprotocol variables ongoing• Open for collaboration
AcknowledgementAcknowledgement• The Cancer Vaccine Consortium, its Executive,
Council, Executive office, and umbrella organizationorganization.
• The Steering Committee of the Assay Working GroupGroup.
• All supporters and suppliers (BD, Beckman C lt CTL NIH/ Di i i f AIDSCoulter, CTL, NIH/ Division of AIDS, Proimmune, Seracare).
• Our participants.