Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems •...

43
Valida&on of Prognos&c Models based on High Dimensional Data 1

Transcript of Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems •...

Page 1: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Valida&on  of  Prognos&c  Models  based  on  High  Dimensional  Data  

1!

Page 2: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Kinds  of  Valida&on  

•  Analy&cal  valida&on  –  Is  the  assay  reproducible  and  robust?    –  Does  it  measure  the  analytes  accurately?  

•  Clinical  valida&on  – Predic&ve  accuracy  – Sensi&vity,  specificity,  ppv,  npv,  ROC  

•  Medical  u&lity  –  Is  the  test  ac&onable  and  improves  outcome?  

2!

Page 3: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Samples  Required  for  Valida&on  

•  Analy&cal  – Samples  collected  and  handled  as  for  the  intended  use    

– No  clinical  annota&on  required  •  Clinical  – Performance  measures  for  intended  use  – Usually  retrospec&ve  

•  Medical  U&lity  – May  require  prospec&ve  study  

3!

Page 4: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

4!

Page 5: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Commonly  Seen  Problems  

•  Samples  not  collected  for  a  clearly  specified  intended  use  

•  Over-­‐es&ma&on  of  predic&on  accuracy  –  Develop  prognos&c  classifier  of  survival  on  a  dataset  –  Classify  the  cases  of  the  same  dataset  as  good  prognosis  or  poor  prognosis  using  the  classifier  

–  Calculate  Kaplan  Meier  es&mates  of  survival  for  each  risk  group  

5!

Page 6: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

6!

Page 7: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Valida&on  of  a  prognos&c  model  

•  Evalua&ng  predic&on  accuracy  of  a  completely  “locked  down”  model  on  an  independent  dataset  

•  SpliOng  sample  into  training  set  and  valida&on  set  –  Split  by  center  

•  Cross-­‐valida&on  or  bootstrap  re-­‐sampling  to  es&mate  predic&on  accuracy  

7!

Page 8: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  Split  sampling  and  re-­‐sampling  provide  internal  es&mates  of  predic&on  accuracy  that  may  not  reflect  inter-­‐laboratory  assay  varia&on  and  other  sources  of  inter-­‐center  varia&on  

8!

Page 9: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

9!

Page 10: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

LOOCV  for  two  class  predic&on  of  binary  response  

10!

Page 11: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Complete  Cross-­‐Valida&on  

•  Simulates  the  development  of  a  model  on  one  dataset  and  tes&ng  it  on  another  

•  All  aspects  of  the  model  development  process  should  be  repeated  from  scratch  for  each  loop  of  the  cross-­‐valida&on  –  Variable  selec&on  – Model  fiOng  –  Tuning  parameter  op&miza&on  

–  SeOng  threshold  for  classifica&on  

11!

Page 12: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Predic&on  on  Simulated  Null  Data  Simon  et  al.  JNCI:95,14,2003    

•  Genera&on  of  gene  expression  profiles  –  20  specimens,  6000  genes  MVN(0,I6000)  

–  1st  10  specimens  class  1,  last  10  class  2  

•  Predic&on  method  

–  Compound  covariate  classifier  based  on  the  10  most  differen&ally  expressed  genes  

tgxgg selected∑

12!

Page 13: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

13!

Page 14: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  Goodness  of  fit  to  the  data  used  for  building  the  model  is  not  evidence  of  model  accuracy  

•  Sta&s&cal  significance  of  regression  coefficients  or  Kaplan  Meier  curves  is  not  evidence  of  model  accuracy  

•  A  hazard  ra&o  is  a  measure  of  associa&on,  not  predic&on  accuracy;  large  HR  can  mask  poor  predic&on  accuracy  

14!

Page 15: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

15!

Page 16: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  The  cross-­‐validated  misclassifica&on  rate  is  an  almost  unbiased  es&mate  of  the  generaliza&on  error  for  the  classifier  developed  on  the  full  sample  when  applied  to  future  observa&ons  drawn  from  the  same  distribu&on  

•  C(x;T)=classifier  developed  on  training  set  T  •  C(x;D)=classifier  developed  on  full  dataset  D  •  μ(D)=E(x,y)[I{C(x;D)≠y}]  generaliza&on  error  rate  •  Cross-­‐validated  misclassifica&on  rate  ≈  ED[μ(D)]  

16!

Page 17: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  Par&&on  D  into  (T,V).  Develop  classifier  C(  ;T)  •  Evaluate  C(  ;T)  on  V;    

•  For  small  samples,  the  cross-­‐validated  es&mate  is  a  less  biased  es&mator  of  μ(D)  than  the  split-­‐sample  es&mate  and  has  smaller  MSE  

valsplit =1

|V |I C x;T( ) ≠ y{ }

(x,y)∈V∑

= µ̂(T )

17!

Page 18: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

18!

Page 19: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

19!

Page 20: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

20!

Page 21: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

21!

Page 22: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Cross-­‐Validated  Performance  Measures  for  Penalized  Logis&c  Model  Predic&on  

•  Par&&on  full  dataset  D  into  D1,D2,  …,  DK  •  First  training  set  T1=D-­‐D1  

•  Fit  penalized  logis&c  model  to  T1  yielding  model  with  regression  coefficient  vector  β(1)  

•  For  cases  jεD1  compute  predic&ve  score            sj=β(1)xj                        and  response  probability                                  pj=exp(sj)/(1+exp(sj))  

•  Repeat  for  T2=D-­‐D2,  …,  TK=D-­‐DK  

22!

Page 23: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

sensitivity(c) =I(yj = 1)I pj ≥ c( )

j=1

n

∑

I(yj = 1)j=1

n

∑

specificity(c) =I(yj = 0)I pj < c( )

j=1

n

∑

I(yj = 0)j=1

n

∑

ppv(c) =I(yj = 1)I pj ≥ c( )

j=1

n

∑

I(pj ≥ c)j=1

n

∑

npv(c) =I(yj = 0)I pj < c( )

j=1

n

∑

I(pj < c)j=1

n

∑23!

Page 24: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

24!

Page 25: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Cross-­‐Validated  Kaplan-­‐Meier  Curves  for  Survival  Risk  Group  PH  Model  

•  Par&&on  dataset  D  into  D1,D2,…,DK  

•  First  training  set  T1=D-­‐D1  

•  Develop  survival  risk  classifier  (e.g.  penalized  PH)  using  only  T1  yielding  regression  coefficient  vector        β(1)  

•  Compute  prognos&c  scores  β(1)x  for  cases  in  D1  and  for  cases  in  T1  .  For  case  j  in  D1  if                                                                        β(1)xj  <  median{β(1)xi  ,  i  ε  T1}    then  predict  case  j  is  low  risk;  otherwise  high  risk.    

25!

Page 26: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  Repeat  for  T2=D-­‐D2  etc.  

•  Aver  K  loops,  all  cases  have  been  classified  as  low  or  high  risk  using  a  classifier  developed  on  a  training  set  that  they  were  not  part  of.  

•  Compute  Kaplan  Meier  curves  for  the  two  risk  groups.  

26!

Page 27: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

27!

Page 28: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

28!

Page 29: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  To  evaluate  sta&s&cal  significance  of  the  spread  of  the  cross-­‐validated  survival  curves,  the  log-­‐rank  test  cannot  be  used  because  the  pa&ent  group  indicators  are  random  variables  

•  The  null  distribu&on  of  the  log-­‐rank  sta&s&c  can  be  approximated,  however,  by  permu&ng  the  survival  &mes  and  repea&ng  the  en&re  procedure      

29!

Page 30: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

•  For  each  permuta&on,  the  cross-­‐valida&on  is  repeated  and  new  cross-­‐validated  KM  curves  produced.  The  log-­‐rank  sta&s&c  is  calculated.  This  is  repeated  1000  &mes  to  approximate  the  null  distribu&on  of  the  log-­‐rank  sta&s&c.  

30!

Page 31: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Cross-­‐Validated  Time-­‐Dependent  ROC  Curve  

•  Instead  of  compu&ng  cross-­‐validated  risk  indicators,  compute  cross-­‐validated  quan&le  marker  levels  M1,  …,  Mn    

31!

Page 32: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Cross-­‐Validated  Kaplan-­‐Meier  Curves  for  Survival  Risk  Group  PH  Model  

•  Par&&on  dataset  D  into  D1,D2,…,DK  

•  First  training  set  T1=D-­‐D1  

•  Develop  survival  risk  classifier  (e.g.  penalized  PH)  using  only  T1  yielding  regression  coefficient  vector        β(1)  

•  Compute  prognos&c  scores  β(1)x  for  cases  in  D1  and  for  cases  in  T1  .  For  case  j  in  D1                                                                                      Mj  =  rank  of  β(1)xj  in  {β(1)xi  ,  all  i}    

32!

Page 33: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Time Dependent ROC Curve for Survival Data (Heagerty et. Al)

•  Sensitivity(c) vs 1-Specificity(c)

•  Marker M

•  Cut-point c

•  Sensitivity = Pr[M≥c | D=1]

•  Specificity = Pr[M<c | D=0]

Page 34: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Time Dependent ROC Curve for Survival Data (Heagerty et. Al)

•  Sensitivity(c) vs 1-Specificity(c)

•  Marker M

•  Cut-point c

•  Landmark time t*

•  Sensitivity = Pr[M≥c | T≥t*]

•  Specificity = Pr[M<c | T<t*]

Page 35: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Time Dependent ROC Curve

Page 36: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)
Page 37: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

37!

Page 38: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Probabilis&c  Binary  Predic&on  

log p1− p

= β 'x

Penalized likelihood estimates β̂

p̂ = exp(β̂ 'x) / [1+ exp(β̂ 'x)]

38!

Page 39: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

39!

Page 40: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

40!

Page 41: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Well  calibrated  Probabilis&c  Classifier  

p̂−1(u) = x ∍ p̂ x( )∈(u − ε,u + ε){ },u ∈(0,1)

p(x)dF(x | x ∈ p̂−1(u))∫ u

41!

Page 42: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Evalua&on  of  calibra&on  

•  Compute  LOOCV  es&mate                      for  i=1,...,n  •  Plot                      versus  the  propor&on  of  responders  in  cases  j  for  which            

p̂− i (xi )

p̂− i (xi )

p̂− j (x j )∈( p̂− i (xi ) − ε, p̂− i (xi ) + ε)

42!

Page 43: Validaon)of)Prognos&c)Models)based)on) High)Dimensional)Data · CommonlySeenProblems • Samples)notcollected)for)aclearly)specified)intended) use) • OverIes&maon)of)predic&on)accuracy)

Evalua&on  of  calibra&on  

•  Perform  linear  logis&c  regression  of  

•  Slope  1  and  intercept  0  indicates  perfect  calibra&on      

(yi , p̂− i (xi ))

43!


In Search of a King. Gibeah Ramah Bethel AMMONITES Nob Bethlehem Adullam MOAB Keilah W. of Ziph W. of Maon Ziph.

In Search of a King. Gibeah Ramah Bethel AMMONITES Nob Bethlehem Adullam MOAB Keilah W. of Ziph W. of Maon Ziph.

Artikel Prognos diagnose en therapie - Nederlands Centrum voor

Artikel Prognos diagnose en therapie - Nederlands Centrum voor

The German Energy Transition and the Future ofRenewables · 2019. 7. 22. · BCG/Prognos– REF BCG/Prognos – 80% BCG/Prognos– 95% ewi –REV 95% ewi –EV 95% Photovoltaics 90

The German Energy Transition and the Future ofRenewables · 2019. 7. 22. · BCG/Prognos– REF BCG/Prognos – 80% BCG/Prognos– 95% ewi –REV 95% ewi –EV 95% Photovoltaics 90

Automatic)program)generationfor) detecting)vulnerabilities ...maon/teaching/2015-2016/workshop/workshop1516b-intro.pdfAutomatic)program)generationfor) detecting)vulnerabilities)anderrors)

Automatic)program)generationfor) detecting)vulnerabilities ...maon/teaching/2015-2016/workshop/workshop1516b-intro.pdfAutomatic)program)generationfor) detecting)vulnerabilities)anderrors)

A Renaissance of Nuclear Energy? - Prognos

A Renaissance of Nuclear Energy? - Prognos

ANALOG UND DIGITAL - Prognos

ANALOG UND DIGITAL - Prognos

Prognos culture guide carta libertatum

Prognos culture guide carta libertatum

Validaon of STAR‐CCM+ with the OECD/NEA T‐Juncon Blind ...mdx2.plm.automation.siemens.com/sites/default/... · STAR European Conference, 22-23 Mar 2011 16 Results Mean & RMS ...

Validaon of STAR‐CCM+ with the OECD/NEA T‐Juncon Blind ...mdx2.plm.automation.siemens.com/sites/default/... · STAR European Conference, 22-23 Mar 2011 16 Results Mean & RMS ...

PROGNOS PROJECT PLAN AND PROPOSAL - … PROJECT PLAN AND PROPOSAL DEVELOP PROGNOS EXCHANGE MODULE INTERFACE P R O G N O S T E A M ... (Air Force) and Land WarNet (Army), need to bridge

PROGNOS PROJECT PLAN AND PROPOSAL - … PROJECT PLAN AND PROPOSAL DEVELOP PROGNOS EXCHANGE MODULE INTERFACE P R O G N O S T E A M ... (Air Force) and Land WarNet (Army), need to bridge

Program Analysis and Verification 0368-4479 maon/teaching/2013-2014/paav/paav1314b.html maon/teaching/2013-2014/paav/paav1314b.html.

Program Analysis and Verification 0368-4479 maon/teaching/2013-2014/paav/paav1314b.html maon/teaching/2013-2014/paav/paav1314b.html.

System Dynamics for Complex Adapve Systems · System Dynamics • Strengths: – Easy model construcon and validaon with available data – Simulaon methods computaonally efficient

System Dynamics for Complex Adapve Systems · System Dynamics • Strengths: – Easy model construcon and validaon with available data – Simulaon methods computaonally efficient

Corporate Basics - The Newberry Group · 2017. 11. 15. · ArmyITSSB% ITservices,InformaonAssurance (IA),IndependentVerificaon& Validaon(IV&V),InternetProtocol Version6(IPv6),EngineeringServices,%

Corporate Basics - The Newberry Group · 2017. 11. 15. · ArmyITSSB% ITservices,InformaonAssurance (IA),IndependentVerificaon& Validaon(IV&V),InternetProtocol Version6(IPv6),EngineeringServices,%

“De- and Remanufacturing” Pilot Network · Validaon - Demonstraon - Ready to market Lack of infrastructures ... , to access the pilot network and to carry on ... Planning Achieved

“De- and Remanufacturing” Pilot Network · Validaon - Demonstraon - Ready to market Lack of infrastructures ... , to access the pilot network and to carry on ... Planning Achieved

PROGNOS PROJECT PLAN AND PROPOSAL - GMU...PROGNOS, a system for Predictive Naval Situation Awareness currently being developed at George Mason University’s C4I Center, is a project

PROGNOS PROJECT PLAN AND PROPOSAL - GMU...PROGNOS, a system for Predictive Naval Situation Awareness currently being developed at George Mason University’s C4I Center, is a project

PROGNOS: APPLYING PROBABILISTIC ONTOLOGIES TO DISTRIBUTED

PROGNOS: APPLYING PROBABILISTIC ONTOLOGIES TO DISTRIBUTED

PROGNOS I ntegration S trategy for PROGNOS K nowledge E xchange M odule I nterface (ISP-KEMI) FINAL PRESENTATION May 7, 2010 Richard Rockweiler (rrockwei@gmu.edu)

PROGNOS I ntegration S trategy for PROGNOS K nowledge E xchange M odule I nterface (ISP-KEMI) FINAL PRESENTATION May 7, 2010 Richard Rockweiler ([email protected])

Clinovo CDISC Soluons - PRWebww1.prweb.com/prfiles/2011/05/09/8400970/CDISC Services... · 2011-05-09 · Generic SDTM validaon tool Define.XML Download the applicaon free of charge

Clinovo CDISC Soluons - PRWebww1.prweb.com/prfiles/2011/05/09/8400970/CDISC Services... · 2011-05-09 · Generic SDTM validaon tool Define.XML Download the applicaon free of charge

Validaon of TES Methane with HIPPO Observaonsacmg.seas.harvard.edu/presentations/2010/wecht_AURA_STmeeting… · 2010-09-29  · Aura Science Team Meeng 29 September 2010 Adjoint

Validaon of TES Methane with HIPPO Observaonsacmg.seas.harvard.edu/presentations/2010/wecht_AURA_STmeeting… · 2010-09-29  · Aura Science Team Meeng 29 September 2010 Adjoint

This ducument familiarizes you with the diversity of the PROGNOS ® -Software. You will find out about everything you need to know about the following PROGNOS.

This ducument familiarizes you with the diversity of the PROGNOS ® -Software. You will find out about everything you need to know about the following PROGNOS.

INTERFACE CONTROL DOCUMENT (ICD) FOR PROGNOS … · PROGNOS, a system for Predictive Naval Situation Awareness currently being developed at George Mason University’s C4I Center,

INTERFACE CONTROL DOCUMENT (ICD) FOR PROGNOS … · PROGNOS, a system for Predictive Naval Situation Awareness currently being developed at George Mason University’s C4I Center,