A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR
description
Transcript of A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR
![Page 1: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/1.jpg)
Combining the strengths of UMIST andThe Victoria University of Manchester
A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR
Roy Schestowitz, Carole Twining, Tim Cootes, Vlad Petrovic, Chris Taylor and Bill Crum
![Page 2: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/2.jpg)
Overview• Motivation
• Assessment methods– overlap-based
– model-based
• Experiments– validation
– comparison of methods
– practical application
• Conclusions
![Page 3: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/3.jpg)
Motivation• Competing approaches to NRR
– representation of warp (including regularisation)
– similarity measure
– optimisation
– pair-wise vs group-wise
• Different results for same images
• Need for objective method of comparison
• QA in real applications (how well has it worked?)
![Page 4: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/4.jpg)
Existing Methods of Assessment• Artificial warps
– recovering known warps
– may not be representative
– algorithm testing but not QA
• Overlap measures– ground truth tissue labels
– overlap after registration
– subjective
– too expensive for routine QA
• Need for new approach
A
B
Warp NRR
![Page 5: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/5.jpg)
Combining the strengths of UMIST andThe Victoria University of Manchester
Model-Based Assessment
![Page 6: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/6.jpg)
Model-based Framework• Registered image set statistical appearance model
• Good registration good model
– generalises well to new examples
– specific to class of images
• Registration quality Model quality
– problem transformed to defining model quality
– ground-truth-free assessment of NRR
![Page 7: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/7.jpg)
Building an Appearance Model
…
Imag
es
…
…
Sha
pe
NRR
Text
ure
Model
![Page 8: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/8.jpg)
Training and Synthetic Images
Image Space
Training Model Synthetic
![Page 9: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/9.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 10: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/10.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 11: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/11.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 12: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/12.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 13: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/13.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 14: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/14.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 15: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/15.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
NRR
![Page 16: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/16.jpg)
Training and Synthetic Images
Image Space
Training
Image Space
Training Model Synthetic
![Page 17: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/17.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 18: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/18.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 19: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/19.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 20: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/20.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 21: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/21.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 22: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/22.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 23: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/23.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
Generate
![Page 24: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/24.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
![Page 25: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/25.jpg)
Training and Synthetic Images
Image Space
TrainingSynthetic
Image Space
Training Model Synthetic
![Page 26: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/26.jpg)
Model Quality
TrainingSynthetic
Given measure d of image distance
STjd
• Euclidean or shuffle distance d between images
• Better models have smaller distances, d
• Plot [-Specificity], which decreases as model degrades
![Page 27: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/27.jpg)
Combining the strengths of UMIST andThe Victoria University of Manchester
Validation Experiments
![Page 28: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/28.jpg)
Experimental Design• MGH dataset (37 brains)
• Selected 2D slice
• Initial ‘correct’ NRR
• Progressive perturbation of registration
– 10 random instantiations for each perturbation magnitude
• Comparison of the two different measures
– overlap
– model-based
![Page 29: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/29.jpg)
Brain Data• Eight labels per image
– L/R white/grey matter– L/R lateral ventricle– L/R caudate nucleus
Image LH Labels RH Labels
lv
gm cn
wmwm lv
gm cn
![Page 30: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/30.jpg)
CPS with 1 knot point Example warp
Examples of Perturbed Images
Increasing mean pixel displacement
![Page 31: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/31.jpg)
Results – Generalised Overlap • Overlap decreases monotonically with misregistration
![Page 32: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/32.jpg)
Results – Model-Based• [-Specificity] decreases monotonically with misregistration
![Page 33: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/33.jpg)
Results – Sensitivities• Sensitivity
– ability to detect small changes in registration– high sensitivity good
• Specificity more sensitive than overlap
![Page 34: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/34.jpg)
Combining the strengths of UMIST andThe Victoria University of Manchester
Practical Application
![Page 35: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/35.jpg)
Practical Application• 3 registration algorithms compared
– Pair-wise registration– Group-wise registration– Congealing
• 2 brain datasets used– MGH dataset– Dementia dataset
• 2 assessment methods– Model-based (Specificity)– Overlap-based
![Page 36: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/36.jpg)
Practical Application - Results• Results are consistent
• Group-wise > pair-wise > congealing
MGH Data MGH Data Dementia Data
![Page 37: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/37.jpg)
Extension to 3-D• 3-D experiments
• Work in progress
– validation experiments laborious to replicate
– comparison of 4-5 NRR algorithms
• Fully-annotated IBIM data
• Results can be validated by measuring label overlap
![Page 38: A Generic Method for Evaluating Appearance Models and Assessing the Accuracy of NRR](https://reader033.fdocuments.in/reader033/viewer/2022052914/56814b74550346895db861d6/html5/thumbnails/38.jpg)
Conclusions• Overlap and model-based approaches ‘equivalent’
• Overlap provides ‘gold standard’
• Specificity is a good surrogate
– monotonically related
– robust to noise
– no need for ground truth
– only applies to groups (but any NRR method)