Spatio-Temporal Shape Modeling and Analysis

Guido Gerig

James Fishbaugh, Marcel Prastawa

Scientific Computing and Imaging Institute, University of Utah

Martin Styner, Stephen Pizer, UNC

Stanley Durrleman, Xavier Pennec, INRIA

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free Modeling

• Driving Applications:

• Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

WM Maturation

Longitudinal/Serial Image Data

Neo 1yr 2yr

Pediatrics: Brain Growth Aging / Neurodegeneration

Trauma: Baseline – Follow-up

Tumor Growth

• Image analysis technology for 4D data is lagging behind acquisition

• Often: individual time-point analysis, ignores causality

Jaw Growth 15-22 yrs

Spatiotemporal Modeling: Natural Task in Clinical Reasoning

Motivation:

Development, degeneration, effects of therapeutic intervention are dynamic processes.

Personalized health care: Individual trajectories compared to expected “norm”.

Clinical terminology: Atypical, Monitoring

Departure from typical development, deviation from healthy

Typical but delayed growth patterns, catch-up, atypical development

Analysis of recovery for each patient

Predict onset of clinical symptoms

Monitor efficacy of treatment

→ Focus on longitudinal design & longitudinal analysis

Shape and Registration

D’Arcy Wentworth Thompson, On Growth and Form (1917, mathematics and biology)

Why Shape?

Shape Variability

link

Shape Similarity in Twins

Upper row: identical twin pairs

Lower row: non-identical twin pairs

Styner/Gerig PNAS 2005

Shape >> Volume

Example Infant Study: Cross-sectional vs. Longitudinal

A: age 2 A: age 4 B: age 2 B: age 4

Cross-sectional: Huge changes between sets of shapes

Longitudinal: Subtle changes of sets of shapes with time

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free modeling

• Driving Applications: • Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

Shape and Registration

Ch. G. Small, The Statistical Theory of Shape

Homology:

Corresponding

(homologous)

features in all

skull images.

Geometric Correspondence: Shapes

PDM Model

(Cootes/Taylor):

• Point to point

correspondence for

shape modeling

• PCA analysis

• Major eigenmodes

of shape variability

Correspondence through spherical

harmonic surface representations

Normalization using first order ellipsoid:

•Rotation of parameter space to align major axis

•Spatial alignment to major axes

Parameters rotated to

first order ellipsoids

Optimizing 3D Correspondences

Rhodri Davies, Carole Twining, Tim Cootes, Danny Allen, Chris Taylor: MDL Model Szekely/Brechbuehler/Styner/Gerig, SPHARM

Optimization of Correspondence in 3D and over time: Particle Systems

2 entropy terms:

• Surface entropy -> distribute the particles “evenly” across surfaces.

• Ensemble entropy -> how are particles similar across a set of surfaces.

Particle system correspondence (Cates, Whitaker et al., Utah, Styner et al., UNC): Minimization of Entropy:

Particle Correspondence Model C

onfigura

tion S

pace

S

hape S

pace

Accurate Representation (in Configuration Space)

vs.

Compact Model (in Shape Space)

Surface Entropy Ensemble Entropy

[Cates et al. IPMI 2007]

Modeling Head Shape Change

Link to Movie

Changes in head size with age

Changes in head shape with age

Datar, Cates, Fletcher, Gouttard, Gerig, Whitaker, Particle-based Shape Regression, MICCAI 2009

Medial Axis / Skeletal Representations: Intrinsic Shape Model

M-rep: Pizer et al.

(discrete)

CM-rep: Yushkevich

(continuous, parametric)

Gorczowski et al., T-PMI 2010,

Stats on deformations vs. thickness

Graph Spectra/Laplacian

Courtesy Hervé Lombaert, Jon Sporring, Kaleem Siddiqi, Mc Gill, IPMI 2013

Graph Spectra/Laplacian

Courtesy Hervé Lombaert, Jon Sporring, Kaleem Siddiqi, Mc Gill, IPMI 2013

Discussion Correspondence

• Automatic optimization of correspondences efficient and robust for simple 3D shapes.

• PDM and Entropy Methods require fixed topology: Time-consuming pre-processing.

• Challenges: – thin, spiky objects

– gaps, missing parts

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free Modeling

• Driving Applications: • Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

Problem: Correspondences between Shapes with Variable Topology

Brain ventricles for infants 6mo to 2yrs DTI Fiber Tracts from two subjects

movie

Durrleman, Pennec, Ayache et al.

“Correspondence-free” Registration: Currents

Topology and shape differences and noise can make point-to-point correspondence hard:

Currents: Objects that integrate vector fields

Shape: Oriented points = Set of normals (tangents)

Distance between curves:

[Glaunes2004] Glaunes, J., Trouve, A., Younes, L. Diffeomorphic matching of distributions: a new approach, … CVPR 2004.

[Durrleman2008] S. Durrleman, X. Pennec, A. Trouvé, P. Thompson, N. Ayache, Inferring Brain Variability from Diffeomorphic

Deformations of Currents: an integrative approach, Medical Image Analysis 2008

Currents integrate vector field:

•W: test space of vector fields (Hilbert space)

•W*: the space of continuous maps W->R

•W* includes smooth curves, polygonal lines, surfaces,

meshes.

[Vaillant and Glaunès IPMI’05, Glaunès PhD’06]

Durrleman, PhD thesis, 2010

Durrleman, PhD thesis, 2010

Durrleman, PhD thesis, 2010

Durrleman, PhD thesis, 2010

Distance between shapes:

• No point correspondence

• No individual line correspondence

• Robust to line interruption

• Need consistent orientation of lines/surfaces

• Is a norm

The space of currents: a vector space:

• Addition = union

• Scaling = weighting different structures

• Sign = orientation

Durrleman, PhD thesis, 2010

Shape Regression w/o Correspondence

• Regression model:

• Variational problem, minimize:

d: modeling shapes as “currents” → distance metric

• → From sampled to continuous shape model:

shape baseline :

ndeformatio varyingtime:

)()(

0

0

M

MtSS

t

tii

Vaillant, M., Glaunes, J.: Surface matching via currents (IPMI 2005). Springer LNCS Vol. 3565) 381–392 Durrleman, S., Pennec, X., Trouve, A., Thompson, P., Ayache, N.: Inferring brain variability from diffeomorphic deformations of currents: an integrative approach. Medical Image Analysis 12/5(12) (2008) 626–637

ii MtS )(

• Concept: Given a set of discrete shapes, interpolate a continuous 4D

growth model via shape regression.

• Assumption: Growth/degeneration of biological tissue is inherently

smooth in space and time & nonlinear, locally varying process.

• Method: Continuous flow of diffeomorphisms via correspondence-free

“currents”. Cost function = Data Matching + Regularity.

Durrleman, Pennec, Ayache, Trouve, Gerig, MICCAI ‘09

Fishbaugh, Durrleman, Gerig, MICCAI ’11, SPIE’12, MICCAI’12, IPMI’13

4D Shape Modeling from Time-Discrete Data

Acceleration Controlled Shape Regression

We define the acceleration field a(x(t)) as a vector field of the form

the shape points carrying a point force vector αi

a Gaussian

kernel with standard deviation λV

Time varying deformation given by:

initial position

initial velocity

Regression Criterion

Acceleration Controlled Shape Regression

Point forces α Acceleration Velocity

Evolution of cerebellum from 6 to 24 months

Piecewise Geodesic vs Acceleration Controlled

Synthetic experiment comparing piecewise geodesic and acceleration controlled shape regression

Piecewise geodesic Acceleration controlled

Interpolation Properties

Summary

Benefits:

● More biologically realistic trajectories ● Nice interpolation properties

Drawbacks:

● Not compact or generative

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free modeling

• Driving Applications:

• Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

Autism: Longitudinal Infant Neuroimaging Study

ACE-IBIS NIH Study: UNC (PI), McGill, Seattle, WU, CHOP, Utah (Image Analysis)

Brain enlargement in autism starts at year 1.

Why? What? Effect?: Longitudinal MRI/DTI study w. >1250 MRI/DTI

Better understanding → Early intervention to improve outcome

30

35

40

45

50

55

0 3 6 9 12 15 18 21 24 27 30 33 36

Age (months)

Hea

d C

irc

um

fere

nc

e

Combined Controls

Autism

N= 113, ADI, ADOS N= 189 community controls

Brain Enlargement on MRI

Longitudinal Head Circumference (Cody

Hazlett et al., ArchGen Psyc Dec. 2005)

Longitudinal Shape Regression

Durrleman, Fishbaugh, Gerig, MICCAI 2011, MICCAI 2012

movie

4D Shape Regression

From discrete 3D shapes (6m,12m,24m) to continuous 4D shape model

Fishbaugh, Durrleman, Gerig, MICCAI 2011, 2012

6 12 24 Time (months)

Individual 4D Growth Profiles

HR+

HR-

LR-

Longitudinal Shape Modeling

Individual 4D model

Work in progress: Stats of 4D growth profiles

Autism Research Collaboration UNC (Piven, Hazlett)

HR+: High risk infant ADOS pos.

HR-: High risk infants ADOS neg.

LR-: Low risk healthy infants

Autism Research Collaboration ACE-IBIS (PI J. Piven, UNC)

Motivation: Longitudinal Imaging in HD Searching for noninvasive biomarker with imaging…

• Symptomatic HD imaging findings

– Atrophied caudate and putamen

– Disproportionate loss of white matter

• Prodromal HD imaging findings

– Striatal atrophy correlates with:

• Neurological impairment

• Poorer performance on cognitive assessments

• Years to motor symptom onset

– Decreased white matter volume

• Seen >15 years before symptom onset

• Morphological study: cognitive deficits had stronger correlation with cerebral white matter atrophy than striatal atrophy Courtesy Jane Paulsen, Hans Johnson, U-Iowa

Motivation: Huntington’s Disease

Courtesy Prof. Hans Johnsen, IOWA

NIH PREDICT Study: Longitudinal imaging study (3-5 scans over 2yr intervals).

Relationship between estimated years to diagnosis of Huntington's disease and

motor exam score and striatal volumes. Distribution of age of onset for individuals

with 36-56 CAG repeats based on the parametric model. Red indicates most likely

time of diagnosis. Blue line is proposed time period when interventional therapies

would have greatest impact.

PREDICT-HD: Longitudinal Imaging Study

58yrs 59yrs 60yrs

Huntington’s Disease: • Neurodegenerative, progressive disease Visual assessment: Subtle changes over time: • Requires well-calibrated image data • Precise, robust segmentations

Huntington’s Disease: Joint analysis of sets of anatomical structures

• Data: Iowa Huntington Disease (HD) study (NAMIC)

• Goal: Prediction of onset of HD from longitudinal preclinical imaging

Clinical Application: Neurodegeneration

in Huntington’s Disease

Continuous individual

subjects’ growth models

Colllaboration Hans Johnson, U-Iowa

Quantitative information

derived from 4D shapes

Muralidharan, Fletcher, Gerig, 2013

Problem for Personalized Profiles: Variability in 3D Segmentation

Volumes from one subject

Volumes from multiple subjects

with varying disease burden

HD: Joint 4D Modeling of subcortical structures

Subject-Specific Shape Modeling

Caudate volume for 32 subjects (3 time pts) extracted after shape regression.

Observed volumes are shown as circles, which highlight the noise in segmentation.

We estimate consistent shape trajectories by considering all shapes simultaneously

which respects the interplay between shape boundaries and locations.

Muralidharan et al, MICCAI 2014

National Alliance for Medical Image Computing

http://na-mic.org

Longitudinal Shape Modeling

Example mandibular surgery case (L. Cevidanes, NA-MIC ancillary

grant).

• V2- 16 years of age before jaw surgery

in the upper jaw maxilla

• V6- 16 years of age 2 months post

surgery

• V10- 18 years of age, 2 years post-

surgery

• V12- 22 years of age, 6 years post-

surgery

National Alliance for Medical Image Computing

http://na-mic.org

Longitudinal Shape Modeling

Example mandibular surgery case (L. Cevidanes, NA-MIC ancillary

grant).

EXOSHAPE ACCEL Tool: Correspondence-free, controlled

acceleration, no tuning, 20’ computing time

Color: speed:

blue = slow and red = fast

Application: Craniosynostosis

4D atlas from 350 full brain shapes (6 – 825 days)

Paniagua, B. et al. 3D of brain shape and volume after cranial vault remodeling surgery for Craniosynostosis correction in infants. SPIE Medical Imaging 2013

velocity

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free modeling

• Driving Applications:

• Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

Shape Analysis via Transformations

D’Arcy Wentworth Thompson, On Growth and Form (1917, mathematics and biology)

GEODESIC REGRESSION OF IMAGE AND SHAPE DATA

Fishbaugh et al, ISBI 2014

GEODESIC REGRESSION OF IMAGE AND SHAPE DATA

Fishbaugh et al, ISBI 2014

Pediatric Brain Development

1) Estimation with images only

2) Estimation jointly but only showing image

3) Estimation jointly and showing both image and white matter

4) Estimation with white matter surfaces only

Pediatric Brain Development

Pediatric Brain Development

Fishbaugh et al, ISBI 2014

Huntington’s Disease: Joint 4D Modeling of Shapes and Images

Subject-specific 4D shape & image regression

Control 2yrs Interval Huntington’s D. 2yrs Interval

Fishbaugh et al., IPMI 2013

Work in Progress: Patient-specific 4D

shape & image regression Control Extrapolated HD Extrapolated

interpolation extrapolation time

Fishbaugh et al., ISBI ‘13, IPMI ‘13

Deformetrics with Sparsity: Tackling fundamental problem of high-dim

features & low-dim sample size (HDLSS)

Image evolution described by considerably fewer parameters, Concentrated in areas undergoing most dynamic changes

Durrleman, 2013 / Fishbaugh IPMI 2013, GSI 2013

Initialization Atlas

Statistics

Common template for

8 Down’s syndrome patients + 8 Ctrls

Most discriminative axis

Classification (leave-2-out) with 105 control points:

specificity sensitivity

Max Likelihood 100% (64/64) 100% (64/64)

LDA 98% (63/64) 100% (64/64)

Importance of optimization in control points positions!

Classification (leave-2-out) with 8 control points:

specificity sensitivity

Max Likelihood 97% (62/64) 100% (64/64)

LDA 94% (60/64) 89% (57/64)

Content

• Motivation Longitudinal Modeling

• Overview Correspondence

• Currents: Correspondence-free modeling

• Driving Applications: • Early Brain Development in Autism

• Huntington’s Disease (HD)

• Mandibular Growth

• Shape Analysis via Grid Deformations

• Growth Trajectories via Time Warp

4D Shape Analysis: Spatiotemporal Registration

Comparing two evolutions: Align shape trajectories:

Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache, MICCAI’10, IJCV 2012

ii MtS )(

ii NtT )(

Formalism captures: • continuous regression: ϕ(t) • morphological change: χ • change of growth speed: ѱ(t)

)()( 0MtS t

function change time:

ndeformatio lgeometrica :

where))),((()(

tStT

4D Shape Analysis: Spatiotemporal Registration

Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache

4D Shape Analysis: Spatiotemporal Registration

Spatiotemporal Variability: • morphological changes • change of growth speed

Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache

(Homo habilis to sapiens vs. homo erectus)

Preliminary Results: Amygdala Growth in Autism

Results: Amygdala Growth in Autism

Conclusions

• Spatio-temporal Image & Shape Analysis: Emerging field with

new fundamental problems (math, stats, imaging, modeling):

– Multidisciplinary by definition.

– Actively developing field driven by new imaging technologies and

novel biomedical driving problems.

– Challenging fundamental, algorithmic and statistical problems.

– Research progress enables new scientific discoveries.

• Main driving motivation: Trajectory of change vs. Cross-

sectional comparisons.

• Clinically highly relevant for quantitative analysis of

longitudinal image data.

Acknowledgements • NIH-NINDS: 1 U01 NS082086-01: 4D Shape Analysis

• NIH-NIBIB: 2U54EB005149-06 , NA-MIC: National Alliance for MIC

• NIH (NICHD) 2 R01 HD055741-06: ACE-IBIS (Autism Center)

• NIH NIBIB 1R01EB014346-01: ITK-SNAP

• NIH NINDS R01 HD067731-01A1: Down’s Syndrome

• NIH P01 DA022446-011: Neurobiological Consequences of Cocaine Use

• USTAR: The Utah Science Technology and Research initiative at the Univ. of Utah

• UofU SCI Institute: Imaging Research Team

• Insight Toolkit ITK

Acknowledgements Methodology Development:

• James Fishbaugh, Utah

• Stanley Durrleman, INRIA Paris

• Xavier Pennec, INRIA Sophia Antipolis

• Ross Whitaker, Utah

• Martin Styner, UNC

Clinical Longitudinal Imaging:

• Joseph Piven, UNC Psychiatry

• Jane Paulsen and Hans Johnson, U of Iowa

• Lucia Cevidanes, UMICH

National Alliance for Medical Image Computing

http://na-mic.org

Freely available Software

ExoshapeAccel: C/C++ NAMIC toolkit SW

for estimating continuous evolution from a

discrete collection of shapes,

James Fishbaugh Public download

Stanley Durrleman

http://www.deformetrica.org/