Simultaneous segmentation and correspondence improvement

using statistical modesAyushi Sinhaa, Austin Reitera, Simon Leonarda, Masaru Ishiib,

Gregory D. Hagera, Russell H. Taylora

aDept. of Computer Science, the Johns Hopkins UniversitybDept. of Otolaryngology-Head and Neck Surgery, Johns Hopkins Medical Institutions

114th February, 2017Orlando, Florida

Functional Endoscopic Sinus Surgery

• What is it?• Minimally invasive procedure

• Chronic sinusitis, nasal polyps, etc.

• 600,000 procedures in the US per year[1]

• 5-7% result in minor complications[2]

• ~1% result in major complications[2]

Image from Toffelcenter.comWHY?

2

[1] Bhattacharyya, N.: Ambulatory sinus and nasal surgery in the United States: Demographics and perioperative outcomes. The Laryngoscope. 120, 635-638 (2010)

[2] Dalziel, Kim; Stein, Ken; Round, Ali; Garside, Ruth; Royle, P.: Endoscopic sinus surgery for the excision of nasal polyps: A systematic review of safety and effectiveness. American Journal of Rhinology. 20(5), 506-519 (2006)

Frontal sinus

Ethmoid sinuses

Maxillary sinuses

Sinuses & Nasal Airway:Complex structures with thin boundaries

Fovea ethmoidalis: separates the ethmoid cells from the anterior

cranial fossaThickness: ~ 0.5 mm[3]

Boundary between the sinuses and the orbit

Thickness: ~ 0.91 mm[4]

3

[3] Kainz, J. and Stammberger, H., “The roof of the anterior ethmoid: A place of least resistance in the skull base,” American Journal of Rhinology 3(4), 191-199 (1989).

[4] Tao, H., Ma, Z., Dai, P., and Jiang, L., “Computer-aided three-dimensional reconstruction and measurementof the optic canal and intracanalicular structures,” The Laryngoscope 109(9), 1499-1502 (1999).

Enhanced Endoscopic Navigation

4

Pre-op CT

Intra-op video

Labeled Template

Deformable Registration

Structure from motion

Registration(ICP[13]/IMLP[14]/IMLOP[15]/

V-IMLOP[16]/etc.)

[13] P.J. Besl, H.D. McKay, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992

[14] Billings SD, Boctor EM, Taylor RH. Iterative Most-Likely Point Registration (IMLP): A Robust Algorithm for Computing Optimal Shape Alignment. PLOS ONE 10(3): e0117688, 2015

[15] Seth D. Billings, RH Taylor. Iterative Most Likely Oriented Point Registration. MICCAI, Boston, Proceedings, Part I. Vol. 8673: pp. 178-185, 2014

[16] Seth D. Billings, A Sinha, A. Reiter, S Leonard, M Ishii, GD Hager, RH Taylor. Anatomically constrained Video-CT registration via the V-IMLOP algorithm. MICCAI, Athens, Proceedings, Part III. Vol. 9902: pp. 133-141, 2016

Segmentation & Statistics

5

Statistics

Set of CTs

Segmentation Statistics

Our paper: Better segmentation & statistics

6

BeforeAfter

Before

After Before

After

Mesh Quality

Statistical Shape Model (SSM)[5]

𝑉1

𝑉2

𝑉3

𝑉𝑛𝑠

𝑉 =1

𝑛𝑠

𝑖=1

𝑛𝑠

𝑉𝑖

Σ =1

𝑛𝑠

𝑖=1

𝑛𝑠

𝑉𝑖 − 𝑉𝑇(𝑉𝑖 − 𝑉)

Σ = [𝑚1 ⋯ 𝑚𝑛𝑠]

𝜆1⋱𝜆𝑛𝑠

𝑚1 ⋯ 𝑚𝑛𝑠 𝑇

7[5] Cootes, T., Taylor, C., Cooper, D., and Graham, J., "Active shape models-their training and application,"

Computer Vision and Image Understanding 61(1), 38-59 (1995).

Correspondence Improvement[8]

8

Project shape onto the modes𝑏𝑖 = 𝑚𝑖

𝑇(𝑉𝑖 − 𝑉)

Compute estimate shape

𝑉 = 𝑉 + 𝑖=1

𝑛𝑠𝑏𝑖𝑚𝑖

Move vertices of original shape along the surface toward the corresponding vertex on estimated shape[8]

[8] Seshamani, S., Chintalapani, G., and Taylor, R., "Iterative refinement of point correspondences for 3D statistical shape models," in Medical Image Computing and Computer-Assisted Intervention, 417-425 (2011).

Assumption

• High accuracy segmentations

• Segmentation improvement

• E.g.: Using gradient vector flow (GVF) snakes[6][7]

• Use gradient in corresponding CT image

• Move mesh vertices toward structure boundaries

• Correspondences between shapes

• Lost during segmentation improvement

[6] Xu, C. and Prince, J. L., "Gradient vector flow: A new external force for snakes," in Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, 66-71 (1997).

[7] Xu, C. and Prince, J., “Snakes, shapes, and gradient vector ow," Image Processing, IEEE Transactions on, 7, 359-369 (Mar 1998).

9

Simultaneous segmentation and correspondence improvement

10

Constrained segmentation improvement

• Using GVF• Move vertices toward large gradients in

image to obtain new surface, 𝜙

• Estimate 𝜙 using pre-existing SSM

• Slide vertices on 𝜙 along the surface toward corresponding vertices on estimated shape

11

Simultaneous segmentation and correspondence improvement

12

𝑃 = 5 𝑄 = 3

5 iterations

ResultsFrom 52 publicly available CTs[9][10][11][12]

13

[9] Bosch, W. R., Straube, W. L., Matthews, J. W., and Purdy, J. A., “Data from head-neck-cetuximab. The cancer imaging archive.," (2015).

[10] Beichel, R. R., Ulrich, E. J., Bauer, C., Wahle, A., Brown, B., Chang, T., Plichta, K. A., Smith, B. J., Sunderland, J. J., Braun, T., Fedorov, A., Clunie, D., Onken, M., Riesmeier, J., Pieper, S., Kikinis, R., Graham, M. M., Casavant, T. L.,

Sonka, M., and Buatti, J. M., “Data from qin-headneck. The cancer imaging archive.," (2015).[11] Fedorov, A., Clunie, D., Ulrich, E., Bauer, C., Wahle, A., Brown, B., Onken, M., Riesmeier, J., Pieper, S., Kikinis,

R., Buatti, J., and Beichel, R. R., “DICOM for quantitative imaging biomarker development: a standards based approach to sharing clinical data and structured pet/ct analysis results in head and neck cancer research," PeerJ

4(e2057) (2016).[12] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Matt, D., Pringle, M.,

Tarbox, L., and Prior, F., \The cancer imaging archive (tcia): Maintaining and operating a public information repository," Journal of Digital Imaging 26(6), 1045{1057 (2013).

Results: Segmentation

14

Red contour: Segmentation

via label transfer using deformable registration

Blue contour:Hand-labeled gold standard

Green contour:Improved

segmentation using our method

Results: Segmentation

Mean Error ± Std. Dev. (mm) Max Error (mm)

Deformable registration 0.3327 ± 0.3147 2.338

GVF 0.1135 ± 0.1316 1.1548

GVF + SSM (our method) 0.0985 ± 0.128 1.0364

15

Truth Deformable Registration GVF GVF+SSM (our method)

Segmentation errors compared against hand-segmented gold standard computed using the Hausdorff distance metric.

0

0.2

0.4

0.6

0.8

1

Results: Correspondence

16

Results: Mesh Quality

17

Segmentation improved using GVF Segmentation improved using our method

Triangle Quality

Conclusion

• Our method improves segmentation while maintaining correspondences

• Demonstrate improved segmentation and correspondence

18

Our shape model contains more accurate information

Our shape model is able to estimate a new shape accurately

Thank you!

FacultyRuss TaylorGreg HagerMasaru IshiiAustin ReiterSimon Leonard

FrameworkRob Gruppcisst Developers

19

FundingNIH R01R01-EB015530: Enhanced Navigation for Endoscopic Sinus Surgery through Video Analysis (PI: Hager)

Johns Hopkins University internal funds

References[1] Bhattacharyya, N.: Ambulatory sinus and nasal surgery in the United States: Demographics and perioperative outcomes. The Laryngoscope. 120, 635-638 (2010)

[2] Dalziel, Kim; Stein, Ken; Round, Ali; Garside, Ruth; Royle, P.: Endoscopic sinus surgery for the excision of nasal polyps: A systematic review of safety and effectiveness. American Journal of Rhinology. 20(5), 506-519 (2006)

[3] Kainz, J. and Stammberger, H., “The roof of the anterior ethmoid: A place of least resistance in the skull base,” American Journal of Rhinology 3(4), 191-199 (1989).

[4] Tao, H., Ma, Z., Dai, P., and Jiang, L., “Computer-aided three-dimensional reconstruction and measurement

of the optic canal and intracanalicular structures,” The Laryngoscope 109(9), 1499-1502 (1999).

[5] Cootes, T., Taylor, C., Cooper, D., and Graham, J., "Active shape models-their training and application," Computer Vision and Image Understanding 61(1), 38-59 (1995).

[6] Xu, C. and Prince, J. L., "Gradient vector flow: A new external force for snakes," in Computer Vision and Pattern Recognition, IEEE Computer Society Conference on, 66-71 (1997).

[7] Xu, C. and Prince, J., “Snakes, shapes, and gradient vector ow," Image Processing, IEEE Transactions on, 7, 359-369 (Mar 1998).

[8] Seshamani, S., Chintalapani, G., and Taylor, R., "Iterative refinement of point correspondences for 3D statistical shape models," in Medical Image Computing and Computer-Assisted Intervention, 417-425 (2011).

[9] Bosch, W. R., Straube, W. L., Matthews, J. W., and Purdy, J. A., “Data from head-neck-cetuximab. The cancer imaging archive.," (2015).

[10] Beichel, R. R., Ulrich, E. J., Bauer, C., Wahle, A., Brown, B., Chang, T., Plichta, K. A., Smith, B. J., Sunderland, J. J., Braun, T., Fedorov, A., Clunie, D., Onken, M., Riesmeier, J., Pieper, S., Kikinis, R., Graham, M. M., Casavant, T. L., Sonka, M., and Buatti, J. M., “Data from qin-headneck. The cancer imaging archive.," (2015).

[11] Fedorov, A., Clunie, D., Ulrich, E., Bauer, C., Wahle, A., Brown, B., Onken, M., Riesmeier, J., Pieper, S., Kikinis, R., Buatti, J., and Beichel, R. R., “DICOM for quantitative imaging biomarker development: a standards based approach to sharing clinical data and structured pet/ct analysis results in head and neck cancer research," PeerJ 4(e2057) (2016).

[12] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Matt, D., Pringle, M., Tarbox, L., and Prior, F., \The cancer imaging archive (tcia): Maintaining and operating a public information repository," Journal of Digital Imaging 26(6), 1045{1057 (2013).

[13] P.J. Besl, H.D. McKay, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992

[14] Billings SD, Boctor EM, Taylor RH. Iterative Most-Likely Point Registration (IMLP): A Robust Algorithm for Computing Optimal Shape Alignment. PLOS ONE 10(3): e0117688, 2015

[15] Seth D. Billings, RH Taylor. Iterative Most Likely Oriented Point Registration. MICCAI, Boston, Proceedings, Part I. Vol. 8673: pp. 178-185, 2014

[16] Seth D. Billings, A Sinha, A. Reiter, S Leonard, M Ishii, GD Hager, RH Taylor. Anatomically constrained Video-CT registration via the V-IMLOP algorithm. MICCAI, Athens, Proceedings, Part III. Vol. 9902: pp. 133-141, 2016

20

21

Questions?

Code will be available on github soon!