Post on 16-Mar-2020
Eyes are the windows of the body: the analysis of corneal and retinal images
Alfredo Ruggeri
BioimLab – DEI, University of Padova, Padova, Italy
ICIAR 201815th International Conference on Image Analysis and Recognition
2
Eyes are window to the body
National Geographic on Youtube: https://youtu.be/BPAbANevTqM
3
Biomedical Image Analysis
• Over the years, our group developed several algorithms for the vascular analysis of retinal images.
• From “tools to perform image analysis” to “tools to provide diagnostic information”
• First (naive) attempt was to develop a complete diagnostic system for diabetic/hyper-tensive retinopathies, in cooperation with NidekTechnologies (Japan).
4
RET-H system
5
Biomedical Image Analysis
• Over the years, our group developed several algorithms for the vascular analysis of retinal images.
• From “tools to perform image analysis” to “tools to provide diagnostic information”
• Measurement of single clinical parameters with web-based tools
Image normalization
7
Inter
Intra
Image normalization
Image normalization
L(x,y)(x,y)IC(x,y)I(x,y) o +×=
),(ˆ),(ˆ),(),(ˆ
yxCyxLyxI
yxIo-
=
Acquired image
Contrast Luminosity
Original image
(Foracchia, Grisan, Ruggeri, Med Image Anal, 2005)
The vessel tracking engine
14
Tracking: pre-processing and seed-finding
• Luminance and contrast drifts removed (Foracchia et al, Medical Image Analysis, 2005)
• Equally-spaced 1-pixel rows and columns of the image are analyzed• From each line, the gray-level profile is
extracted• Using LoG filter at different scales,
patterns corresponding to candidate vessels are searched.
15
Tracking: multi-directional graph search
100 120 140 160 180 200 220
40
50
60
70
80
90
100
110
Seed points are identified and then connected by a multi-directional, lowest-cost graph search approach
16
Tracking: results
[Poletti, Fiorin, Grisan, Ruggeri; Proc. WC2009, Munich.Fiorin, Poletti, Grisan, Ruggeri; Proc. WC2009, Munich]
Optic disc identification
18
Geometrical model
• The course of the main vessels originating from OD can be modeled as two parabolas, having a common origin at OD.
( ){ }2, :x y ay xG = =
• Moving away from OD, vessels inside the parabolas bend towards the center of the image, while those outside bend towards the external edges of the image.
19
Geometrical model
÷÷÷÷
ø
ö
çççç
è
æ
-+-
----
+-
--=
=
)()(
||)sgn()(
||)sgn()sgn(arctan
);,(
21 ODOD
ODODOD
OD
ODOD
mod
xxcxxcaxxyyyy
xxasyyxx
pyxJ
On-parabola contribution Off-parabola contribution
20
[ ]{ }å=
-=N
iiimodiimeasi yxyxwRSS
1
2),(),( JJParameter estimation byminimizing cost function:(Simulated Annealing)
),( iimeas yxJModel of vessel direction: Measured vessel direction:
€
ϑ mod (xi,yi, p) = f (xi,yi;xDO,yDO,β)
Geometrical model
(Foracchia, Grisan, Ruggeri; IEEE TMI, 2004)
21
Objective function for xOD yOD estimation
21(Foracchia, Grisan, Ruggeri; IEEE TMI, 2004)
22
OD identification results
23
OD identification results
23
TorTNETGlobal Vessel Tortuosity Estimation
33
Introduction
In many retinopathies, vessel tortuosity is among the
first alterations appearing in the retinal vessel network
Need for a definition able to express in mathematical terms tortuosity
as perceived by retina experts
35
Tortuosity estimation
Proposed tortuosity index t :
When evaluating tortuosity of a line, human experts integrate information on:1. how many times a line changes curvature sign2. how large is the amplitude of each turn curve (the curve segment between
two changes in curvature sign)
(Grisan, Foracchia, Ruggeri; IEEE TMI, 2008)
38
1.05
3.87
6.86
9.15
12.78
10.13
Vessel tortuosity estimation
40
• Ground-truth derived as average on the 3 graders’ orderings.
New tortuosity estimation
Twist-based tortuosity:
Angle-based tortuosity:Based on difference between direction angles of adjacent samples
Based on arc-to-chord length ratio, for every twist(Grisan et al., IEEE TMI, 2008)
• 20 images (10 normal, 6 pre plus, 4 plus), 640x480 pxls 130° FOV, acquired with RetCam (Clarity Medical Systems, USA).
• Sorted by increasing tortuosity by 3 clinical graders and 3 ROP experts using TorTsorT(http://bioimlab.dei.unipd.it/TorTsorT.html)
41
IMAGE-LEVEL tortuosity 1
FinalIMAGE-
TORTUOSITY
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation ‶1
‶2
‶8
∑
IMAGE-LEVEL tortuosity 1
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation
VESSEL -LEVEL tortuosity 2
Image-level tortuosity
42å= vesselall nn
tortuosityvesseltortuosityimage 1=n
VESSEL -LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 1
FinalIMAGE-
TORTUOSITY
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation ‶1
‶2
‶8
∑
IMAGE-LEVEL tortuosity 1
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation
VESSEL -LEVEL tortuosity 2
5=n
Image-level tortuosity
43
VESSEL -LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 1
FinalIMAGE-
TORTUOSITY
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation ‶1
‶2
‶8
∑
IMAGE-LEVEL tortuosity 1
VESSEL - LEVEL tortuosity 1
VESSEL-LEVEL tortuosity 8
IMAGE-LEVEL tortuosity 2
IMAGE-LEVEL tortuosity 8
aggregation
VESSEL -LEVEL tortuosity 2
FinalIMAGE-
TORTUOSITY
‶1
‶2
‶8
∑
Combination coefficients determined by REGRESSION on ground-truth ordering
Image-level tortuosity
44
Image Tortuosity = 3.5
Image Tortuosity = 12.0
Image Tortuosity = 21.5
Results: examples
45
CORNEAAnterior Chamber
Corneal image analysis
Anatomy of the cornea
46
Microscopic anatomy of cornea
OutsideInside
EPITHELIUMSTROMAENDOTHELIUM
Descemet’smembrane
Bowman’s membrane
47
• Corneal confocal microscopy allows the rapid and non-invasive acquisition of images of all corneal layers.
• Images are acquired with ConfoScan4 (Nidek Technologies, Italy) or HRTII-RCM (Heidelberg Engineering, Germany).
Confocal microscopy
Endothelial cells
Stromal keratocytes
Sub-basal nerve fibers
Epithelial cells
+
48
• Corneal specular microscopy allows acquisition of images only of the most reflective corneal layer, the endothelium.
• Konan, Tomey, Topcon are among the most common microscopes.
Specular microscopy
Endothelial cells
49
Automatic recognition and features measurement of
sub-basal nerve fibers
50
Analysis of subbasal layer images
OutsideInside
EpitheliumStromaEndothelium
Endothelial cells
Stromal keratocytes
Sub-basal nerve fibers
Epithelial cells
51
Motivation
• Confocal microscopy allows the rapid and non-invasive acquisition of images of sub-basal nerve plexus (SNP) fibers.
• Their analysis appears to be an important clinical issue:• Malik Diabetologia 2003 (diabetes)• Midena J Refract Surg. 2006 (diabetes)• Tavakoli J Diabetes Sci Technol 2013 (diabetes)
• Calvillo, IOVS 2004 (LASIK)• Moilanen, IOVS 2003 (PRK)• Patel S., IOVS 2002 (contact lens)
• Patel D., IOVS 2005 (keratoconus)• Benitez del Castillo, IOVS 2004 (dry eyes)
• Rosenberg, Cornea 2002 (herpes keratitis)
• To provide useful information, quantitative morphometry should be available based on nerve tracing.
• To avoid long manual analysis, automated nerve tracing and quantification is required.
52
A first technique for corneal nerves tracing
• Corneal basal layer images are acquired and their luminosity and contrast are normalized.
• Seed points are detected and used as starting points for the nerve tracing procedure(based on our work on retinal vessel tracing).
• Nerve segments are recognized.
• Post-processing to:- remove false positive recognitions- increase sensitivity.
59
Some results
(Scarpa et al, Invest Ophthalmol Vis Sci, 2008)
60
N=90 images, with Nidek Technologies ConfoScan-4
Results
(Scarpa et al, Invest Ophthalmol Vis Sci, 2008)
61
Nerve tortuosity
N=30 images, Heidelberg Engineering HRTII-RCM
(Scarpa et al, Invest Ophthalmol Vis Sci, 2008)
r=0.783
62
New nerve tracing
• Images are first enhanced for uneven illumination and contrastby top-hat filtering.
• To enhance the corneal nerves, the corrected images are filtered with a bank of log-Gabor filters.
• Thresholding is applied to recognize all “elongated white structures”, which include corneal nerves.
• Recognition of true nerves is carried out with a Support Vector Machine classifier.
63
New nerve tracing
SENSITIVITY FALSE DETECTION RATEmean sd mean sd
Automatic Tracing 0.86 0.07 0.08 0.07Second Observer 0.92 0.05 0.08 0.05
• Nerve tracings were compared with ground-truth (first observer).
• 246 images from healthy subjects acquired with the Heidelberg HRTII-RCM. Nerve traced by two expert observers.
• 50 images used for training and optimization, 196 images for validation.
(Guimarães et al, Trans Vis Sci Tech, 2016)
64
New nerve tracing
• From several minutes to less than one second per image.
Correlation: 0.93
Density
(Guimarães et al, Trans Vis Sci Tech, 2016)
65
New nerve tracing tortuosity
Tortuosity = 1.7 Tortuosity = 3.9 Tortuosity = 7.5
Tortuosity = 12.9 Tortuosity = 17.2
66
New nerve tracing
• From several minutes to less than one second per image.
Accuracy: 28/30Correlation: 0.93
Density Tortuosity
(Guimarães et al, Trans Vis Sci Tech, 2016)
68
Automatic mosaicking of sub-basal nerve fibers images
69
Image mosaic
Architecture of the SNP on healthy[1] and keratoconus[2] subjects.
Mosaic manually built from ~500 images.Hugely time-consuming process (from 10–20 hours)
Patel DV, McGhee CNJ. Mapping of the normal human corneal sub-basal nerve plexus by in vivo laser scanning confocal microscopy. IOVS, 2005.Patel DV, McGhee CNJ. Mapping the corneal sub-basal nerve plexus in keratoconus by in vivo laser scanning confocal microscopy. IOVS, 2005.
72
Image mosaic
An algorithm for image mosaic / registration.
Problem: 100+ images in random order; how can I find the adjacent ones to register?
First step:Find the registration parameters between each pair of images
73
Mosaicking process
(xt, yt, r, score)
For each image find the closest other image using a “similarity score”
74
Mosaicking process
Build the best ordered sequence of pairs of images to be merged (MinimumSpanning Tree)
(xt, yt, r, score)
75
Mosaicking process
Move along the sequence merging the pairs of images (with rotation, translation and affine transform).
81
Image blending
82
Image blending
85Run-time 46 seconds.
…. back to nerve tracing
86
…. back to nerve tracing
Run-time 32 seconds.
87
Automatic analysis of endothelial cells
88
Analysis of endothelium images
OutsideInside
EpitheliumStromaEndothelium
Endothelial cells
Stromal keratocytes
Sub-basal nerve fibers
Epithelial cells
89
• Normal corneal endothelium is a single layer of uniformly sized cells with a predominant hexagonal shape. This regular tessellation is affected by age and pathologies.
• Segmentation of a large number of endothelial cells required for a reliable estimation of clinical morphometric parameters:
endothelial cell density (ECD) = total cell area / nr of cells
pleomorphism = % of hexagonal cells polymegethism = fractional SD of cell areas
(last one requires contour detection)
• To avoid long manual analysis, automated cells contour segmentation is required.
Motivation
90
Cell contour recognition
91
Cell contour recognition
Human visual processing is very powerful and complex …
Kanisza triangles Kanisza square
(Gaetano Kanisza, 1913-1993, psychologist and artist, University of Trieste)
92
Cell contours appear nice and clearon a broad view….
Human visual processing is very powerful and complex …
… but local gray-scale values do not give all the information necessary to identify all cell contours.
false contoursmissed contours
Cell contour recognition
93
1. Gray-scale information
Þ contour extraction
2. Shape information
Þ contour completion and fixing
Two levels of information
94
Contour extraction: Artificial Neural Network with weight-filters arrays
Good as a �proof of concept�, but not usable in the clinicalroutine yet.
Cell contour recognition
95
1. Gray-scale information
Þ contour extraction
2. Shape information
Þ contour completion and correction
Two levels of information
96
Connected boundariescorrect
not correct
Boundaries determined by the ANN based only on gray-scale values
Contour completion: connection of floating facing boundaries
Cell contour recognition
97
False ContoursDetected by the excessively small sizeof the cell bodies
Missed ContourDetected by the excessively large sizeexcessively large aspect ratioof the cell body
Contour correction (from shape information)
Cell contour recognition
98
missed contour
false contours
Back to floating boundaries connection
Contour correction (from shape information)
Cell contour recognition
108
Nidek Technologies NAVIS-ENDO system
• The ENDO software is a module of the system for ophthalmology.
109
A little toy …
(Foracchia Ruggeri, EMBC, 2003)
110
A bigger toy …
(Foracchia Ruggeri, EMBC, 2007)
112
A new project for segmentation of cells contour• The contours of cells detected by a genetic algorithm.• A small set of vertices (individuals) forming regular hexagons is the
starting population. • At each step, the location of each vertex is randomly modified,
evolving into polygons with possibly different number and positions of vertexes.
• Each vertex is evaluated by considering both its correspondence with the actual image (pixels intensity) and the regularity of the resulting polygons.
115
Some results
normal subject subject with high polymegethism
subject with low ECD
116
Some results
normal subject subject with high polymegethism
subject with low ECD
ECD (cells/mm2) 2999
Pleomorphism (%) 83,3
Polymegethism (%) 24,4
ECD (cells/mm2) 2151
Pleomorphism (%) 57,9
Polymegethism (%) 31,6
ECD (cells/mm2) 970
Pleomorphism (%) 55,6
Polymegethism (%) 32,2
117
Results
Automated Manual 1 Manual 2 abs diff % abs diff %ECD cells/mm2 cells/mm2 cells/mm2 Auto vs M1 M1 vs M2
Mean 2562 2572 2575 0.60 % 0.46 %Sd 827 828 833 0.51 % 0.60 %
Min 457 468 465 0.09 % 0.00 %Max 3627 3653 3659 2.35 % 3.41 %
Pleomorphism % % %Mean 58.51 58.16 58.15 3.11 % 2.60 %
Sd 10.06 10.02 10.45 3.33 % 3.62 %Min 42.30 41.20 38.20 0.00 % 0.00 %Max 83.30 83.30 83.30 12.23 % 13.05 %
Polymegethism % % %Mean 34.85 36.68 37.35 5.33 % 2.89 %
Sd 5.92 6.54 6.65 2.94 % 1.67 %min 20.80 21.70 22.30 0.27 % 0.30 %max 46.30 49.00 50.40 11.82 % 6.67 %
• 30 images acquired with a Topcon SP3000 specular microscope.• Differences between automated and two manual assessments.
(Scarpa Ruggeri, Cornea, 2016)
118
The current team at BioImLab UniPD
http://bioimlab.dei.unipd.it
Marco ForacchiaEnrico GrisanMassimo De LucaLara TramontanDiego Fiorin
Alumni: Pedro GuimāraesJeff WigdahlEnea Poletti
119
A warning from the past …
“Things which we see are not by themselveswhat we see … It remains completely
unknown to us what the objects may be by themselves ... We know nothing but our manner of
perceiving them ...”
Immanuel Kant