Cell Tracking via Proposal Generation & Selection tracking.pdfCell Tracking via Proposal Generation...
25
Cell Tracking via Proposal Generation & Selection Saad Ullah Akram, Juho Kannala, Lauri Eklund and Janne Heikkilä [email protected] 2
Transcript of Cell Tracking via Proposal Generation & Selection tracking.pdfCell Tracking via Proposal Generation...
Cell Tracking via Proposal Generation & Selection
Saad Ullah Akram,Juho Kannala, Lauri Eklund and Janne Heikkilä
Overview
• Introduction• Importance
• Challenges: detection & tracking
• Current state of cell tracking field
• Our method• Cell proposal generation (CPN)
• Proposal graph
• Cell tracking via proposal selection
• Summary
Overview
• Introduction• Importance
• Challenges: detection & tracking
• Current state of cell tracking field
Time
Cell Tracking
• Process of:• Locating a moving cell over time &
• Detecting cellular events (division, death, entry, exit)
Lineage of a cell
: highlights links between parent & daughter cells
Applications
• Understanding dynamic cellular behavior• Spread of diseases (e.g. cancer)
• Effectiveness/safety of a drug
• Organ/embryo development
• Gene profiling
Automation
• Huge datasets• 1,000s of 3D images/day
• 10,000s cells in each image
• > 10 TBs of image data/day
• Subtle patterns
• Accurate/objective/repeatable
Video from: F. Amat et al., “Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data.,” Nat. Methods, 2014.
Challenges: Tracking
• Difficult to model motion
• Similar cell appearances and shapes
Similar shapes & appearances
Challenges: Tracking
• Low frame rate• Large movements
• Large shape and appearance changes
• Difficult to detect cell division
[email protected] 9Parent cells in last frame before division
Large movement
Challenges: Detection
• Detection failures account for most tracking errors
• Large variation in cell shapes & appearances
• Microscopy modality
• Labeled structure
• Resolution
• Cell density
Bri
ght
Fie
ldP
has
e
Co
ntr
ast
DIC
M
icro
sco
py
Fluo
resce
nce
Dye
s & P
rote
ins
Image from: E. Meijering, “Cell Segmentation: 50 Years Down the Road,” IEEE Signal Process. Mag., 2012.
Current cell tracking methods
• Tracking by detection• Detect cells in each frame
• Associate detections in nearby frames
• Pros₊ Low frame rate/resolution₊ Simple and modular
• ImageJ (TrackMate, MTrack2), CellProfiler (TrackObjects), etc
• Tracking by model evolution• Detect cells in 1st frame and
represent it using some model
• Segment current frame using the model from last frame
• Pros₊ Can handle noisy images
Limitations of current cell tracking methods
• Cell detection methods assume• Cell appearances
• High intensity near the center
• Characteristic intensity profile at cell boundaries
• Cell shapes• Round or elliptical shapes
• Cell size within some narrow range
• Assumptions are often not satisfied for new sequences
• Lack robustness• Require tuning multiple parameters
Overview
• Our method• Cell proposal generation (CPN)
• Proposal graph
• Cell tracking via proposal selection
Tracking by proposal selection
1. Generate cell proposals• Use CNNs to learn the cell shapes and appearances
2. Link cell proposals using edges (represent cellular events)
3. Select subset of cell proposals & their associations (provides cell tracks)
Cell proposals/candidates
• Cell detection can benefit from• Temporal information
• Machine learning to decide what a cell looks like
• A cell proposal is a candidate segmentation of a cell• Enables representation of multiple segmentation hypothesis in ambiguous regions
Image Cell Detection Cell Proposals: 2 Cell Proposals: 3 Cell Proposals: 4
Cell proposal network (CPN)
3
32
3
64
3
32
3
128
3
64
3
256
3
128
RO
IPo
olin
g 12
8
Can
did
ate
Bo
un
din
g B
oxe
s
Nx5
7
32
3
32
5
64
3
64
3
128
3
512
1
6k
3
256
4k
bb
ox
2k
sco
re
NMS NMS
NMS
1
16
33
64
3
32128Proposals
Proposal Masks
25
.25
.2
+
+
+
Fig: CPN: Convolutional, max-pooling, fully connected, and deconvolutional layers are shown. Proposed bounding boxes and segmentation
masks after non-maxima suppression (NMS) are shown for a selected area from Fluo-N2DL-HeLa dataset.
Bounding box network
Segmentation network
S
T
Tracking graph (g): Ground truth
D1
D2 D3
D5D4
b)
• Cell tracks can be represented using a graph
• Nodes represent cells
• Special nodes (S & T): Initiate/terminatecell lineage trees
• Edges represent cell events• Cell entering field of view
• Cell exiting field of view
• Cell division (mitosis)
• Cell moving
a)
Tim
e
Graphical model (G)
• A graphical model (G) is used to reason about which cell tracks should be selected
• Nodes represent cell proposal probabilities
• Black edges represent proposal constraints
• Special nodes (S & T): Initiate/terminate cell lineage tress
• Only few edges of each type are shown to avoid clutter
• Edge weights represent cellular event probabilities• Cell entering field of view
• Cell exiting field of view
• Mitosis• Cell moving
0.94 0.94
0.95 0.99
0.88 0.81
0.98
0.62
b)
0.42 0.860.11
a)
Tim
e
S
T
0.010.01
0.070.03
0.86
Proposal selection
c)a)Ti
me
• Each sub-graph of our model (G) represents a tracking hypothesis
• Optimal tracks can be obtained by finding the sub-graph with the highest probability
0.94 0.94
0.95 0.99
0.88 0.81
0.98
0.62
b)
0.42 0.860.11
S
T
0.010.01
0.070.03
0.86
0.94 0.94
0.95 0.99
0.88 0.81
0.98
b)
0.86
S
T
0.86
Results
• Same model structure is used for all datasets
• 4 datasets from ISBI Cell Tracking Challenge [1]
• TRA:• Tracking score (0-1 (perfect))• penalizes errors in the tracks
• SEG:• Segmentation score• Instance level intersection over union
overlap.
Video [1] Method TRA SEG
Fluo-N2DL-HeLa-02
CPN 0.983 0.845
EPFL [3] 0.97
KTH [4] 0.975 0.837
Fluo-N2DH-GOWT1-02
CPN 0.972 0.873
HEID [2] 0.95
EPFL [3] 0.95
PhC-C2DH-U373-02CPN 0.935 0.738
U-Net [5] 0.955 0.830
PhC-C2DL-PSC-01 CPN 0.943 0.661
[1] M. Maška et al., “A Benchmark for Comparison of Cell Tracking Algorithms,” Bioinformatics, 2014.
[2] M. Schiegg, et al., “Graphical Model for Joint Segmentation and Tracking of Multiple Dividing Cells,” Bioinformatics, 2015.
[3] E. Turetken, et al., “Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences,” in T-MI, 2016.
[4] K. E. G. Magnusson, et al., “A Batch Algorithm using Iterative Application of the Viterbi Algorithm to Track Cells and Construct Cell Lineages,” in ISBI, 2012.
[5] O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in MICCAI, 2015.
Summary
• Our tracking method₊ (Potentially) can be applied to other sequences₊ Has better performance than existing methods₋ Requires annotated data
• More Information:• Paper: https://arxiv.org/abs/1705.03386
• Code: https://github.com/SaadUllahAkram/CellTracker
