Image Analysis: Basics and Practice
Dr. Xianke ShiZEISS Research Microscopy Solutions
2Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Microscopy Images: A Picture is worth a thousand words!
How Many Cells are DAB positive?
A: 1-10%B: 10-20%C: 20-30%D: 30-40%
3Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Microscopy Images: A Picture is worth a thousand words!
How Many Cells are DAB positive?
A: 1-10%B: 10-20%C: 20-30%D: 30-40%
đˇđˇđˇđˇđˇđˇ =234
234 + 418% = 35.9%
4Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Visual perception is not quantitative, and sometimes not reliable
What is the grey level difference between black and white surface (8 bit image)?
A: 0B: 10C: 20D: 30
Use your finger to block the center region (red arrow indicated). There is no grey level difference!
Imagine you are judging protein expression level by visually compare neighboring tissue brightness!
5Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Visual perception is not quantitative, and sometimes not reliable
Internet sensation from 2015
#blackandblueVS#whiteandgold
6Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Visual perception is not quantitative, and sometimes not reliable
Humans recognize, machines measure⢠Human vision is qualitative and comparative⢠It is very powerful in recognizing patterns⢠The performance for intensity, distance and
size measurements is poor
Machine based image analysis improves:⢠Objectivity⢠Reproducebility⢠Speed
7Dr. Xianke Shi, Zeiss Research Microscopy Solutions
The earliest surviving Analog Fotograph of the World
First Photograph
⢠In the year 1826⢠Bitumen plate⢠Exposure time: 8 h ⢠Photographer:
Joseph NicÊphore Nièpce
8Dr. Xianke Shi, Zeiss Research Microscopy Solutions
The first digital photograph
First Digital Photograph
⢠In the year 1975⢠20 year before the first digital
camera by Kodak⢠Resolution: 176x176 pixels⢠Photographer: Russel Kirsch⢠His 3 months old son⢠Digitally scanned from analog
photograph
9Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Digital Imaging: the foundation of Image Analysis
A digital image is a two dimensional array of number matrix. Each number refers to as a pixel (2D) or voxel (3D).The word pixel is an abbreviation for picture element.Higher dimensions is common for microscopy:
Z-stack (z)Time (t)Spectral Channels (Îť)Tiles and Position (p)
10Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Digital Imaging: the foundation of Image Analysis
For microscopy imaging, additional Metadata is usually built into the raw data:⢠Objective information (model, magnification, numerical aperture, refractive index)⢠Dimensional information (pixel size, frame size X/Y/Z)⢠Coordinate information (absolute stage position X/Y/Z, well number, position number)⢠Acquisition parameters (camera exposure, binning, zoom, gain, offset, pixel dwell time)⢠Environment information (Temperature, humidity, CO2 concentration)⢠Image information (Acquisition date, time, duration, name)⢠Many moreâŚ
https://docs.openmicroscopy.org/bio-formats/5.9.2/metadata/ZeissCZIReader.htmlThe Zeiss .czi file format has 476 metadata entries:
Those metadata is useful for image analysis, additional image processing, and programming based microscopy automation!
11Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understand Digital ImageBit depth and grey levels
⢠With 8-bit images 0 to 255 (âgrey valuesâ).⢠With 14-bit 0 to 16383 grey values⢠With 16-bit 0 to 65535) grey values
More Bits:⢠More accurate representation of the
intensity values⢠More time it takes to transfer data, slower
image acquisition
12Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understand Digital ImageColor image
A standard RGB color image is a combination of 3 layers of Red, Green, and Blue channels.It can be also defined by HLS color space for: Hue, Lightness, and Saturation.
Each color channel has its own grey value
13Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understand Digital ImageRGB image vs Multichannel images
x
y
x
y
Multichannel imageMultiple channels, each with 1 Histogram
RGB imageOne channel with 3 Histograms
DAPIeGFP
âŚ
14Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understand Digital ImageHistogram
A histogram is a graphical representation of the pixel grey value distributions of an image.
Normalized number of pixels
Pixel grey values
Black pixels (low grey level)More such pixels
Brightest pixels (highest grey level)Less such pixels
Logarithmic scale
Linear scale
15Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understanding Digital ImagesHistogram: RGB image vs Multichannel images
A histogram is a graphical representation of the pixel grey value distribution of an image.
Fluorescence Multichannel image
RGB Image
⢠Multiple histograms⢠Each histogram has its own display setting⢠Default gamma 1.0
⢠RGB histograms⢠One display setting⢠Default gamma 0.45
16Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Understand Digital ImageHistogram and adjustment of display curves for visual enhancement
Display curves of the histogram can be adjusted to better visualize the imageSuch adjustment wonât change the grey value of the image
17Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Image Analysis and its Workflow
Pre-Processing Segmentation Feature
DetectionData
Presentation
⢠Smoothing⢠Sharpening⢠Noise reduction⢠Shading correction⢠Histogram Stretching
⢠Intensity⢠Size⢠Shape⢠Coordinates
⢠Thresholding⢠Variance based⢠Machine learning
⢠Line plot⢠Scatter plot⢠Heatmap
âImage analysis is the extraction of meaningful information from images; mainly from digital images by means of digital image processing techniques.â --- Wikipedia
For Microscopy, it is also a essential part of microscopy automation.
18Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Pre-Processingoverview
⢠Reduction of acquisition artefacts which could influence the results, e.g. Shading and noise ⢠Improving image quality for better target identification, e.g. edge enhancement⢠Can be applied for segmentation purpose only, then subsequent parameter measurement still performed on raw data
Typical pre-processing steps:
⢠Shading Correction ⢠Backgroud subtraction⢠Smoothing⢠Noise reduction⢠Spectral unmixing⢠Deconvolution⢠Histogram stretching⢠âŚ
shading correctionoriginal
Corrected image after thresholdingoriginal after thresholding
Pre-Processing Segmentation Feature
DetectionData
Presentation
19Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Pre-ProcessingPixel based Image Processing (aka filtering)
input image output image
⢠One pixel in the input image is processed and is assigned to the same pixel in the output image
⢠The new greyvalue can also be calculated from all pixels in a small neighborhood around the input pixel
⢠The neighborhood size is usually called a kernel
3x3 kernel
⢠This procedure is repeated for all pixels of the input image
⢠The kernel operator can be linear (e.g. Gaussian and mean filter) and non-linear (e.g. median)
20Dr. Xianke Shi, Zeiss Research Microscopy Solutions
88 81 152 227 255 214 211 145
219 185 255 193 193 142 162 140
255 242 240 216 194 109 125 118
119 66 142 93 96 125 204 173
125 71 43 64 102 75 94 163
166 94 100 45 89
147 108 100 75 71
67 83 87 104 54
â đđđđđđ
1 2 1
2 4 2
1 2 1
=
163 147 171 193 200 191 178 160
169 171 182 186 183 171 161 151
175 171 171 167 159 150
3x3 Gauss
Pre-ProcessingExample of 3x3 kernel Gaussian filter
21Dr. Xianke Shi, Zeiss Research Microscopy Solutions
SegmentationThe most critical step
⢠Segmentation: partitioning a digital image into multiple segments, i.e. find groups of pixels that belong together
⢠Binary process: the object of interest is 1, the rest is 0. The result is a âmaskâ of the segmented objects.
⢠The âmaskâ is usually one additional layer on the original image. The mask can be further refined by operations on the binary mask (e.g. dilation, separation,âŚ)
⢠The segmentation can be based on different image information (e.g. intensity, color, pattern, pixel position,âŚ)
Pre-Processing Segmentation Feature
DetectionData
Presentation
22Dr. Xianke Shi, Zeiss Research Microscopy Solutions
SegmentationThe most critical step
Threshold (dividing the histogram basedon intensity)
- Fluorescence images
Variance(local change in intensity, usually linked to edges)
- transmission images
Intellesis (machine learning)
- Everything (computationally heavy)- Nothing else works- Ease of use
23Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation by global thresholdingThe most common method
# pixels
Gray valueIntensity based separationThreshold between two populations
Foreground
Background
Original After Threshold
24Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation by global thresholdingManually
Manually define upper/lower threshold
3 sets of threshold for RGB image
Useful software tools for direct mouse-click-on-object and thresholding
25Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation by global thresholdingAutomatically
There are 5 different automatic thresholding methods in ZEN:⢠Otsu
(objects and background separation with minimum sum of variance for each group)⢠Max Peak
(separation at histogram maximum value)⢠Iso Data
(separation at the middle of two histogram maximum values)⢠Triangle
(separation at the sum of the average histogram value)⢠Three Sigma
(separation at the sum of the average histogram value with 3 sigma)
The success rate of auto thresholding depends very much on image SNR.Improve image SNR during acquisition is vital to the final result!As a simple guide, draw a profile along the object, and its SNR should be >5 for auto thresholding to work efficiently.
26Dr. Xianke Shi, Zeiss Research Microscopy Solutions
There is no absolute good or bad automatic thresholding method. Each of them is applicable to different type of images.It is usually good to start with Otsu.
Original Otsu Max Peak Iso data Triangle Three sigma
2202-6866 324-6866 2202-6866 813-6866 4851-6866
Segmentation by global thresholdingAutomatically
27Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation by Variance-based thresholdingsegmentation based on local variance
Transmission based images, like BF, DIC, Ph, PGC, has little grey value difference between object and background. Histogram thresholding based segmentation usually wonât work.
Profile tool
PGC
One population Two populations
VS
Higher localvariance
28Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Measure stdev
Set threshold 100-1000
Necessary binary process
Segmentation by Variance-based thresholdingExample in ZEN
29Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Classify Segmented Image
Input Data
Feature ExtractorsLabel
TrainedModel
Retrain
Segmentation by Machine LearningA nutshell
30Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation mask refinementBinary operation
Roeder et. al. Development (2012)
The âsegmented maskâ can be refined / cleaned up by morphological operations.
This affects only the mask itself, but not the original image.
⢠Dilate: enlarge the boundary of the mask by counts/pixels
⢠Erode: shrink the boundary of the mask by counts/pixels
⢠Open: Erode then dilate: smoothens and removes isolated pixels.
⢠Close: Dilate then erode, smoothens and fills small holes.
31Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Separate objects that are connected.Acts only on the binary image.
⢠MorphologyErode then dilate, objects that are separated wonât be merged again
⢠WatershedsSeparate objects assuming they are roughly the same size, might wrongly split elongated objects
Original
Count 6
Count 9
Morphology
Count 1
Count 3
Watersheds
Segmentation mask refinementobject separation
32Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation mask separationSeparate via Morphology
original
erosion 3erosion 1 erosion 2
⢠First an erosion process is applied to all objects, which reducestheir size and separates them.
⢠The larger the object, the more counts needed for separation. Inthis example 3 erosion counts are necessary to separate allobjects completely.
dilation 3dilation 1 dilation 2
⢠After erosion, the objects are enlarged again by a dilation processto get back the original size.
⢠This dilation is done without allowing objects being merged again.
⢠Objects with different sizes cannot be handled properly⢠Erosion and dilation might change the original object shapes
original 60 counts18 counts 26 counts
33Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation mask separationSeparate via Watershed
Step 1: âDistance transformationâ of the binary image(The distance to the closest edge is calculated for each object pixel and is coded as a grey value)
Step 2: grey value as height within a âlandscapeâ(object is a basin)
Step 3: âlandscapeâ is filled with water, and âwatershedâ can be detected between two touching basins.
Step 4: Objects are separated at âwatershedâ
⢠Powerful to separate objects with different sizes⢠Do not change original object shape⢠Hole within binary object confuses the algorithm
(âfill holeâ necessary)
34Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Segmentation summaryA proper segmentation is usually a combination of a few processes
Lowpass smoothing Unsharp Mask sharpening Fill holes
Min area set 150 Watersheds separation Lowpass and watersheds
Original
35Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Feature DetectionWhat to measure?
Pre-Processing Segmentation Feature
DetectionData
Presentation
Do I measure all objects?Or conditioned objects?
e.g. measure only metaphase cells
If >1 object of interest, are they independent or associated? And how associated?
e.g. measure total mitochondria expression per cell
Do I measure parameters for a group of objects? Or a single object?
e.g. counting total cells per image, and measure intensity per cell
36Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Feature DetectionParameters of group vs single
Group parameters:⢠How many nucleus (count): 10⢠How much area do they occupy
(Area percentage): 5%
Single object parameters:⢠Area: 20 um2
⢠Mean Intensity: 500⢠Roundness: 0.9
37Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Feature DetectionObject filtering
⢠Filtering further refine/limit the segmented objects.⢠It allows detecting only specific objects, rather than all segmented objects ⢠e.g. detect only metaphase cells and not interphase cells.
Courtesy of internet
FL image
Metaphase cell
What conditions to use to isolate metaphase cells?(In metaphase, chromosomes condense)⢠Area⢠Mean intensity⢠roundness
Setup filtering condition
38Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Feature DetectionObject association
You want to quantify the total green structures (area and intensity) per cells
A counter stain channel is needed for cell identification (aka master channel)
If object of interest is outside master channel, binary process is needed
A second segmentation is performed within the master channel and parameters measured
The final data structure can be very complicated, when there are:⢠Multiple objects of interest⢠Multi-channel fluorescence data⢠Multi-dimensional data: time, z-stack, positionsPlan the data structure carefully, and minimize detected parameters
39Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Data Presentation Pre-Processing Segmentation Feature
DetectionData
Presentation
Display your data in a way that highlights the question you want to ask.
Line chart for time seriese.g. scratch assay
Histogram for distribution analysis
e.g. cell size distribution
Scatter plot for relationship analysis
e.g. multiple gene expression
Heatmap for HCSe.g. cell growth comparison
40Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Image Analysis and Microscopy Automation
â...Ultimately, the goal is to have a general platform for a wide range of applications that automatically detects the object and acts accordingly. ....â
Life science researchers, these days, require higher number of microscopy images at ever faster speed.
Microscopy automation is essential to achieve such ambitious goal.
Image analysis is one indispensable step inside microscopy automation.
41Dr. Xianke Shi, Zeiss Research Microscopy Solutions
Top Related