Introduction toImage Processing and
Computer Vision
Rahul Sukthankar
Intel Research Laboratory at Pittsburghand
The Robotics Institute, Carnegie Mellon
Rahul Sukthankar15-829 Lecture 4
Image Processing vs. Computer Vision
• Image processing: Image image e.g., de-noising, compression, edge detection
• Computer vision: Image symbols e.g., face recognition, object tracking
• Most real-world applications combine techniques from both categories
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
What is an Image?
• 2D array of pixels• Binary image (bitmap)
Pixels are bits
• Grayscale image Pixels are scalars Typically 8 bits (0..255)
• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha
Rahul Sukthankar15-829 Lecture 4
What is an Image?
• 2D array of pixels• Binary image (bitmap)
Pixels are bits
• Grayscale image Pixels are scalars Typically 8 bits (0..255)
• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha
Rahul Sukthankar15-829 Lecture 4
What is an Image?
• 2D array of pixels• Binary image (bitmap)
Pixels are bits
• Grayscale image Pixels are scalars Typically 8 bits (0..255)
• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha
Rahul Sukthankar15-829 Lecture 4
What is an Image?
• 2D array of pixels• Binary image (bitmap)
Pixels are bits
• Grayscale image Pixels are scalars Typically 8 bits (0..255)
• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha
Rahul Sukthankar15-829 Lecture 4
What is an Image?
• 2D array of pixels• Binary image (bitmap)
Pixels are bits
• Grayscale image Pixels are scalars Typically 8 bits (0..255)
• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha
Rahul Sukthankar15-829 Lecture 4
Canny Edge Detector
cvCanny(…)
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Morphological Operations
• Simple morphological operations on binary images: erosion: any pixel with 0 neighbor becomes 0 dilation: any pixel with 1 neighbor becomes 1
• Compound morphological operations:(composed of sequences of simple morphological ops) opening closing morphological gradient top hat black hat
• Aside: what is the “right” definition of “neighbor”?
Rahul Sukthankar15-829 Lecture 4
Morphological Operations
Opening IoB= (IB)BDilatation IBErosion IBImage I
Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB)-IGrad(I)= (IB)-(IB)
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Hough Transform
Original image Canny edge + Hough xform cvHoughLines(…)
Goal: Finding straight lines in an edge image
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Distance Transform
• Distance for all non-feature points to closest feature point
cvDistTransform(…)
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Flood Filling
cvFloodFill(…) grows from given seed point
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Image Statistics
• Statistics are used to summarize the pixel values in a region, typically before making a decision
• Some statistics are computed over a single image: Mean and standard deviation: cvAvg(…), cvAvgSdv(…) Smallest and largest intensities: cvMinMaxLoc(…) Moments: cvGetSpatialMoment(…), cvGetCentralMoment(…)
• Others are computed over pairs/differences of images: Distances/norms C, L1, L2: cvNorm(…), cvNormMask(…) Others are computed over pairs/differences of images:
• Histograms: Multidimensional histograms: (many functions to create/manipulate) Earth mover distance – compare histograms: cvCalcEMD(…)
Rahul Sukthankar15-829 Lecture 4
Image Pyramids:Coarse to Fine Processing
• Gaussian and Laplacian pyramids
• Image segmentation by pyramids
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Image Pyramids:Coarse to Fine Processing
Original image Gaussian Laplacian
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Pyramid-based Color Segmentation
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
• Useful when camera is still and background is static or slowly-changing (e.g., many surveillance tasks)
• Basic idea: subtract current image from reference image. Regions with large differences correspond to changes.
• OpenCV supports several variants of image differencing: Average Standard deviation Running average: cvRunningAvg(…)
• Can follow up with connected components (segmentation): could use “union find” or floodfill: cvFloodFill(…)
Background Subtraction
Rahul Sukthankar15-829 Lecture 4
Optical Flow
• Goal: recover apparent motion vectors between a pair of images -- usually in a video stream
• Several optical flow algorithms are available: Block matching technique: cvCalcOpticalFlowBM(…) Horn & Schunck technique: cvCalcOpticalFlowHS(…) Lucas & Kanade technique: cvCalcOpticalFlowLK(…) Pyramidal LK algorithm:
cvCalcOpticalFlowPyrLK(…)
Rahul Sukthankar15-829 Lecture 4
Active Contours:Tracking by Energy Minimization
• Snake energy:
• Internal energy:
• External energy:
extEEE int
curvcont EEE int
conimgext EEE
min
,)(
,
imgcurvcont
img
img
EEEE
IgradE
IE
cvSnakeImage(…)
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Camera Calibration
• Real cameras exhibit radial & tangential distortion: causes problems for some algorithms.
• First, calibrate by showing a checkerboard at various orientations:cvFindChessBoardCornerGuesses()
• Then apply an undistorting warp to each image (don’t use a warped checkerboard!)cvUndistort(…)
• If the calibration is poor, the “undistorted” image may be worse than the original.
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
Stereo Vision
• Extract 3D geometry from multiple views
• Points to consider: feature- vs area-based strong/weak calibration processing constraints
• No direct support in OpenCV, but building blocks for stereo are there.
Rahul Sukthankar15-829 Lecture 4
View Morphing
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
Face Detection
Images courtesy of Mike Jones & Paul Viola
Rahul Sukthankar15-829 Lecture 4
Classical Face Detection
SmallScale
LargeScale
Painful!
Images courtesy of Mike Jones & Paul Viola
Rahul Sukthankar15-829 Lecture 4
Viola/Jones Face Detector
• Technical advantages: Uses lots of very simple box features, enabling an
efficient image representation Scales features rather than source image Cascaded classifier is very fast on non-faces
• Practical benefits: Very fast, compact footprint You don’t have to implement it!
(should be in latest version of OpenCV)
Rahul Sukthankar15-829 Lecture 4
Principal Components Analysis
cvCalcEigenObjects(…)
High-dimensional data Lower-dimensional subspace
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
PCA for Object Recognition
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
PCA for Object Recognition
Images courtesy of OpenCV tutorial at CVPR-2001
Rahul Sukthankar15-829 Lecture 4
Outline
• Operations on a single image
• Operations on an image sequence
• Multiple cameras
• Extracting semantics from images
• Applications
Rahul Sukthankar15-829 Lecture 4
Examples of Simple Vision Systems
Shadow Elimination
• Idea: remove shadows from projected displays using multiple projectors
• OpenCV Techniques: Image differencing Image warping Convolution filters Matrix manipulation
PosterCam
• Idea: put cameras in posters and identify who reads which poster
• OpenCV Techniques: Face detection Face recognition Unsupervised clustering
Rahul Sukthankar15-829 Lecture 4
display screen
P
Single Projector: Severe Shadows
Rahul Sukthankar15-829 Lecture 4
display screen
P-2P-1
Two Projectors: Shadows Muted
Rahul Sukthankar15-829 Lecture 4
display screen
P-2P-1
camera
Dynamic Shadow Elimination
Rahul Sukthankar15-829 Lecture 4
Shadow Elimination: Challenges
• Occlusion detection: what does a shadow look like?
• Geometric issues: which projectors are occluded?
• Photometric issues: how much light removes a shadow?
• Performance: how can we do this in near real-time?
display screen
P-2P-1
camera
Rahul Sukthankar15-829 Lecture 4
Shadow Elimination: Solutions
• Occlusion detection: difference image analysis
• Geometric issues: single shadow-mask for all projectors!
• Photometric issues: uncalibrated – feedback system
• Performance: only modify texture map alpha values
display screen
P-2P-1
camera
Rahul Sukthankar15-829 Lecture 4
Shadow Removal witha Single Mask
Rahul Sukthankar15-829 Lecture 4
Shadow Elimination Algorithm
ProjectedCamera images
Rahul Sukthankar15-829 Lecture 4
PosterCam Overview
• PosterCam Hardware: Camera in each poster Embedded computer in
each poster (~ iPAQ) Network connection to
other posters
Rahul Sukthankar15-829 Lecture 4
PosterCam Details
• Face detection: Viola/Jones (no float ops)
• Lighting compensation:histogram equalization
• Pose variation:additional synthetic faces
• Unsupervised clustering:k-means and nearest neighbor with non-standard distance metric
Rahul Sukthankar15-829 Lecture 4
Tips on Image Processing and Coding with OpenCV
• Use the OpenCV documentation only as a guide(it is inconsistent with the code)
• Read cv.h before writing any code• OpenCV matrix functions work on images: e.g., cvSub(…)• Beware camera distortion: cvUnDistort(…) may help• Beware illumination changes:
disable auto gain control (AGC) in your camera if you are doing background subtraction
histogram equalization (often good for object recognition)• Image processing algorithms may require parameter
tuning: collect data and tweak until you get good results
Rahul Sukthankar15-829 Lecture 4
Reference Reading
• Digital Image ProcessingGonzalez & Woods,Addison-Wesley 2002
• Computer VisionShapiro & Stockman,Prentice-Hall 2001
• Computer Vision: A Modern ApproachForsyth & Ponce,Prentice-Hall 2002
• Introductory Techniques for 3D Computer VisionTrucco & Verri,Prentice-Hall 1998
Rahul Sukthankar15-829 Lecture 4
The End
Acknowledgments• Significant portions of this lecture were derived from the Intel
OpenCV tutorial by Gary Bradski et al. at CVPR-2001
• Thanks to my former colleagues at Compaq/HP CRL for additional slides and suggestions: Tat-Jen Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg, Gita Sukthankar, Nuno Vasconcelos, Paul Viola
Contact [email protected] if you need more information
Top Related