Introduction to Image Processing and Computer Vision

Introduction toImage Processing and

Computer Vision

Rahul Sukthankar

Intel Research Laboratory at Pittsburghand

The Robotics Institute, Carnegie Mellon

[email protected]

Rahul Sukthankar15-829 Lecture 4

Image Processing vs. Computer Vision

• Image processing: Image image e.g., de-noising, compression, edge detection

• Computer vision: Image symbols e.g., face recognition, object tracking

• Most real-world applications combine techniques from both categories


Outline

• Operations on a single image

• Operations on an image sequence

• Multiple cameras

• Extracting semantics from images

• Applications


What is an Image?

• 2D array of pixels• Binary image (bitmap)

Pixels are bits

• Grayscale image Pixels are scalars Typically 8 bits (0..255)

• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha


Canny Edge Detector

cvCanny(…)

Images courtesy of OpenCV tutorial at CVPR-2001


Morphological Operations

• Simple morphological operations on binary images: erosion: any pixel with 0 neighbor becomes 0 dilation: any pixel with 1 neighbor becomes 1

• Compound morphological operations:(composed of sequences of simple morphological ops) opening closing morphological gradient top hat black hat

• Aside: what is the “right” definition of “neighbor”?


Morphological Operations

Opening IoB= (IB)BDilatation IBErosion IBImage I

Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB)-IGrad(I)= (IB)-(IB)



Hough Transform

Original image Canny edge + Hough xform cvHoughLines(…)

Goal: Finding straight lines in an edge image



Distance Transform

• Distance for all non-feature points to closest feature point

cvDistTransform(…)



Flood Filling

cvFloodFill(…) grows from given seed point



Image Statistics

• Statistics are used to summarize the pixel values in a region, typically before making a decision

• Some statistics are computed over a single image: Mean and standard deviation: cvAvg(…), cvAvgSdv(…) Smallest and largest intensities: cvMinMaxLoc(…) Moments: cvGetSpatialMoment(…), cvGetCentralMoment(…)

• Others are computed over pairs/differences of images: Distances/norms C, L1, L2: cvNorm(…), cvNormMask(…) Others are computed over pairs/differences of images:

• Histograms: Multidimensional histograms: (many functions to create/manipulate) Earth mover distance – compare histograms: cvCalcEMD(…)


Image Pyramids:Coarse to Fine Processing

• Gaussian and Laplacian pyramids

• Image segmentation by pyramids



Image Pyramids:Coarse to Fine Processing

Original image Gaussian Laplacian



Pyramid-based Color Segmentation



Outline





• Applications


• Useful when camera is still and background is static or slowly-changing (e.g., many surveillance tasks)

• Basic idea: subtract current image from reference image. Regions with large differences correspond to changes.

• OpenCV supports several variants of image differencing: Average Standard deviation Running average: cvRunningAvg(…)

• Can follow up with connected components (segmentation): could use “union find” or floodfill: cvFloodFill(…)

Background Subtraction


Optical Flow

• Goal: recover apparent motion vectors between a pair of images -- usually in a video stream

• Several optical flow algorithms are available: Block matching technique: cvCalcOpticalFlowBM(…) Horn & Schunck technique: cvCalcOpticalFlowHS(…) Lucas & Kanade technique: cvCalcOpticalFlowLK(…) Pyramidal LK algorithm:

cvCalcOpticalFlowPyrLK(…)


Active Contours:Tracking by Energy Minimization

• Snake energy:

• Internal energy:

• External energy:

extEEE int

curvcont EEE int

conimgext EEE

min

,)(

,

imgcurvcont

img

img

EEEE

IgradE

IE

cvSnakeImage(…)



Camera Calibration

• Real cameras exhibit radial & tangential distortion: causes problems for some algorithms.

• First, calibrate by showing a checkerboard at various orientations:cvFindChessBoardCornerGuesses()

• Then apply an undistorting warp to each image (don’t use a warped checkerboard!)cvUndistort(…)

• If the calibration is poor, the “undistorted” image may be worse than the original.



Outline





• Applications


Stereo Vision

• Extract 3D geometry from multiple views

• Points to consider: feature- vs area-based strong/weak calibration processing constraints

• No direct support in OpenCV, but building blocks for stereo are there.


View Morphing



Outline





• Applications


Face Detection

Images courtesy of Mike Jones & Paul Viola


Classical Face Detection

SmallScale

LargeScale

Painful!

Images courtesy of Mike Jones & Paul Viola


Viola/Jones Face Detector

• Technical advantages: Uses lots of very simple box features, enabling an

efficient image representation Scales features rather than source image Cascaded classifier is very fast on non-faces

• Practical benefits: Very fast, compact footprint You don’t have to implement it!

(should be in latest version of OpenCV)


Principal Components Analysis

cvCalcEigenObjects(…)

High-dimensional data Lower-dimensional subspace



PCA for Object Recognition



Outline





• Applications


Examples of Simple Vision Systems

Shadow Elimination

• Idea: remove shadows from projected displays using multiple projectors

• OpenCV Techniques: Image differencing Image warping Convolution filters Matrix manipulation

PosterCam

• Idea: put cameras in posters and identify who reads which poster

• OpenCV Techniques: Face detection Face recognition Unsupervised clustering


display screen

P

Single Projector: Severe Shadows


display screen

P-2P-1

Two Projectors: Shadows Muted


display screen

P-2P-1

camera

Dynamic Shadow Elimination


Shadow Elimination: Challenges

• Occlusion detection: what does a shadow look like?

• Geometric issues: which projectors are occluded?

• Photometric issues: how much light removes a shadow?

• Performance: how can we do this in near real-time?

display screen

P-2P-1

camera


Shadow Elimination: Solutions

• Occlusion detection: difference image analysis

• Geometric issues: single shadow-mask for all projectors!

• Photometric issues: uncalibrated – feedback system

• Performance: only modify texture map alpha values

display screen

P-2P-1

camera


Shadow Removal witha Single Mask


Shadow Elimination Algorithm

ProjectedCamera images


PosterCam Overview

• PosterCam Hardware: Camera in each poster Embedded computer in

each poster (~ iPAQ) Network connection to

other posters


PosterCam Details

• Face detection: Viola/Jones (no float ops)

• Lighting compensation:histogram equalization

• Pose variation:additional synthetic faces

• Unsupervised clustering:k-means and nearest neighbor with non-standard distance metric


Tips on Image Processing and Coding with OpenCV

• Use the OpenCV documentation only as a guide(it is inconsistent with the code)

• Read cv.h before writing any code• OpenCV matrix functions work on images: e.g., cvSub(…)• Beware camera distortion: cvUnDistort(…) may help• Beware illumination changes:

disable auto gain control (AGC) in your camera if you are doing background subtraction

histogram equalization (often good for object recognition)• Image processing algorithms may require parameter

tuning: collect data and tweak until you get good results


Reference Reading

• Digital Image ProcessingGonzalez & Woods,Addison-Wesley 2002

• Computer VisionShapiro & Stockman,Prentice-Hall 2001

• Computer Vision: A Modern ApproachForsyth & Ponce,Prentice-Hall 2002

• Introductory Techniques for 3D Computer VisionTrucco & Verri,Prentice-Hall 1998


The End

Acknowledgments• Significant portions of this lecture were derived from the Intel

OpenCV tutorial by Gary Bradski et al. at CVPR-2001

• Thanks to my former colleagues at Compaq/HP CRL for additional slides and suggestions: Tat-Jen Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg, Gita Sukthankar, Nuno Vasconcelos, Paul Viola

Contact [email protected] if you need more information

Introduction to Image Processing and Computer Vision

Documents

Transcript of Introduction to Image Processing and Computer Vision