Introduction to Image Processing and Computer Vision Rahul Sukthankar Intel Research Laboratory at...

Introduction toImage Processing and

Computer Vision

Rahul Sukthankar

Intel Research Laboratory at Pittsburghand

The Robotics Institute, Carnegie Mellon

[email protected]

Rahul Sukthankar15-829 Lecture 4

Image Processing vs. Computer Vision

• Image processing: Image image e.g., de-noising, compression, edge detection

• Computer vision: Image symbols e.g., face recognition, object tracking

• Most real-world applications combine techniques from both categories


Outline

• Operations on a single image

• Operations on an image sequence

• Multiple cameras

• Extracting semantics from images

• Applications


What is an Image?

• 2D array of pixels• Binary image (bitmap)

Pixels are bits

• Grayscale image Pixels are scalars Typically 8 bits (0..255)

• Color images Pixels are vectors Order can vary: RGB, BGR Sometimes includes Alpha


Canny Edge Detector

cvCanny(…)

Images courtesy of OpenCV tutorial at CVPR-2001


Morphological Operations

• Simple morphological operations on binary images: erosion: any pixel with 0 neighbor becomes 0 dilation: any pixel with 1 neighbor becomes 1

• Compound morphological operations:(composed of sequences of simple morphological ops) opening closing morphological gradient top hat black hat

• Aside: what is the “right” definition of “neighbor”?


Morphological Operations

Opening IoB= (IB)BDilatation IBErosion IBImage I

Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB)-IGrad(I)= (IB)-(IB)



Hough Transform

Original image Canny edge + Hough xform cvHoughLines(…)

Goal: Finding straight lines in an edge image



Distance Transform

• Distance for all non-feature points to closest feature point

cvDistTransform(…)



Flood Filling

cvFloodFill(…) grows from given seed point



Image Statistics

• Statistics are used to summarize the pixel values in a region, typically before making a decision

• Some statistics are computed over a single image: Mean and standard deviation: cvAvg(…), cvAvgSdv(…) Smallest and largest intensities: cvMinMaxLoc(…) Moments: cvGetSpatialMoment(…), cvGetCentralMoment(…)

• Others are computed over pairs/differences of images: Distances/norms C, L1, L2: cvNorm(…), cvNormMask(…) Others are computed over pairs/differences of images:

• Histograms: Multidimensional histograms: (many functions to create/manipulate) Earth mover distance – compare histograms: cvCalcEMD(…)


Image Pyramids:Coarse to Fine Processing

• Gaussian and Laplacian pyramids

• Image segmentation by pyramids



Image Pyramids:Coarse to Fine Processing

Original image Gaussian Laplacian



Pyramid-based Color Segmentation



Outline





• Applications


• Useful when camera is still and background is static or slowly-changing (e.g., many surveillance tasks)

• Basic idea: subtract current image from reference image. Regions with large differences correspond to changes.

• OpenCV supports several variants of image differencing: Average Standard deviation Running average: cvRunningAvg(…)

• Can follow up with connected components (segmentation): could use “union find” or floodfill: cvFloodFill(…)

Background Subtraction


Optical Flow

• Goal: recover apparent motion vectors between a pair of images -- usually in a video stream

• Several optical flow algorithms are available: Block matching technique: cvCalcOpticalFlowBM(…) Horn & Schunck technique: cvCalcOpticalFlowHS(…) Lucas & Kanade technique: cvCalcOpticalFlowLK(…) Pyramidal LK algorithm:

cvCalcOpticalFlowPyrLK(…)


Active Contours:Tracking by Energy Minimization

• Snake energy:

• Internal energy:

• External energy:

extEEE int

curvcont EEE int

conimgext EEE

min

,)(

,

imgcurvcont

img

img

EEEE

IgradE

IE

cvSnakeImage(…)



Camera Calibration

• Real cameras exhibit radial & tangential distortion: causes problems for some algorithms.

• First, calibrate by showing a checkerboard at various orientations:cvFindChessBoardCornerGuesses()

• Then apply an undistorting warp to each image (don’t use a warped checkerboard!)cvUndistort(…)

• If the calibration is poor, the “undistorted” image may be worse than the original.



Outline





• Applications


Stereo Vision

• Extract 3D geometry from multiple views

• Points to consider: feature- vs area-based strong/weak calibration processing constraints

• No direct support in OpenCV, but building blocks for stereo are there.


View Morphing



Outline





• Applications


Face Detection

Images courtesy of Mike Jones & Paul Viola


Classical Face Detection

SmallScale

LargeScale

Painful!

Images courtesy of Mike Jones & Paul Viola


Viola/Jones Face Detector

• Technical advantages: Uses lots of very simple box features, enabling an

efficient image representation Scales features rather than source image Cascaded classifier is very fast on non-faces

• Practical benefits: Very fast, compact footprint You don’t have to implement it!

(should be in latest version of OpenCV)


Principal Components Analysis

cvCalcEigenObjects(…)

High-dimensional data Lower-dimensional subspace



PCA for Object Recognition



Outline





• Applications


Examples of Simple Vision Systems

Shadow Elimination

• Idea: remove shadows from projected displays using multiple projectors

• OpenCV Techniques: Image differencing Image warping Convolution filters Matrix manipulation

PosterCam

• Idea: put cameras in posters and identify who reads which poster

• OpenCV Techniques: Face detection Face recognition Unsupervised clustering


display screen

P

Single Projector: Severe Shadows


display screen

P-2P-1

Two Projectors: Shadows Muted


display screen

P-2P-1

camera

Dynamic Shadow Elimination


Shadow Elimination: Challenges

• Occlusion detection: what does a shadow look like?

• Geometric issues: which projectors are occluded?

• Photometric issues: how much light removes a shadow?

• Performance: how can we do this in near real-time?

display screen

P-2P-1

camera


Shadow Elimination: Solutions

• Occlusion detection: difference image analysis

• Geometric issues: single shadow-mask for all projectors!

• Photometric issues: uncalibrated – feedback system

• Performance: only modify texture map alpha values

display screen

P-2P-1

camera


Shadow Removal witha Single Mask


Shadow Elimination Algorithm

ProjectedCamera images


PosterCam Overview

• PosterCam Hardware: Camera in each poster Embedded computer in

each poster (~ iPAQ) Network connection to

other posters


PosterCam Details

• Face detection: Viola/Jones (no float ops)

• Lighting compensation:histogram equalization

• Pose variation:additional synthetic faces

• Unsupervised clustering:k-means and nearest neighbor with non-standard distance metric


Tips on Image Processing and Coding with OpenCV

• Use the OpenCV documentation only as a guide(it is inconsistent with the code)

• Read cv.h before writing any code• OpenCV matrix functions work on images: e.g., cvSub(…)• Beware camera distortion: cvUnDistort(…) may help• Beware illumination changes:

disable auto gain control (AGC) in your camera if you are doing background subtraction

histogram equalization (often good for object recognition)• Image processing algorithms may require parameter

tuning: collect data and tweak until you get good results


Reference Reading

• Digital Image ProcessingGonzalez & Woods,Addison-Wesley 2002

• Computer VisionShapiro & Stockman,Prentice-Hall 2001

• Computer Vision: A Modern ApproachForsyth & Ponce,Prentice-Hall 2002

• Introductory Techniques for 3D Computer VisionTrucco & Verri,Prentice-Hall 1998


The End

Acknowledgments• Significant portions of this lecture were derived from the Intel

OpenCV tutorial by Gary Bradski et al. at CVPR-2001

• Thanks to my former colleagues at Compaq/HP CRL for additional slides and suggestions: Tat-Jen Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg, Gita Sukthankar, Nuno Vasconcelos, Paul Viola

Contact [email protected] if you need more information

Introduction to Image Processing and Computer Vision Rahul Sukthankar Intel Research Laboratory at...

Documents

Transcript of Introduction to Image Processing and Computer Vision Rahul Sukthankar Intel Research Laboratory at...