Lecture 2Imaging Geometry and OpenCV

Slides by:David A. Forsyth

Clark F. OlsonJean Ponce

Linda G. Shapiro

• Abstract camera model - box with a small hole in it

• Pinhole cameras work in practice

Pinhole Cameras

Distant objects are smaller

Common to draw film planein front of the focal point.Moving the film plane merelyscales the image.

Parallel lines meet

• Cartesian coordinates:– We have, by similar triangles, that:

– (X, Y, Z) ~ (f X/Z, f Y/Z, f)

– f is called the focal length.),(),,(

Z

Yf

Z

XfZYX

[X, Y, Z]

[fX/Z, fY/Z]

The equation of projection

We won’t worry much about lenses in this class.

The reason for lenses

Lens distortion

“Barrel distortion” of rectangular grid is common for inexpensive lenses.

Precision lenses can be expensive.

Zoom lenses often show severe distortion.

Fish-eye lenses also have severe distortion.

• Images are not continuous• Typically captured with a CCD camera (charge-coupled-device)• The amount of light striking each location on a grid is integrated over

some time period• Rows are read out one at a time• For color images, successive

pixels usually correspond to different colors

• High quality color cameras usea beam splitter and 3 separateCCD chips

• APS (active pixel sensor) is acheaper (lower quality) technology

Image capture

Resolution

Resolution often (but not always) refers to the number of pixels in the image.

Lower resolution has fewer pixels.

Interestingly, faces of people you know can usually be recognized at 64x64 (or less) pixels.

Squint and look at the lowest resolution image.

Some of my research

Time permitting, the following slides give a very brief overview of some of my previous and current research.

10

Visual Terrain Mapping for Mars Exploration

11

Pose Sampling for Efficient Model-Based Recognition

12

Careful sampling of the viewpoints reducesthe complexity from cubic to linear in thenumber of craters.

Robust Registration of Aerial Image Sequences

13

Goal: Provide registration between images and maps for persistent aerial surveillance

Registration results

Simple and Efficient Projective Clustering

14

Projective clustering problems have the following properties:• Many dimensions - d• Many points – n• Clusters do not form in

the full d-dimensional space

• No labeled exemplars

Keypoint recognition

15

A popular object recognition technique uses descriptive “keypoints” that have been extracted from images.

We are investigating getting better use out of the color information in the image when creating keypoint descriptors.

16

Image classificationCurrent techniques for image classification cluster image keypoints into “visual words” similar to text retrieval methods.

We are studying the use of projective clustering to improve performance.

http://www.sccs.swarthmore.edu/users/09/btomasi1/tagging-products.html

Programming in OpenCV

In OpenCV, images are represented as matrices (as in linear algebra).

Mat image = imread("photo.jpg"); // Most generic declaration

The image could have a number of underlying data types for each pixel:

uchar – unsigned byte (greyscale image)

Vec3b – vector of 3 bytes (color image)

Point2f – point in two dimensions, float

many others…

Creating images

Images can be created using a number of methods:

using namespace cv; // all my code assumes thisMat image; // creates 0x0 imageMat image = … // uses copy constructorMat image(rows, cols, type); // type is CV_8U, for exampleMat image(rows, cols, type, scalarValue);

Example: Mat allBlue(360, 480, CV_8UC3, Scalar(255, 0, 0));

Mat_<Vec3b> colorImage = imread(“color.jpg”);// Can be convenient, but now limited to Vec3b images (matrices)// Also, must declare as a similar parameter type when passed

Copying images

Be careful to remember that most image copy and pass by value methods do NOT perform a deep copy.

image2 = image1; // shallow copy

void someMethod(Mat imageParam); // shallow copy

If you want a deep copy, then use clone (or copyTo):

image2 = image1.clone(); // deep copyimage1.copyTo(image2); //

deep copy

Memory management

Memory management is handled by the Mat class.• This is different from the IplImage class in OpenCV 1• This works correctly even if multiple images share the same data• A reference count is kept for each image• The data is deallocated only when the reference count goes to

zero• However, this can allow privacy leaks unless you are careful

Backwards compatibility

OpenCV 2 is backwards compatible with OpenCV 1.

IplImage *iplIm = cvLoadImage("photo.jpg");// Do some work with image herecvReleaseImage(&iplIm); // necessary to prevent

memory leak

Can convert to Mat simply:

Mat converted(iplIm); // do not release image until finished

Image manipulation

OpenCV provides many methods to manipulate entire images:

Filtering: blur, smooth, median, gradient, laplacian

Transformations: resize, affine, perspective, color space, threshold, flood fill

Segmentation: grabCut, watershed

Feature detection: edges, corners, lines, circles, template matching, SIFT