Information from PixelsDave Snowdon@davesnowdon

https://github.com/davesnowdon/ljc-information-from-pixelshttp://www.slideshare.net/DaveSnowdon1/information-from-pixels

Summary• Why? What?• Range operations and colour spaces• Kernels & convolution• Object detection• Contours• Conclusion

Why me?

• Social robotics developer• Social robots need to handle unstructured

environments• Vision is the most versatile way of sensing

the environment

Most general purpose sensor

Machine vision• Tracking movement: Dyson 360, Google

Tango• Recognising people, biometric security• Recognising medication• Image search• …

Why this is hard

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/

Why this is hard• Colour reproduction, lighting & white

balance• Perspective & rotation effects• Noise• Different scales

Rotations & perspective

Open CV

The good news• Open source• Tried and tested• Large collections of algorithms• Language bindings for C, python & java• Runs on pretty much anything (Linux, Mac,

Windows, android, iOS, RaspberryPi)

The less good news

• Native code• Java API is a bit clunky• Not much structure• Not the new shiny

Range operations & colour spaces

RGB

https://en.wikipedia.org/wiki/RGB_color_model#/media/File:RGB_color_solid_cube.png

HSV

https://upload.wikimedia.org/wikipedia/commons/a/a0/Hsl-hsv_models.svg

L*a*b* / CIELAB

https://gurus.pyimagesearch.com/wp-content/uploads/2015/03/color_spaces_lab_axis.jpg

Blob detection

Get an image• From a Java image• From video / webcam

org.opencv.videoio.VideoCapture• From fileimport org.opencv.core.Mat;Mat image = Imgcodecs.imread(filename);

org.opencv.core.Mat

new Mat(numRows, numColumns, CvType.CV_8UC3);

• Dense multi-dimensional matrix• Variants with int, double, byte values• Implements basic matrix operations

B G R B G R B G R B G RB G R B G R B G R B G RB G R B G R B G R B G R

Blur the image

Imgproc.GaussianBlur(image, result, new Size(kernelSize, kernelSize),

0.0);

Convert to HSV

Imgproc.cvtColor(input, hsv, Imgproc.COLOR_BGR2HSV);

Select only pixels in range

Core.inRange(image, low, high, result);

Erode & Dilatefinal Mat se = Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new Size(kernelSize, kernelSize));

Imgproc.erode(image, result, se, new Point(-1, -1), numIterations);

Imgproc.dilate(image, result, se, new Point(-1, -1), numIterations);

Find contours

Imgproc.findContours(image, contours, new Mat(), Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

Find largest contourcontours.stream() .max((c1, c2) -> (Imgproc.contourArea(c1) > Imgproc.contourArea(c2) ? 1 : -1)) .get();

Draw contour (for demo)

Imgproc.circle(image, centre, 5, CENTRE_COLOUR, 2);

Imgproc.drawContours(image, Arrays.asList(contour), 0, OUTLINE_COLOUR, 2);

Output image• Don’t always need to• Grab region of interestMat roi = mat.submat(Rect)• Convert to java imageBufferedImage javaImage = Util.matrixToImage(mat);Util.displayImage(command, javaImage);

• Write to fileImgcodecs.imwrite(filename, mat);

Built-in blob detection

• OpenCV has built-in blob detection: SimpleBlobDetector• blob detection by colour may not work

• Blog post: https://www.learnopencv.com/blob-detection-using-opencv-python-c/

Kernels & Convolution

Convolution

https://developer.apple.com/library/mac/documentation/Performance/Conceptual/vImage/ConvolutionOperations/ConvolutionOperations.html

Example kernels

Gaussian

Example kernels

Laplacian

Detecting blurred images

Detecting blurred images

• Want to discard images that are unlikely to be of use

• The more blurred an image is the fewer sharp edges will be found

• What happens to the laplacian of an image as it’s blurred…

Input image

Grayscale + laplacian

3x3 gaussian kernel

5x5 gaussian kernel

7x7 gaussian kernel

13x13 kernel

19x19 kernel

Variance of the laplacian

Code// apply laplacian to grayscale copy of imageImgproc.Laplacian(gray, laplacian, CvType.CV_64F);

// determine varianceMatOfDouble mean = new MatOfDouble();MatOfDouble stddev = new MatOfDouble();Core.meanStdDev(laplacian, mean, stddev);double sd = stddev.toList().get(0);double var = sd * sd;

Line following

Detect the line

What we want to do

Kernel to detect vertical lines

-1 2 -1

Mat kernel = new Mat(1, 3, CvType.CV_64F);double[] kernel_values = {-1.0, 2.0, -1.0};kernel.put(0, 0, kernel_values);

Convolve image with kernel

Imgproc.filter2D(gray, convolved, -1, kernel);

Threshold

Imgproc.threshold(convolved, thresh, 45.0, 255, Imgproc.THRESH_TOZERO);

Result

Object detection

Sliding window

http://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/

Haar features

Boosting• Train all features on every training

example• For each feature find the best threshold

which distinguished positive from negative• Select features with minimum error rate• Final classifier is weighted sum of these

weak classifiers

Cascade• Hugely expensive to compute all features on

every window location• Group features into different stages with

smaller number of features• Only proceed to next stage when previous

stage passes• In Viola-Jones paper as few as 10 features

out of 6000 might be evaluated per window

Pre-trained classifiers• front face• profile face• Full body• Upper body• Lower body• Left & right eyes (one classifier each for left & right)• Smile• Front cat face• Russian license plate

Using a classifier// create classifier object from XML definitionfinal CascadeClassifier faceClassifier = new CascadeClassifier(classifierFilename);

// apply classifer to get list of matching regionsfinal MatOfRect mor = new MatOfRect();clr.detectMultiScale(image, mor);List<Rect> result = mor.toList();

Front face detection

Training your own classifier

How to train• Create sample vectors from text files listing +ve & -ve images• opencv_createsamples -info positives.txt -num 68 -w 60 -h 98 -

vec nao.vec• Train• Haar: opencv_traincascade -data classifier -vec samples.vec -bg

negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5 -numPos 1000 -numNeg 600 -w 60 -h 98 -mode ALL -precalcValBufSize 1024 -precalcIdxBufSize 1024

• LBP : opencv_traincascade -data classifier.lbp -vec samples.vec -bg negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5 -numPos 1000 -numNeg 600 -w 60 -h 98 -featureType LBP -precalcValBufSize 1024 -precalcIdxBufSize 1024

Training docs & tutorials

• http://docs.opencv.org/trunk/dc/d88/tutorial_traincascade.html

• http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

Results

More uses for contours

Detecting geometric shapes

Find contours// use Canny edge detector on blurred grayscale imageImgproc.Canny(blurred, edges, 75, 200);

// find contoursImgproc.findContours(image, contours, new Mat(), Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

Type conversion

// need to convert the contour from a MatOfPoint to MatOfPoint2ffinal MatOfPoint2f m2f = new MatOfPoint2f();m2f.fromList(contour.toList());

Approximate shapes// approximate contour polygon with 1% or less difference in perimeterdouble perimeter = Imgproc.arcLength(m2f, true);MatOfPoint2f approx = new MatOfPoint2f();Imgproc.approxPolyDP(m2f, approx, 0.01 * perimeter, true);

// check number of line segmentsint numSides = approx.toList().size();

More information• OpenCV docs: http://docs.opencv.org/3.1.0/• Useful blogs:

• http://www.pyimagesearch.com • https://www.learnopencv.com

• https://opencv-java-tutorials.readthedocs.io/en/latest/

• Code for examples: https://github.com/davesnowdon/ljc-information-from-pixels

Summary• Colour spaces: RGB, HSV, L*a*b*• masking images using colour ranges• Finding outline of objects using contours• Convolution• Using cascade classifiers to detect objects