Mobile Image Processing on Android - WordPress.com · Bernd Girod, David Chen: EE368 Mobile Image...

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 3 no. 1

Mobile image processing – Part 3

Mobile application development on AndroidProject 1: “Hello Image”, reading/writing/modifying imagesProject 2: “Viewfinder EE368”, processing viewfinder framesProject 3: “CVCamera”, utilizing OpenCV functionsMobile application examples


http://www.stanford.edu/~gtakacs/publication.html

Mobile feature tracking


http://www.stanford.edu/~dmchen/mvs.html

Mobile book spine recognition


Mobile book spine recognition

Server

Mobile Device

TrackCamera Pose

…

Low Motion

High Motion

Send Query Frame

QueryVocab. Tree

Check Geometry

Send Response

Time

Wireless Network

Viewfinder Frames

Signal Processing and Linear Systems

Extract Features

Segment Spines


Jon’s Java Imaging Library (JJIL)

Publicly available source codehttp://code.google.com/p/jjil/

Collection of common image processing functionsImages are processed through Sequences

Rgb3x3Average

RgbSubSample

SequenceRgbImageRgbImage

Sequence.Add(new Rgb3x3Average)

Sequence.Add(new RgbSubSample)

Sequence.Push(RgbImage) Sequence.GetFront()


JJIL: 3x3 averaging filter

Original Image Processed Image

Rgb3x3Average


JJIL: Horizontal differences

Original Image Processed Image

RgbAvgGray Gray8HorizSimpleEdge


JJIL functions

ApplyMaskRgbBinaryHeapComplex32Gray32Complex32IFFTFft1dGray16Gray8Gray16LinCombGray16ThresholdGray32DivGray32Gray8Gray32Scale2Gray8Gray32ThresholdGray3Bands2RgbGray82Gray32Gray8AbsGray8AddGray8AndGray8CannyHoriz

Gray8ConnCompGray8CropGray8DeblurHorizHalftoneGray8DetectHaarMultiScaleGray8FftGray8GaussDeblurHorizGray8GaussHorizGray8GaussSmoothVertGray8Gray32Gray8HistGray8HistEqGray8HistMatchGray8HorizSimpleEdgeGray8HorizSumGray8HorizVarGray8HorizVertContrastGray8InverseFilterGray8LinComb

Gray8Peak3x3Gray8ReduceGray8ShrinkLinefitHoughRgb3x3AverageRgbAvgGrayRgbClipRgbCropRgbDimMaskRgbHorizGaussSmoothRgbHsvRgbMaskedAbsDiffRgbMaxContrast2GrayRgbMaxDiffRgbMeanStdDevRgbMinDiffRgbSelectGrayRgbSubSample…


“CVCamera” project

Goals of this projectLearn how to incorporate C/C++ code into an Android programLearn how to utilize OpenCV library functionsLearn how to draw feature keypoints on viewfinder frames

Step-by-step instructions available in Tutorial #2:http://ee368.stanford.edu/Android


OpenCV: Open Source Computer Vision

http://opencv.willowgarage.com/wiki/Welcome


OpenCV for Android

Port of OpenCV to Android frameworkOver 2K optimized algorithms written in C/C++Compiled with customized Android NDK (Crystax r4)Enables popular CV functions to be used on mobile images/videos

Differences from regular Android programmingRequires writing C/C++ codeRequires writing Java Native Interface (JNI) wrappersRequires using Android NDK in addition to Android SDK

Class tutorial #2 helps you make the transitionHow to set up OpenCV programming environmentHow to write Android apps that call OpenCV functionsHow to integrate NDK compilation into Eclipse IDE


Project structure for “CVCamera”

CVCameraActivity (Java)

Processor(C++)

CVCameraJNI

(Java)

Manages user interface and program execution

Processes a “pool” of images

Interfaces with C/C++ code

OpenCV Library


CVCamera class: menu options

public boolean onCreateOptionsMenu(Menu menu) {menu.add("FAST");menu.add("STAR");menu.add("SURF");menu.add("MSER");menu.add("Chess");menu.add("Settings");return true;

}


CVCamera class: calling feature extractors

public boolean onOptionsItemSelected(MenuItem item) {LinkedList<PoolCallback> defaultCallbackStack =

new LinkedList<PoolCallback>();defaultCallbackStack.addFirst(myGLView.getDrawCallback());

if (item.getTitle().equals("FAST")) {defaultCallbackStack.addFirst(new FastProcessor());

}else if (item.getTitle().equals("STAR")) {

defaultCallbackStack.addFirst(new STARProcessor());}else if (item.getTitle().equals("SURF")) {

defaultCallbackStack.addFirst(new SURFProcessor());}else if (item.getTitle().equals("MSER")) {

defaultCallbackStack.addFirst(new MSERProcessor());}

myPreview.addCallbackStack(defaultCallbackStack);return true;

}

Stack of callback functions

Draw callback

Feature extraction callbacks




Processor(C++)

CVCameraJNI

(Java)




OpenCV Library


CVCamera JNI class: interface to C/C++ code

static {try {

System.loadLibrary("android-opencv");System.loadLibrary("cvcamera");

} catch (UnsatisfiedLinkError e) {throw e;

}}

public final static native int DETECT_FAST_get();public final static native int DETECT_STAR_get();public final static native int DETECT_SURF_get();public final static native int DETECT_MSER_get();public final static native long new_Processor();public final static native void delete_Processor(long jarg1);

Each function prototype

corresponds to an external C/C++

function

Load OpenCV and CVCamera

libraries built by the NDK




Processor(C++)

CVCameraJNI

(Java)




OpenCV Library


Processor class: initialize feature detectors

class Processor {private:cv::StarFeatureDetector my_stard;cv::FastFeatureDetector my_fastd;cv::SurfFeatureDetector my_surfd;cv::MserFeatureDetector my_mserd;std::vector<cv::KeyPoint> my_keypoints;

public:Processor():

my_stard(STAR detector parameters),my_fastd(FAST detector parameters),my_surfd(SURF detector parameters),my_mserd(MSER detector parameters){}

void detectAndDrawFeatures(int idx, image_pool* pool, int feature_type);

};

Instantiation list

Different feature extractors stored

as member objects


Processor class: detect and draw keypoints// Detect feature keypointsMat grey_im = pool->getGrey(idx);Mat color_im = pool->getImage(idx);my_keypoints.clear();my_mserd->detect(grey_im,

my_keypoints);

// Draw feature keypointsvector<KeyPoint>::const_iterator it;for (it = my_keypoints.begin();

it != my_keypoints.end(); ++it) {// Draw black circlePoint2f pt = it->pt;circle(color_im, pt, it->size,

cvScalar(0,0,0,0), 2);// Draw yellow circlept.x += 1;pt.y += 1;circle(color_im, pt, it->size,

cvScalar(255,255,0,0), 2);}


CVCamera demo

http://ee368.stanford.edu/Android


Communications between phone and server

Phone

Server

HTTP Post (e.g., Image, Features)

HTTP Get (e.g., Identification)

Wireless Network

Apache

PHP

MATLAB / C


HTTP file upload

URL connectURL = new URL(web address);HttpURLConnection conn = connectURL.openConnection();conn.setRequestMethod(“POST”);DataOutputStream dos = new DataOutputStream(conn.getOutputStream());FileInputStream fis = new FileInputStream(file location);

int bufferSize = 1024;byte[] buffer = new byte[bufferSize];int bytesRead = fileInputStream.read(buffer, 0, bufferSize);

while (bytesRead > 0) {dos.write(buffer, 0, bufferSize);int bytesAvailable = fis.available();bufferSize = Math.min(bytesAvailable, bufferSize);bytesRead = fis.read(buffer, 0, bufferSize);

}

dos.flush();dos.close();fis.close();


Class Project: Mobile Panorama Creation

T. Chu, B. Meng, and Z. Wang, Spring 2010http://ee368.stanford.edu/Project_10


Class Project: Mobile Face Recognition

G. Davo. K. Sriadibhatla, and X. Chao, Spring 2010http://ee368.stanford.edu/Project_10


Android resources

Developer websitehttp://developer.android.comClass tutorialshttp://ee368.stanford.edu/AndroidAndroid-related office hoursWednesday, 3:30pm – 5:00pm, SCIEN LabReference bookMeier, Professional Android 2 Application Development

Mobile Image Processing on Android - WordPress.com · Bernd Girod, David Chen: EE368 Mobile Image...

Documents

Transcript of Mobile Image Processing on Android - WordPress.com · Bernd Girod, David Chen: EE368 Mobile Image...