ITS Master 15963 Paper PDF

7/29/2019 ITS Master 15963 Paper PDF

1/6

OBJECT POSITION ESTIMATION AND TRACKING

USING AN ACTIVE CAMERA NETWORK

Agus Budi Dharmawan1) 2), Prof. Dr. Karl Kleinmann1), Prof. Dr. Hermann Meuth1), Prof Mauridhi Hery Purnomo2)

1) Master of Science in Electrical Engineering, Fachbereich EIT,

University of Applied Sciences Hochschule Darmstadt (h-da), 2011.2)

Graduate Study Program, Departement Of electrical Engineering, Faculty of IndustrialTechnology, ITS Surabaya, Indonesia. 2011.

AbstractThis paper presents 3-D estimation of objects position, combining static and moving camera with the Template-Color

matching method. The template-Color matching is used to track an object of interest. Color-based tracking method

gives more confidence to track an object from many moving objects in arbitrary direction. Template is updated when

the same 3-D and 2-D position matching is found. 3-D object position is calculated by merging line of Sight from

multiple cameras. Our goal is to develop an object tracking and 3-D estimating system to aid in locating object target

within image.

1. INTRODUCTIONMachine can see and knows where the object is,

something that seems impossible. Many techniques

could be adopted to make the system more intelligent.

In image processing, camera acts as an eye. System

takes input from camera i.e. image pixels such as color

information. Image processing and networking together

is helpful in various applications. One of them is used

to find the objects position using camera networking.

Multiple cameras can be used to observe the object of

interest. As a Surveillance system, camera is used tomonitor the objects behavior, activity, or any other

changing information such as a movement and objects

localization (position). This system can run

automatically to help human for observation purpose.

Camera coordinates and object coordinates are

related and can be calculated using Denavit-Hartenberg

Calculation. It includes calculation of rotation and

translation from the base coordinates to the camera and

from camera to the object coordinate system. Position

of the object or the camera can be calculated using this

calculation.

The Idea is to track the object of interest in

arbitrary movement. Operator can chose the object by

clicking with the mouse pointer. Computer saves theselected object as a template which can be set with

different sizes. System will search the object, based on

template and color matching method. Afterwards it

sends the found objects position to server (Explorer) to

get the world coordinate. Tracking will process by

combining moving and static cameras. Each client

(Backend) will be equipped with one IP camera.

Our project will combine Image processing and

networking together. Tracking of an object and

estimating its position is done in distributed processing.

Two static and one moving camera is put in a

networking system. Operator will work with both the

systems, either in manual or automatic mode. Operator

will choose an object among many moving objects to

track it. Image processing and Kinematics calculation

will guide the system to view the desired object and

estimate its position.

In this project, another solution is proposed to

solve the problem by implementing semi-automatic

active camera system. This project is related to the

previous project. Previous system uses image

processing in a networked sensor system which

consists of several moving cameras. With this Active

Object Tracker (AOT), system can track objects using

camera. Each camera is connected, and exchanges

information of objects position. If one camera iscovered by an obstacle, the others can still follow the

object.

2. BACKGROUND2.1. Camera ModelRelationship between image plane in single camera and

image plane displayed on the monitor must be

discovered to have the proper camera calculation.

Combining between image processing and image

geometric also become important part to design the

algorithms.

Figure 1. Camera Model


2/6

Image from same scene will have relationship

between two cameras Cand C' illustrated in Figure 4.

These could be two physically separate cameras or a

single moving camera at different points in time

[Aghajan, H. et al]. Image point in C(ximage, y image) are

corresponding with image point in C' (x'image, y'image)

called point correspondence.

Relation between C and C' can be projected in

equations bellow.

Z

Xfx =

Z

Yfy = (1)

Z

Xfx

=

Z

Yfy

= (2)

f and f are focal length from C and C.

Rotation R and translation [tX tY tZ]T

are used to find

the Ccoordinate system.

+

=

z

y

x

t

t

t

Z

Y

X

R

Z

Y

X

(3)

With

++

++

=

coscossincossin

cossincossinsincoscoscossinsinsincos

sinsincoscossinsincossinsinsincoscos

R

(4)

Where and are rotation angles around X-, Y-,and Z-axes in Ccoordinate system. By substituting

equation and, relations between two coordinate are:

Z

try

f

rx

f

r

Z

ftfry

f

frx

f

fr

Z

Xfx

z

x

+++

++

+

=

=

333231

131211

(5)

Z

try

f

rx

f

r

Z

ftfry

f

frx

f

fr

Z

Yfy

z

Y

+++

++

+

=

=

33

3231

232221

(6)

Figure 2: Line Of Sight from Two Cameras

2.2. Image ProcessingThe basic method from template matching is to

have a template as a mask and use this mask to search

specific feature from the image. Templates are formed

using simple geometric shapes or silhouettes [Fieguthand Terzopoulos]. The advantage from template is it

carries both spatial and appearance information.

However, templates only encode the object appearance

generated from a single view. Thus, they are only

suitable for tracking objects whose poses do not vary

considerably during the course of tracking [Yilmaz et

al].

Template matching is a function matches an

actual image patch against an input image by sliding

the patch over the input image using one of the

matching methods [Bradsky G. et al]. RGB Channel is

used to have information from the template and find

the best likelihood from the image. Matching algorithm

will perform scan on the image to get minimumTemplate-Source Difference (TSD) value.

==

++=Tcolms

j

Trows

i

jiiyixDiffyxTSD00

),,,(),( (7)

==

Scolms

y

Srows

x

yxTSD00

),( (8)

Figure 3:Scan A Template Across An Image For

Matches

The approach is to computes the distance between

template and image source and find the minimum-

distance. It chooses smallest distance to make decision

image [Gonzales R. C. et al].

A template is a small region including a feature to

be tracked in a certain image. Using the template,

feature positions are estimated in some consecutive

images by searching minimum dissimilarity area. Then,the template is replaced by a new image.[Toshimitsu

Kaneko]

2.3. KinematicsTo modeling the serial kinematics, Denavit-

Hartenberg transformation will be used in

mathematical description. The description includes

rules on applying the coordinate systems of themembers in the kinematics chain

Relationships between inter-connected joint,

position, and orientation in rigid robots can be

calculated with the kinematics equation. Forward

kinematics is used to find the position and orientation

of the end-effectors between individual joint. The

variable are angle and distance link that calculated

by the rotation and the revolution matrix.

In Denavit-Hartenberg convention, each motor is

assumed to have degree of foredoom. Each frame will

be attached to each link and use transformation to get

relationship between those frames. Four basictransformations are two rotation matrix and two

translation matrix calculated with homogeneous

transformation.


3/6

)()()()()(111 nxnxnznznn nnnn

RotaTransRotdTransFFT = (9)

=

1000

100

0010

0001

)(1

n

nzd

dTransn (10)

=

1000

0100

00cossin00sincos

)(1

nn

nn

nznRot

(11)

=

1000

0100

0010

001

)(

n

nx

a

aTransn

(12)

=

1000

0cossin0

0sincos0

0001

)(nn

nn

nxnRot

(13)

From substitution from equation 9 13 to equation 14 :

=

1000

cossin0

sinsincoscoscossin

cossinsincossincos

)( 1nnn

nnnnnnn

nnnnnnn

nnd

a

a

FFT

(14)

Where:n : joint angle, angle betweenx in origin n-1 axes to x

in origin n axes.

n : link twist, angle betweenzin origin n-1 axes tozin

origin n axes.

dn : link offset, length between the origin n-1 to the

normal n alongZn-1 axes.

an : link length, length between the origin n-1 to the

normal n alongXn axes.

The four parameters are fulfilling each

homogeneous transformation for Denavit-Hartenberg

convention. Equation 27 can be substituted with

formula 20 and the results will be obtained in new

matrix distance. Each homogeneous transformationmatrix is represented for rotation and translation.

Parameter an is distance between Zn-1 and Zn

axis and measured along the axis x1. Angle is the

angle between the axis Zn-1 and Zn, Xn is measured

in the normal plane. Positive sense for is determined

from Zn-1 to Zn by the right-hand rule

3. APPROACH & IMPLEMENTATIONTo get more understanding from this system, first we

will define the name to be used: Explorer: Denoted for

Server. The Form will be designed with GUI Interface

for operator. Backend: Denoted for client side. One

camera will be attached to each Backend. Two Stepper

Motors can also be connected in the backend (moving

camera system).

Explorer gives the position of object of interest tothe backend. Backend will mark the object position

using the position value. In Automatic mode, Backend

will execute tracking process after having information

of the object position from the Explorer. Template and

color matching algorithm are running in Tracker class.

With this algorithm, Backend is able to perform

tracking object.

The results from tracking class are new 2-D

object positions (xobj, yobj). These results will be sent

and calculated in Kinematics class. This class performs

LOS from 2-D calculation, 2-D from 3-D calculation

and Angle from 3-D calculation.

Figure 4:Use cases overall system

To perform object tracking and estimating the

task, three different classes are implemented on each

Backend. They are Tracker System, Kinematics System

and Can Communication System.

Tracker system handles the image processing

algorithms. First process in Tracker System is showing

the image merged with template marker. To perform

this task, Tracker needs the object position information

from the Explorer. To search the objects of interest,

template matching method is implemented on the

Tracker System. After Backend gives command to

tracking the object, this algorithm will scan the image

and compare with the template (template matching).

When the minimum difference value between image

source and template is found, the algorithm will save

the position and calculate this value to get 2-D

position.

Kinematics system will perform The Denavit-

Hartenberg and Matrix calculations. 2-D position from

tracker system will be calculated to get Line of sight

(start and direction point). The LOS value is sent to the

explorer to get 3-D estimate. 3-D estimate is calculated

and stored in LOS handling class.

3-D to 2-D calculation for static camera and 3-D

to Angle calculation for rotation of the camera is also


4/6

processed in the system. In moving ca

Backend, angle values will be se

Communication system to drive the Steppe

The system uses Can Communicatio

drive the Stepper Motor. This system will

angle value from kinematics system t

direction value for driving the Stepper

Communication system will send ste

microcontroller and then with driver

values will make the Stepper Motordesired position.

Figure 5:Packet Diagram Of Overall

Tracker, Kinematics and Can Coclass are implemented on each Backend.

Cases were used in this system are LOS fr

from LOS and 3-D to 2-D-Angle.

Use Case 1: Every Backend will per

Line Of Sight under Automatic mode. T

this use case is to detected image positionclass to produce Line of Sight valu

Hatrenberg was used for calculating from

in camera to get start and direction poin

Sight.

Use Case 2 is calculating Line Of S

This process will be done in Explorer. All

values from every Backend will be sent

merge class. Explorer will estimate all LOobject positions based on world coordinate

in AOT point class. This 3-D estimation w

in System control page on GUI and se

Backend.

Use Case 3 performs two different

First is 2-D from 3-D and the second is an

D calculation. First calculation will produc

position in 2-D coordinate and the second

1 and 2. Result from the first calculation

to mark the new object in image plane,

from the second calculation will be s

Communication Class.

In Can Communication Class, the An

Input from Kinematics class which will tsteps for rotating camera. These steps wi

Stepper Motors (horizontal and vertical)

moving the camera to the desired pos

movements are being processed to make

always appear in the center of image plane.

To perform communication, twoDiscovery and Library class will plac

AOT_Discovery allow the components

network to comunicate each other. To re

components in the network, AO

creates only one discovery transmitter.

era in the

t to Can

Motor.

n system to

translate the

o step and

Motor. Can

value to

odule, the

ove to the

System

municationThree Use-

m 2-D, 3-D

form 2-D to

e goal from

rom Trackere. Denavit-

2-D position

of Line of

ight to 3-D.

ine of sight

to the LOS

S to get 3-Ds and stored

ill be shown

d to every

calculations.

gles from 3-

e new object

ill produce

will be used

and results

ent to Can

le value are

anslate intoll drive two

, afterwards

tion. These

the objects

ackages ofd between.

in the

cognize all

_Discovery

4. TESTINGTest will use robot arm (K

object. The base robot position are

Robot will move with given positio

track the object on the robot arm.

move the object with two differ

motion and PTP motion. LIN mo

object along linear paths and PTP i

from point to point until it reaches t

The system is able to find t

some failure also appears. Syste

interested object because the objecin the image. To solve this probl

can be used. With motor attached,

to always put the object in the cente

Figure 6:Tracking result wi

Failure also detected in when

the wrong object. This failure a

object moves very fast. Since t

networking to transfer data an

multiple Backend, the time proc

critical. To solve this problem, opeinterested object by clicking the act

Different Image results ar

moving camera. To solve the firs

camera will drive the Stepper Mo

value from Angle from 3-D calcul

class.

Figure 6:Tracking

KA) to move the

(1989, 3201, -930).

n path. System will

he Kuka robot will

ent positions, LIN

ion is moving the

s moving the object

e desired position.

e object. However,

cannot find the

is not as appearedm, moving camera

the camera will try

r of image.

h Kuka robot

the system detected

pears because the

e system is using

d connected with

ess becomes more

ator can correct theal object position.

e generated from

t problem, moving

tor using the angle

tion in Kinematics

esult


5/6

Hardware testing also performed to find out

whether the hardware is working properly or not. This

test was observing the Stepper Motor for moving

camera. Some movements are produce to get the

desired position. From this testing, AOT was perfectly

moving the camera with precision angle. In slave

mode, operator gives some positions to look at the

robot using result value from Tracking Testing. Camera

cam perfectly put the object in the middle of image.

5. DISCUSSION and CONCLUSIONSError object position value appears from base

value incorrect calculation (D-H parameters). Camera

calibration can be performed to solve this problem.

However, some problem still appears in this

system. Those are data transfer in networking system

and performance in tracking algorithm. This problem

appears when the system is in automatic mode. Some

delay appears in transfer data, especially for image

transfer. False Tracking position appears when the

moving camera and static camera are unsynchronized.

Different loop calculations make the moving camera

are late to send the real time LOS. Since data traffic

become issues in this project, a better tracking method

could help to improve the processing time.

Template-Color Matching also has a problem in

processing time. Template matching always scans the

image in every sequence. This problem can be reducing

with adopted the prediction method.

For better system, three or more moving camera

can be used in this system. With more moving cameras,

system can still capture the object while object try to

move to outside the camera boundary. More motor can

be attached to the system get more degree of freedom

in camera movement.

Image tracking method can be improved withprediction method. A prediction for object movement

such as optical flow and Kalman Filter can be used to

reduce the time processes. Other sensor can be used to

give another input for the system. This input will

provide more confidences to tracking and estimating

the object.

Improve the speed of Stepper Motor, to provide

fast object tracking. Fast image processing also needed

to avoid this purpose. Stereo camera can be used and

placed on a rigid body to provide moving in three

dimensional space (three degrees of freedom) such as

robot arms to get the flexibility, speed and

high accuracy of movement.

6. REFERENCESBradsky G and Kaehler A: Learning OpenCV

Computer Vision with the OpenCV Library. OReilly

(2008).

C. Zhang, J. Eggert, and N. Einecke: Robust Tracking

by Means of Template Adaptation with DriftCorrection, Springer-Verlag Berlin Heidelberg (2009).

Q. Cai and J. K. Aggarwal: Tracking Human Motion

Using Multiple Cameras. Proceedings of ICPR '96.

IEEE (1996).

Kaneko, T., Hori, O.: Template update criterion for

template matching of image Sequences. In: 16th

International Conference on Pattern Recognition, New

York, pp.15. IEEE Press, Los Alamitos (2002).

L. Velho et al., Image Processing for ComputerGraphics and Vision, Text in Computer Science, DOI

10.1007/978-1-84800-193-0-4, Springler-Verlag

London Limited (2009).

M. Kristan, J. Pers, S. Kovac, A. Leonardis: A Local-

motion-based probabilistic model for visual tracking.

Pattern Recognition Journal, Elsevier July 17, (2009).

Schmitt, Manuel; Liss, Patrick; Schmidt, Steffen ;

Hackenberger, Bernd: Project Report, Kinematics

Software Library. Hochschule Darmstadt, Mrz (2010).

Schmitt, Manuel. Entwicklung und Implementierung

einer Methode zur Kalibrierung von Schwenk-Neige-

Kameras in Sensornetzwerken. Master-Thesis.

Hochschule Darmstadt, (2011).

Wikipedia. Wikipedia Denavit-Hartenberg

Parameters. Retrieved from

http://en.wikipedia.org/wiki/Denavit-

Hartenberg_Parameters. (2011).

Zimmermann, Stephan. Design und Implementierung

einer Systemplattform fr ein aktives Kameranetzwerk.

Master-Thesis. Hochschule Darmstadt (2010).

Dharmawan, Agus Budi. Master of Science in

Electrical Engineering, Fachbereich EIT, University of

Applied Sciences Hochschule Darmstadt (h-da),

Germany.

Graduate Study Program, Departement Of electrical

Engineering, Faculty of Industrial Technology, ITS

Surabaya, Indonesia. 2011.

Object Position Estimation And Tracking Using An

Active Camera Network.


6/6

ITS Master 15963 Paper PDF

Documents

Transcript of ITS Master 15963 Paper PDF