NC STATE UNIVERSITY | ntelligent Space with Time Sensitive Applications Wai-Lun Danny Leung,...

NC STATE UNIVERSITY

| ntelligent Space with Time Sensitive Applications

Wai-Lun Danny Leung, Rangsarit Vanijjirattikhan, Zheng Li, Le Xu, Tyler Richards, Rachana A Gupta and Bulent Ayhan,

Student Member, IEEE and Mo-Yuen Chow, Senior Member, IEEEAdvanced Diagnosis, Automation and Control (ADAC) Lab

Department of Electrical and Computer EngineeringNorth Carolina State UniversityRaleigh, NC 27695-7911, USA

http://[email protected]

NC STATE UNIVERSITY

ADAC, NC State University 2

Overview of presentation

Introduction to Intelligent Space (iSpace)

iSpace @ ADAC, NCSU realization and development

– Flow diagram

– iSpace settings

– Software

– Hardware

– Communication network

– Graphic User Interface (GUI)

Current research projects related to iSpace

NC STATE UNIVERSITY


What is Intelligent Space

A new concept to effectively use distributed sensors, actuators, robots, computing processors, and information technology over a physically and/or virtually connected space. For examples, a room, a corridor, a hospital, an office, or a planet.

It fuses global information within the space of interest to make intelligent operation decision such as how to move a mobile robot effectively from one location to another.

Human

Webcam

Webcam

Mobile Robot

MicrophoneMicrophone

Human-machine interaction in iSpace

NC STATE UNIVERSITY


Examples of iSpace

Space Tele-operation (Hubble telescope)

Futuristic Nursing Homes

Enterprise Main Computer

NC STATE UNIVERSITY


Current research in iSpace

Research topics by other universities– Intelligent navigation and manufacturing (Lab for Comput.

Sci., MIT)– Intelligent room (Artificial Intelligent Lab, MIT)– Distributed vision sensor fusion (Hashimoto Lab, U. of

Tokyo) Our work at ADAC

– iSpace is A large Scale Network based Mechatronics system.

– To realize and develop iSpace infrastructure to investigate research for time sensitive distributed network-based control, Real-time applications, tele-operation.

NC STATE UNIVERSITY


Johnny6 plays fetch in iSpace

Prototyping project – Johnny6 plays fetch in iSpace

– Demonstrate how iSpace makes superior decisions based on global information from distributed sensors

Game rules– The space with obstacles is

continuously monitored by a webcam.

– Johnny6 (an Unmanned Ground Vehicle, UGV) is to be commanded to go from its current location to a specific destination chosen by a user in a remote computing interface as quickly as possible while avoiding collisions.

Johnny5

NC STATE UNIVERSITY


Structure of iSpace at ADAC

Hardware – Sensors

» Webcam– Actuators

» Motors on Johnny6 (UGV)– Computer Network (IP)

» Wired and wireless connection» Remote computer controller

Software (main network controller)– Image acquisition– Image processing– Path generation– Path tracking controller

Graphic User Interface (GUI)– Any remote Computing interface in

the world (Internet)

NC STATE UNIVERSITY


Template Matching

Image acquisition– Top view

Image processing– Image Thresholding – Black

and white– Template matching– Circles used for rotation

invariance

2

( , ) ( , )( , ) 1i j i jA B A B

j i

h f f f f

1tanb f

b f

y y

x x

fB – Template image

NC STATE UNIVERSITY


Path generation

Find a path from the starting point A to the end point B for the UGV

– The path of the UGV should be as short as possible (minimize time)

– The path of the UGV should not collide with any obstructions

Fast Marching Method (by J.A. Sethian, Dept. of Mathematics, UC Berkeley)

– A numerical technique that counts the shortest distance from a point to the original point with a shortest distance update algorithm

NC STATE UNIVERSITY


Path tracking

The path tracking algorithm runs in every control loop and adjusts the speed and turn rate of the UGV to track the generated path.

1. Calculate the closest point on the generated path from the current UGV position.

2. Pick a reference point on the generated path that is a set distance in front of the UGV

3. Calculate the speed and turn rate for the UGV to reach the reference point given its current position and orientation.

NC STATE UNIVERSITY


UGV receives speed and turn rate information from control software via 802.11b wireless network channel.

Speed and turn rate are converted to left and right linear wheel velocities on the UGV and applied to the motors.

As the UGV moves, a new image is captured by the webcam to perform the next calculation of speed and turn rate using the software.

johnny6call.c johnny6.c main_all.c

Left Motor

Right Motor

velocity, turn rate

On UGVMain

Controller

Linear velocities

Johnny6 communication interface

NC STATE UNIVERSITY


Main Components of UGV (Johnny6)– PC104

» Pentium 266MHz, 64MB SDRAM» 4MB Flash Array» 10GB External Hard-drive» 802.11b wireless module» 5V DC

– Motor» 9V DC» Rated Torque=600g-cm» 165mA no load current» 415mA at 600g-cm load» Insulation R: 10M Ohm

– Interface Board» 5V DC» Uses latches and logics to perform PWM» Communicate with PC104 via parallel port

– Power Supply» 3 batteries» 3 voltage regulation circuits (5V,5V, 9V)

Hardware on Johnny6

Johnny6

NC STATE UNIVERSITY


Hardware on Ana - Next generation UV

Main Components of UGV (Ana)– PIC18F452 on board (Ana)

» 4MB Flash Array» 10GB External Hard-drive» 5V DC

– Motor» 5V Servo motor» Rated Torque=600g-cm

– ER25 wireless module» PIC18F452 for wireless card» 802.11b wireless module» Ethernet and Serial port interface» Communicate with PIC18f452 on Ana using

i2c protocol– Power Supply

» Single 12 V battery» 2 voltage regulation circuits (5V for

microcontroller and motor and 9 V for the wireless module)

NC STATE UNIVERSITY


A new path planning method

Previous Algorithm Template matching

– Need of templates for the obstacles

– Shape and size restriction on these obstacles.

New HPF based Algorithm Edge Detection

– capture a full representation of the environment

– No restrictive assumptions regarding its contents. – computationally efficient than template matching

Fast Marching– Path generation before the UGV starts to move

– Computationally inefficient.

Harmonic Potential Field– Path tracking problem converted into a goal seeking problem. (Region to point feature)

– No need to calculate the whole path to the destination beforehand.

NC STATE UNIVERSITY


Edge Detection

The zero crossing contours from Laplacian of Gaussian operator are taken as the candidate edges. The filtering adopted here is based on the strength of the edge, i.e. its contrast.

An approximated Gaussian first derivative gradient operator is used to produce an estimate of the contrasts. If the contrast of an edge is above a certain threshold, the edge is accepted as authentic; otherwise it is rejected as noise.

Image

I

Laplacian of Gaussian 2G

Gradient of Gaussian G

Zero Crossing

Magnitude Threshold(C)

Edge

E

NC STATE UNIVERSITY


A Model Based 2-D HPF

2 is the Laplace operator, is the workspace of the UGV ( 2), is the boundary of the obstacles (output of the edge detection stage), and is (xT, yT) the target point. The obstacles were represented by the repelling force and the point of destination was represented by the attractive force.

– The potential function in closed contour of will converge to a constant potential

– The obstacle free path to the target is generated by traversing the negative gradient() i.e. . The normalized gradient at each point represents the directional guidance at that point in the workspace.

Goal point chosen, shown as the small

red dot2

T T

( , ) 0 , Ωsubject to ( , ) 1 , Γand ( , ) 0 ( , ) ( , )

x y x y

x y x yx y x y x y

NC STATE UNIVERSITY


Results with HPF

0 0.2 0.4 0.6 0.8 1 1.2 1.40

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Network delay (second)

Max

imum

Dis

tanc

e E

rror

(m

eter

)

Maximum Distance Error Comparison

HPF

Fast Marching

0 0.2 0.4 0.6 0.8 1 1.2 1.40

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Network delay (second)

Mea

n D

ista

nce

Err

or

(met

er)

Mean Distance Error Comparison

Fast Marching

HPF

Actual workspace images with different backgrounds, different obstacle shapes and sizes. Red line is actual path and white line is the ideal path.

NC STATE UNIVERSITY


Main controller GUI

Remote user interface

– Display window

– Status display

– Action buttons

– Display options

– Plot selections

– Manual Commands

NC STATE UNIVERSITY


Time delay issue

Capabilities:

– Hard real-time data collection

» Image acquisition time

» Image processing time

» Network delay between remote user interface and the UGV

– Network-based control system

NC STATE UNIVERSITY


Actual network delay Histogram of network delay

Network delay and Processing delay

Time for a image to be captured and saved in the hard disk

– The average time for image acquisition ~ 0.2 seconds

Time required to extract position and orientation information of the objects and UGV

– Non real-time processing takes ~1.2 seconds (initial)

– Hard real-time processing takes ~0.02 seconds (remaining)

Round-trip time between the remote user interface and the UGV over a wireless link. The mean is 0.129 second, the median is 0.01 second

NC STATE UNIVERSITY


Network-based control and GSM

Controller

Gain Scheduler MiddlewareNetwork

traffic estimator

Feedback preprocessor

Gain scheduler

Communication Network Remote system

Probing

Control signal

Feedback signal( )

Cg

,ˆ( )ˆCty

( )Rtu

( )Ctu

( )Rty

( )Cty

( )ψq̂

( )

( ) ,()Ctu

( , , , )R R R R R tx f x p u( , , )R R R R Ry h x p u

q̂

q

N

Network delay is a critical issue in network-based control, as shown in the results of the iSpace project

Gain Scheduler Middleware (GSM) is a technology that allows the communication network to be transparent to controller and remote system, alleviating the adverse effects of network delay

GSM enables a conventional controller for network-based control purpose

Research Challenges– Time delay alleviation– Path tracking control algorithms– Remote wireless control via Internet

NC STATE UNIVERSITY


GSM results

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

x [m]

y [m

]

Local Tracking Control

Actual Robot TrajectoryReference Path

0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

x [m]

y [m

]

Remote Tracking Control with GSM


0 0.2 0.4 0.6 0.8 1 1.2

0

0.2

0.4

0.6

0.8

1

x [m]

y [m

]

Remote Tracking Control Without GSM


0 5 10 15 20 250

0.05

0.1

0.15

0.2

0.25

time [s]

Pa

th D

evi

atio

n E

rro

r [m

]

Error of Local Tracking Control

StraightLine

LargeCurve

Small Curve

0 5 10 15 20 250

0.05

0.1

0.15

0.2

0.25

time [s]

Pa

th D

evi

atio

n E

rro

r [m

]

Error of Remote Tracking Control without GSM

StraightLine

Large Curve

0 5 10 15 20 250

0.05

0.1

0.15

0.2

0.25

time [s]

Pa

th D

evi

atio

n E

rro

r [m

]

Error of Remote Tracking Control with GSM

Large Curve Small Curve StraightLine

Operation terminated

P a t h - t r a c k i n gc o n t r o l l e r M o b i l e R o b o t

F e e d b a c k D a t a

C o n t r o l S i g n a l

H a s h i m o t oL a b

A D A CL a b

NC STATE UNIVERSITY


Introduction to Intelligent Space (iSpace) iSpace at ADAC realization and development

– iSpace structure (A large Scale Mechatronics System)– Software

» Image acquisition and processing» Path generation and tracking

– Hardware – Communication network– Main Controller GUI

Infrastructure for research– Network-based control system– Hard real time data Analysis– Path planning and Path tracking

Experimental results Future research

– Network delay alleviation, Path planning, Security

Conclusion

NC STATE UNIVERSITY


ADAC Members

http://[email protected]

NC STATE UNIVERSITY


NC STATE UNIVERSITY | ntelligent Space with Time Sensitive Applications Wai-Lun Danny Leung,...

Documents

Transcript of NC STATE UNIVERSITY | ntelligent Space with Time Sensitive Applications Wai-Lun Danny Leung,...