Cognitive Computer Vision

Kingsley Sagekhs20@sussex.ac.ukand

Hilary Buxtonhilaryb@sussex.ac.uk

Prepared under ECVision Specific Action 8-3http://www.ecvision.org

Lecture 15

Active Vision & cameras Research issues

Active vision

During recent years, there has been a growing interest in the use of active control of “image formation” to simplify and accelerate scene understanding

Examples of “image formation” include, for example:– gaze or focus of attention (saccadic control)– stereo viewing geometry (vergence control)– a head mounted camera

Active vision

Historical roots of “Active Computer Vision”– 1982: Term first used by Bajcsy (Nato workshop)– 1987: Paper by Aloimonos et al (ICCV)– 1989: Entire session at ICCV

References:– “Active Percpetion”, R. Bajcsy, IEEE Proceedings Vol

76, No 8, pp 996-1006, August 1988– “Active Vision”, J. Y. Aloimonos, I. Weiss and A.

Bandopadhay, ICCV, pp 333-356, 1987

Active visionTo reconstruct or not reconstruct?

“Classical” stereo correspondence reconstructs a scene in reference frame based on stereo geometry

Active vision changes vergence angles, focus etc. making reconstruction by traditional means intractable

Active systems avoid reconstruction wherever possible Many visual control tasks such as driving a car or

grasping an object can be performed by servoing directly from measurements made in the image:

– “A New Approach to Visual Servoing in Robotics”, B. Espiau, F. Chaumette and P. Rives, IEEE Trans. on Robotics and Automation 8(3), June 1992

Active visionApplication areas

Task based visual control– Example in the ActIPret project– Need to get reference for second video!!

Navigation Telepresence Wearable computing Panoramic cameras Saccadic control

Task based visual controlThe ActIPret project

In ActIPret, information about the current task (what objects are we likely to be interacting with, what types of behaviour) are used to determine in real-time an optimum viewing geometry (gaze vector, focus, zoom)

Task based visual controlSource unknown (for now)

The vision system is using an appearance based model to determine how and when it is appropriate to pickup up the part

Active vision in navigationExample: GTI Project http://www.robots.ox.ac.uk/~lav/Research/GTI/section1.html

One approach to visual navigation in cluttered environments is to recover the boundaries of free space, and then move conservatively along the middle of it.

Humans tend to prefer to cut corners by "swinging" from protruding corner to protruding corner.

Using a stereo head to recover range to a fixated point, can take the vehicle into "orbit" around the fixated point at a chosen safe radius |R| of clearance. (The sense of rotation can by chosen by using R>0 or R<0.)

TelepresenceExample: VFR Project http://www.robots.ox.ac.uk/~lav/Research/VFR/section1.html

Telepresence can be defined as the process of sensing sufficient information about the operator and task environment, and communicating this information in a sufficiently natural way to the human operator, that the operator feels physically present at the remote site.

The top movie shows an early version of a tracker using infra-red light to control 2 degrees of freedom of the head at 50Hz. The bottom movie shows a more sophisticated version controlling the head at the end of a robot arm.

Wearable computingExample: DyPERS from MIT

Panoramic vision

360° images usually achieved using a 2D imaging array looking into a rotating mirror or hemi-spherical reflector

Rotating mirror approach allows variable resolution at different angular ranges

Lots of good web links at: http://www.cis.upenn.edu/~kostas/omni.html

Panoramic vision

Panorama pictures taken from: http://cmp.felk.cvut.cz/demos/Omnivis/Photos/omniphotos.html

Panoramic vision applicationHoming robot (ICS, Greece)http://www.ics.forth.gr/~argyros/research/pan_homing.htm

Perceptual processes are addressed in the context of goals, environment and behaviour of a system

A novel, vision-based method for robot homing, the problem of computing a route so that a robot can return to its initial “home” position after the execution of an arbitrary “prior” path.

Robot tracks visual features in panoramic views of the environment that it acquires as it moves.

Panoramic vision applicationHoming robot (ICS, Greece)http://www.ics.forth.gr/~argyros/research/pan_homing.htm

When homing is initiated, the robot selects Milestone Positions (MPs) on the “prior” path by exploiting information in its visual memory. The MP selection process aims at picking positions that guarantee the success of the local control strategy between two consecutive MPs.

See website for panoramic view

Saccadic controlAttention – recognition loop (KTH, Sweden)http://www.nada.kth.se/~celle/

Scene is observed using a stereo head

Disparity between two images can be used in localise objects in a 3D plane

User saccades to an object, localised object is then recognised

Attention – recognition loop

Robots that interact with humansSONY QRIO robot

The end

Please feed back comments to Kingsley Sage or Hilary Buxton at the University of Sussex, UK