Cognitive Computer Vision

Kingsley Sagekhs20@sussex.ac.ukand

Hilary Buxtonhilaryb@sussex.ac.uk

Prepared under ECVision Specific Action 8-3http://www.ecvision.org

Course outline

What is Cognitive Computer Vision (CCV) ? Generative models Graphical models Techniques for modelling cognitive aspects of CCV

– Bayesian inference– Markov Models

Research issues Coursework and case studies

So what is CCV ?

D. Vernon, Dagstuhl 2003 Monday 27th October 2003

Bernd Neumann, 2003 (ECVision Summer School on Cognitive Vision)

Cognitive Vision research requires multidisciplinary efforts and escape from traditional research community boundaries.

Computer Vision• object recognition, tracking• bottom-up image analysis• geometry and shape• hypothesize-and-test control• probabilistic methods

Knowledge Representation & Reasoning• KR languages• logic-based reasoning services • default theories• reasoning about actions & change• Description Logics• spatial and temporal calculi

Robotics• planning, goal-directed behaviour• manipulation• sensor integration• navigation• localization, mapping, SLAM• integrative architectures

Learning & Data Mining• concept learning• inductive generalization• clustering• knowledge discovery

Cognitive Science• psychophysical models• neural models• conceptual spaces• qualitative representations• naive physics

Uncertain Reasoning• Bayesian nets, belief nets• decision & estimation• causality• probabilistic learning

Natural Language• high-level concepts• qualitative descriptions• NL scene descriptions• communication

CognitiveVision

Cognitive Systems LaboratoryCSL

So what is CCV ?

In this course, we focus on using of ideas from cognitive science and psychology to do CCV

To show how we can build effective CCV systems that are more robust and more capable of solving non-trivial problems than those that do not embrace these ideas

Use statistical inference and machine learning as our tools for modelling cognitively inspired processes

We are not claiming “hard AI” in this course

Key Cognitive Elements

Objects, events, activities and behaviours– “What is it that we are observing?”

Attention and control– “How is it that we observe?”

Key Cognitive Elements

Visual learning and memory– Representation of objects and their behaviour– Recognition– Categorisation– These are “what” problems

Visual control and attention– Perception for tasks using models of expectation– Goals, task context– Resources, embodiment– These are “how” problems

Cognition– From perception to action

Key Cognitive Elements

Visual learning and memory - examples– Learning about objects and how their appearance

can change– Recognising activities by the interactions between

objects– Extracting invariant models from training data

Learning and “recognising” objects

(Murase and Nayar, 1996)

Learn and recognise activities

Coupled Hidden Markov Models (CHMM) techniques(Oliver, Rosario & Pentland, 1999)Activities with interactions via coupled states in a HMM

Learning invariant models

Variances for 3 clusters

Means for 3 clusters

Key Cognitive Elements

Visual control and attention– A framework for attentional control– Inferring likely behaviour using Bayes nets– Deictic markers– Attentional selection of objects

A Framework For Task Based Visual Control

Scene Interpretation

……

CONTROL POLICY(WITH STATE MEMORY)

FEATURECOMBINATION

d1 d2dNImage Data

Driven

Task Based Control

BBN Inference of likely vehicle tracks

Fixed camera gives direct set ofdependencies Image Grid PositionBBN has size/orient hidden nodes Leaf nodes ls1/2, lo1/2 observables

IGP

size orient

ls1 lo1 lo2 ls2

Gong and Buxton, 1993

Deictic Markers in inference of behaviour

Left: attention for overtake (overtaken & overtaking vehicle)

Right: attention for giveway (stopped & blocker vehicle plus

ground-plane conflict zone)

Howarth and Buxton,1996

Attentional selection using eye gaze

Attentional selection using predicted trajectory data

Attentional selection using predicted trajectory data

Attentional selection using predicted Space of Interest

Summary

Cognitive Computer Vision is a multi-disciplinary area of research

Here we use statistical inference and learning for robust models

Task based attentional control is key to prediction and cognitive systems design

Useful reference: “Visual surveillance in a dynamic and uncertain world” Buxton, H and Gong, S, Artificial Intelligence 78, pp 431-459, 1995

Next time …

Generative models– What are they?– Why are they so important to Cognitive Vision?