Lecture 8: Interacting with NAO - uni-bamberg.de 8: Interacting with NAO Cognitive Systems - Reading...

Lecture 8: Interacting with NAOCognitive Systems - Reading Club

Christian Reißner

Based on slides by Mike Beiter, Brian Coltin and Somchaya Liemhetcharat

Applied Computer Science, Bamberg University

Last change: May 28, 2014

Christian Reißner (CogSys, WIAI) RC – Interacting May 28, 2014 1 / 34

Outlook

Today’s Topics

Task Delivery

Term PaperWork Project

Interaction with NAO

Speech RecognitionFace RecognitionObject RecognitionFinite State Machine

Outlook

Today’s Topics

Task Delivery

Term Paper

Work Project

Outlook

Today’s Topics

Task Delivery

Outlook

Today’s Topics

Task Delivery

Outlook

Today’s Topics

Task Delivery

Speech Recognition

Face RecognitionObject RecognitionFinite State Machine

Outlook

Today’s Topics

Task Delivery

Speech RecognitionFace Recognition

Object RecognitionFinite State Machine

Outlook

Today’s Topics

Task Delivery

Speech RecognitionFace RecognitionObject Recognition

Finite State Machine

Outlook

Today’s Topics

Task Delivery

Task Delivery Term Paper

Written Report

Write a report about your work project

Show me in a understandable way what you did and why you did it

Give some reference to your paper talk topic

Boundary Data

Teams of two, alone or a team of four? ⇒ Decision is yoursEach of you will write one article of the paper alone ⇒ paper talk sectionThe page number should be about 10-20 per personDeadline will be the 15. of august

Written Report

Boundary Data

Written Report

Boundary Data

Written Report

Boundary Data

Written Report

Boundary DataTeams of two, alone or a team of four? ⇒ Decision is yours

Each of you will write one article of the paper alone ⇒ paper talk sectionThe page number should be about 10-20 per personDeadline will be the 15. of august

Written Report

Boundary DataTeams of two, alone or a team of four? ⇒ Decision is yoursEach of you will write one article of the paper alone ⇒ paper talk section

The page number should be about 10-20 per personDeadline will be the 15. of august

Written Report

Boundary DataTeams of two, alone or a team of four? ⇒ Decision is yoursEach of you will write one article of the paper alone ⇒ paper talk sectionThe page number should be about 10-20 per person

Deadline will be the 15. of august

Written Report

Boundary DataTeams of two, alone or a team of four? ⇒ Decision is yoursEach of you will write one article of the paper alone ⇒ paper talk sectionThe page number should be about 10-20 per personDeadline will be the 15. of august

Task Delivery Work Project

SirBot NAO - Robot of Passion

NAO is a cultivated and civilisedhumanoid robot!

He never forgets a name to a face

He shows any time an autonomousbehaviour

He loves to make smalltalk, when hemeets someone.

If he meet somebody new, heintroduces himself and people healready knows

He always greets ladies first

NAO is very sociable, when he is alonehe normally walks or looks around andlooks for somebody

Your Work

Realize the behaviour of SirBot NAO

Split the tasks into your group parts

At the end you will merge the parts into an autonomous program

Your Work

Task Delivery Further Dates

The Next Dates

04.06.: No lecture

11.06. & 18.6: The last two paper talks

25.06: Time for your work

09.07: Presentation of the work ⇒ Maybe later if you need more time

The Next Dates

04.06.: No lecture

The Next Dates

04.06.: No lecture

The Next Dates

04.06.: No lecture

The Next Dates

04.06.: No lecture

Task Delivery Questions

Questions?

Questions to the task?

Interaction

Speech Recognition

Face Recognition

Object Recognition

Speech Recognition

We will learn how to use NAO’s speech recognition

We learn NAO to recognize its own name and to give a greeting inresponse

Speech Recognition

Say ”Hallo NAO”

1 Drag a Speech Recognition box (Audio→Voice) to the workspace

2 Set the box parameter with some different names, NAO should beone of it

3 10 % should be a good setting for the recognition threshold4 Response of each name of the Speech Recognition box with a Switch

Case box (Flow Control)

⇒ The names are the cases

5 Give different responses for each understood name

6 Don’t forget the case if NAO don’t understand a word correct

7 Stop the recognition after it recognized something

Play with the threshold

Speech Recognition

Solution

Speech Recognition

Python Parakeet

Let’s do Python

Create a box and combine it with a Speech Recognition box

Give your box the output of the Speech Rec. box as a String input

Let Nao repeat what he is understanding

For the Code remember the last lectures

ttsProxy = ALProxy(“ALTextToSpeech”)

ttsProxy.say(p)

Speech Recognition

Solution

Speech Recognition

Different Answers

You may modify your box by add some ”if - elif - else” commands

give different answers for each case

Speech Recognition

Different Answers

You may modify your box by add some ”if - elif - else” commands

give different answers for each case

Face Recognition

Face Recognition is a good feature for Human-Machine-Interaction

We can now detect faces and react to sounds

At first we need to detect a face . . .

Face Recognition

Face Recognition is a good feature for Human-Machine-Interaction

We can now detect faces and react to sounds

At first we need to detect a face . . .

Face Recognition Face Detection

Face Detection

Link up . . .

a Select Camera boxa Face Detection boxa Switch boxand a Say box

→ Insert a text into the Say box if no face could be detected

⇒ Play the behaviour and show your face into the NAO’s camera

Maybe you must move your head a little bit

Face Detection

Link up . . .

a Select Camera box

a Face Detection boxa Switch boxand a Say box

Face Detection

Link up . . .

a Select Camera boxa Face Detection box

a Switch boxand a Say box

Face Detection

Link up . . .

a Select Camera boxa Face Detection boxa Switch box

and a Say box

Face Detection

Link up . . .

Face Detection

Link up . . .

Face Detection

Link up . . .

Face Detection

Link up . . .

Face Detection

Link up . . .

Face Detection

Now add a Sound Tracker box

Link it parallel to the Select Camera box

Nao should now be able to detect you by a noise (e.g. Snap yourfinger)

Face Detection

Now add a Sound Tracker box

Link it parallel to the Select Camera box

Nao should now be able to detect you by a noise (e.g. Snap yourfinger)

Face Recognition Face Scan

Face Recognition

We now can detect faces and react on sounds

Next step is to scan the direct environment for faces

Face Scan

⇒ You can add keyframes to a custom box to make the head move fromside to side

⇒ You also can track and remember the last two positions it has heard anoise, and to cycle through these positions

Not now

Face Scan

Not now

Face Scan

Not now

Face Recognition Face Recognition

Face Recognition

The robot now looks at sounds and oscillates around

Next step are to giving a greeting, make Nao wave when Nao sees aface (or other reaction)

Make Nao recognize two different faces and greet the peopledifferently

Stop scanning when recognize a face and look at that person

Object Recognition

Object recognition means to find and identify objects in an image or videosequence

Next topics

OpenCVNAO MarksExplore defining and detection general images

Object Recognition OpenCV

Computer Vision

Since version 1.14 NAOqi SDK supports OpenCV 2.3.1

Open CV is a C++ library containing various state-of-the-art visionalgorithms

from object recognition to video analysis, image processing etc.

The main modules are:

core: it contains the basic structures and functionalitieshighgui: anything to do with display, opening or saving imagesimgproc: color conversion, filters...video: optical flow, background substraction...calib3d: homography, fundamental matrices... anything to do with cameracalibration and stereovisionfeatures2d: feature detection, descriptors (FAST, SURF), matchingml: machine learning (neural networks, SVM, boosting, KNN etc)flann: efficient clustering and search in databasesobjectdetection: object detection with Cascade or Haar classifiers

Computer Vision

The main modules are:

core: it contains the basic structures and functionalitieshighgui: anything to do with display, opening or saving imagesimgproc: color conversion, filters...video: optical flow, background substraction...calib3d: homography, fundamental matrices... anything to do with cameracalibration and stereovisionfeatures2d: feature detection, descriptors (FAST, SURF), matchingml: machine learning (neural networks, SVM, boosting, KNN etc)flann: efficient clustering and search in databasesobjectdetection: object detection with Cascade or Haar classifiers

Computer Vision

The main modules are:core: it contains the basic structures and functionalitieshighgui: anything to do with display, opening or saving imagesimgproc: color conversion, filters...video: optical flow, background substraction...calib3d: homography, fundamental matrices... anything to do with cameracalibration and stereovisionfeatures2d: feature detection, descriptors (FAST, SURF), matchingml: machine learning (neural networks, SVM, boosting, KNN etc)flann: efficient clustering and search in databasesobjectdetection: object detection with Cascade or Haar classifiers

Object Recognition NAO Marks

NAO Marks

NAO can perform Computer Vision by an integrated algorithm thatdetects NAO Marks

NAO Marks have unique shapes that can be detect by the algorithm

We can use these NAO Marks as input to control NAO’s actions

Controlling with NAO Marks

Recognize different NAO Marks and react differently for each one

1 Drop a NAOMark box from the Vision category

First output returns the identification number of the NAO Markdetected (if any)Second output triggers if no NAO Mark are detected

2 Run the behaviour and show NAO the different NAO Marks

3 He should show you the different numbers(maybe let him speak them)

HINT: some Marks looks quite equal choose different looking ones to get abetter recognition

4 Connect a Switch Case box

5 Let him act different with each NAO Mark(e.g. Walk, Talk or Wave the hand)

First output returns the identification number of the NAO Markdetected (if any)

Second output triggers if no NAO Mark are detected

Solution

Object Recognition Recognize General Images

Remember General Images

1 Connect the robot and change into the Video Monitor View

2 Press the play button an see what nao is seeing

3 Look for an object you want to recognize and place it in front of theNAO’s camera

4 Click the learn button (Next to the play button)5 Click around the boarder of the object

⇒ The features of the object will be extracted within this outline

6 Enter name and title of the object, the features will be associatedwith this object in the local Choregraphe database

7 Update vision recognition database on NAO by pressing theright-most button

4 Click the learn button (Next to the play button)

5 Click around the boarder of the object

Recognize General Images

Now NAO knows the new object and we may recognize it

Drop a Vision Reco box (Vision Category)

Start the behaviour and place the object int the NAO’s camera view

The box should return the objects name

⇒ If not repeat the steps above to create a new database and send it toNAO

Walk and Find

We now want to walk around until we find the learned object

For this we use . . .a Walk Toward boxa Tactile Head boxand the Vision Rec box from before

NAO should stop walking when it sees an object and a head sensor istouched

For do so, we further need . . .a Switch Case box for the recognizable objectsand a Wait For Signals box that waits for both of its inputs beforetriggering its outputs

⇒ logical-AND operation

Now we need to stop NAO if it recognize the object and is touchedinsert a Walk Toward box and connect the trigger output with the inputset the parameters of the second box like NAO don’t moveAlso connect the trigger output the first Walk Toward box to stop it

A FSM is a model of behaviour that consists of states, state transitionsand actions.

it is dependent means it has dependent states

a state saves information about changes of the past until the present

there exist different machines like Moore, Mealy and Medwedewmachines

we want create our own FSM

a closed sequence of states

in our situation a state is a state of NAO’s behaviour

if NAO change its position or its behaviour it is a state transition

A FSM is a model of behaviour that consists of states, state transitionsand actions.

it is dependent means it has dependent states

a state saves information about changes of the past until the present

there exist different machines like Moore, Mealy and Medwedewmachines

we want create our own FSM

a closed sequence of states

in our situation a state is a state of NAO’s behaviour

if NAO change its position or its behaviour it is a state transition

normally you illustrate a FSM by a diagram that shows the states andthe transitions

Grafic by Mike Beiter at al.

”An introduction to robotics with NAO.” ALDEBARAN Robotics, 2012

Exercise

Now we will implement the exercise from the last lecture with an FSM

The robot turn its ear LED’s on and off by pressing one foot bumper,and toggle its eye colors by pressing the other foot bumper

The features are:

⇒ LED’s [on/off] – eye color[A/B]

What are the states?

⇒ four states:[ears off and eyes as color A][ears off and eyes as color B][ears on and eyes as color A][ears on and eyes as color B]

What are the transitions?

⇒ Each press of a bumper is an trigger event to the next state

Exercise

The features are:

Exercise

The features are:

Exercise

The features are:

Exercise – cont.

Let’s do it

Create the first state

A new custom box → Box type = Flow diagram and name itadd two inputs ”left”, and ”right”

Go inside the box and add a Bumpers box, an Eyes LED Box, and aEars box

Connect the boxes like shown below

Set ears intensity to 100%, and choose a eye color

We now have defined the state [ears on and eyes as color A]

Exercise – cont.

Create the other states!

Order the boxes clockwise :(ears on and eyes as color A), (ears on and eyes as color B),(ears off and eyes as color B) and (ears off and eyes as color A)If all states are configured and ordered add the transitions

The left bumper should switch states such that the eyes change colorThe right bumper event should switch states so that the ear LED’sswitch on and offConnect also the start event to the first state this defines the initialstate

Exercise – cont.

Create the other states!Order the boxes clockwise :(ears on and eyes as color A), (ears on and eyes as color B),(ears off and eyes as color B) and (ears off and eyes as color A)

If all states are configured and ordered add the transitions

Exercise – cont.

Create the other states!Order the boxes clockwise :(ears on and eyes as color A), (ears on and eyes as color B),(ears off and eyes as color B) and (ears off and eyes as color A)If all states are configured and ordered add the transitions

Exercise – cont.

The left bumper should switch states such that the eyes change color

The right bumper event should switch states so that the ear LED’sswitch on and offConnect also the start event to the first state this defines the initialstate

Exercise – cont.

The left bumper should switch states such that the eyes change colorThe right bumper event should switch states so that the ear LED’sswitch on and off

Connect also the start event to the first state this defines the initialstate

Exercise – cont.

Questions?

Lecture 8: Interacting with NAO - uni-bamberg.de 8: Interacting with NAO Cognitive Systems - Reading...

Documents

Transcript of Lecture 8: Interacting with NAO - uni-bamberg.de 8: Interacting with NAO Cognitive Systems - Reading...