High-level robot behavior control using POMDPs Joelle Pineau and Sebastian Thrun Carnegie Mellon...

High-level robot behavior control using POMDPsJoelle Pineau and Sebastian Thrun

Carnegie Mellon University

CogRob2002 workshop Joelle Pineau

Introducing Pearl

• Pearl is a prototype nursing robot, providing assistance to both nurses and elderly people.

“thinkers”

eyes with cameras

sonar sensors

handlebars

wheeled base

carrying tray

LCD smile/frown


Profile of our aging population

450,000 more nurses needed by 2008 campaign to recruit and retain nurses

and other health care providers


Moving thingsaround

Moving thingsaround

Enabling use of remote

health services

Enabling use of remote

health services

Supportinginter-personal

communication

Supportinginter-personal

communication

Calling for helpin emergencies

Calling for helpin emergencies

Monitoring Rx adherence

& safety

Monitoring Rx adherence

& safety

Providinginformation

(TV, weather)

Providinginformation

(TV, weather)

Management support of

ADLs

Management support of

ADLsReminding

to eat, drink, & take meds

Reminding to eat, drink, & take meds

Providing physical

assistance

Providing physical

assistance

Linking the caregiver to resources

Linking the caregiver to resources

Our vision of robotic-assisted health-care


The Nursebot project in its early days


We need a high-level controller that can:

• prioritize goals from specialized modules

• trade-off goals with widely different costs/rewards

• trade-off between information-gathering and goal- satisfaction

• switch between tasks and share sensory information

• handle uncertainty

Speech recognition&synthesis(Ravishankar, 1996; Black et al., 1999)

Autonomous navigation(Burgard et al., 1999)

Autominder(Pollack et al., 2002)

People tracking/following(Montemerlo et al., 2002)

Top controller


High-level robot behavior control usingPartially Observable Markov Decision Processes

USER + WORLD + ROBOT

ACTIONS

OBSERVATIONSBELIEF STATE

STATE


What are POMDPs?

POMDP is n-tuple { S, A, , b, T, O, R }

POMDP task 1: State tracking:

After an action, what is the state of the world?

POMDP task 2: Computing a policy:

Which action should the controller apply next?

Very hard!

Not so hard.

Robot belief:

World state:

Control layer:

bt-1 ??

at-1

ot

st-1 st

...

...

??ot-1

...

...

rt-1 rt


Our approach: Hierarchical POMDPs

• Key Idea: Exploit hierarchical structure in the problem domain to break a problem into many “related” POMDPs.

Act

InvestigateHealth

Move

NavigateCheckPulse

AskWhere

North South East West

CheckMeds

subtask

abstract action

primitive action


Planning with Hierarchical POMDPs

• Given POMDP model M = { S, A, , b, T, O, R } and hierarchy H

• For each subtask h H:1) Set components

Ah children nodes

Sh S

h bh, Th, Oh, Rh

2) Minimize modelSh {zh(s0), …, zh(sn)}

h {yh(o0), …, yh(op)}

3) Solve subtask hh {bh, Th, Oh, Rh}

Move

Navigate AskWhere

South East WestNorth

ANav ={N,S,E,W}SNav ={X,Y}Nav ={o0,…,om}

AMove ={AskWhere,Navigate}SMove ={X’,Y’,Destination}Move ={o0,…,op}


Execution with Hierarchical POMDPs

• Step 1 - Update belief:

• Step 2 - Traversing hierarchy top-down, for each subtask:1) Get local belief.

2) Consult local policy.

3) If a is leaf node, terminate.

Else, go to that subtask.

Act

InvestigateHealth Move

NavigateCheckPulse

AskWhere

North South East West

CheckMeds

baoP

sbassPasoP

sbSs

jjii

ij

t

t

,|

,|,| 1


Experimental Setup

• Task: Robot provides reminders and guidance to elderly user.

• Action hierarchy:


Sample Scenario


Still unconvinced about the importance of uncertainty?


On the question of numericanumerical vs logicallogical representations...

• We haven’t tried using logical representations to control the robot…

• But our experience tells us that:

Uncertainty is crucial when dealing with people.

Probabilistic techniques are necessary to reason about uncertainty.

Real belief tracking and planning really matters!

High-level robot behavior control using POMDPs Joelle Pineau and Sebastian Thrun Carnegie Mellon...

Documents

Transcript of High-level robot behavior control using POMDPs Joelle Pineau and Sebastian Thrun Carnegie Mellon...