testconso.typepad.com · 9x9! i!e,)*v82! 86&+%* %3+w (2! 8( 0!*2!>y >! :; ! '()*+' (),-+!!!!! *!!!!!
Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs ›...
Transcript of Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs ›...
•
•
•
•
•
•
Image source: octomap.github.io
Image source: pirobot.org/blog/0015/
• Map from first-person images to actions
• Need to learn how to reason about changing observations
• Add explicit Camera Projection and Differentiable Mapping
• Reason about the instruction on a static map
• Automatically handle changing first-person observations
Each pixel in the feature encodes an image neighbourhood
Input Image
Feature Map
Feature MapProjected Features(Map Frame)
(Image Plane in Camera Frame)
Projected features(time )
Semantic Map (time )
Semantic Map(time )
Semantic Map Grounding Map Goal Map
1x1 F
ilter
9x9 F
ilter
LSTMgo to the left side of plane
Inferred goal locationRecognized airplane
Grounding Map Goal Map
Perceptron
Forward velocity
• Output the velocity
command, given Grounding
and Goal maps
• Sent to quadcopter’s flight-
controller
Yaw rate
Agent
Action
Image Instruction
Oracle
Ground truth action
Ground truth trajectory
Modified variant of DAgger
Trade convergence guarantees for speed and memory efficiency
3500 Instructions + Environments
Ground-truth trajectories
63 Landmarks
252 Possible Tasks
Go to right side of mushroom
Total number of rollouts:3500 oracle
2000 policy
83.47
28.67
87.87
0
20
40
60
80
100
GSMN
(Ours)
NN with no
Mapping
Oracle
Outperform standard NN with no mapping
Very close to oracle performance
Feature
ExtractionMapping
1x1 Filter
9x9
FilterMLP
LSTMGo to the left side of
plane
Action
Image Features
Instruction Embedding
Grounding Map
Semantic Map
Goal Map