Learning by Demonstration
Transcript of Learning by Demonstration
![Page 1: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/1.jpg)
Learning by Demonstration
1
![Page 2: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/2.jpg)
2
How to teach a robot to play table tennis with a human?
Or how to flip pancakes?
![Page 3: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/3.jpg)
Learning from Demonstration
3
Definition: an end-user development technique for teaching a robot new behaviors by demonstrating the task to transfer directly instead of programming it.
![Page 4: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/4.jpg)
Reinforcement Learning
4
Definition: an area of machine learning inspired by behaviorist psychology, concerned with how robots ought to take actions in an environment so as to maximize a cumulative reward.
![Page 5: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/5.jpg)
Learning from Demonstration and Reinforcement Learning
5
In robotics, learning from demonstration and reinforcement learning often go hand by hand
• Learning from demonstration provides initial solution
• Reinforcement learning provides adaptation capability
![Page 6: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/6.jpg)
Examples:
6
![Page 7: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/7.jpg)
Examples:
7
![Page 8: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/8.jpg)
Examples:
8
![Page 9: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/9.jpg)
Programming by Demonstration (PbD)
9
![Page 10: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/10.jpg)
Programming by Demonstration (PbD)
10
![Page 11: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/11.jpg)
Programming by Demonstration (PbD)
11
![Page 12: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/12.jpg)
Programming by Demonstration (PbD)
12
![Page 13: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/13.jpg)
Programming by Demonstration (PbD)
13
Programming robots is hard!
•Huge number of possible tasks•Unique environmental demands•Tasks difficult to describe formally•Expert engineering impractical
![Page 14: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/14.jpg)
Programming by Demonstration (PbD)
14
How can robots be shown to perform tasks?
•Natural, expressive way to program•No expert knowledge required•Valuable human intuition•Program new tasks as-needed
![Page 15: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/15.jpg)
Record and Replay
15Then, how to integrate multiple demonstrations?
![Page 16: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/16.jpg)
Programming by Demonstration (PbD)
16
Definition (from Wiki): In computer science, programming by demonstration (PbD) is an end-user development
technique for teaching a robot new behaviors by demonstrating the task to transfer directly instead of
programming it through machine commands.
Also called:• Learning from demonstration• Imitation learning • Apprenticeship learning
![Page 17: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/17.jpg)
A.G. Billard - SHS Program in Cognitive Psychology - Spring 2007Calinon, S. and Billard, A. (2007) Incremental Learning of Gestures by Imitation in a Humanoid Robot. in Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI). (Slides credits: Aude Billard)
![Page 18: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/18.jpg)
Imitation
Level of granularity: What is copied?
Should it copy the intention,
or dynamics of movement?
Gesture Recognition
How are actions perceived?
How is information parsed?
Motor Learning
How is information transferred
and implemented on a
physical robot?
![Page 19: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/19.jpg)
Learning by Imitation
Gesture Recognition
Motor Learning
Biological
Inspiration
Robotic
Implementation
![Page 20: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/20.jpg)
Programming by Demonstration (PbD)
20
Prior to building any capability in robots, we might want to understand how the equivalent capability works in humans and other animals: BIOLOGICAL INSPIRATION
![Page 21: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/21.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Imitation Capabilities in Animals
Which species may exhibit imitation is still a main
area of discussion and debate
One differentiate “true” imitation from copying
(flocking, schooling, following), stimulus
enhancement, or contagion
![Page 22: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/22.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Imitation Capabilities in Animals
• Copying and Mimicry: Rats, Monkeys
• Observe companion actor rats performing different
spatial tasks differing according to the experimental
requirements. After the observational training,
surgical ablation to block any further learning
• The observer rats displayed exploration abilities that
closely matched the previously observed behaviors.
![Page 23: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/23.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Imitation Capabilities in Monkeys
Subjects who saw the Lever demonstrations tended to
use a levering movement to pop open the lid whereas
subjects who viewed Poke, as well as the controls, did
not display this behavior at all.
![Page 24: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/24.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Imitation Capabilities in Animals
• “True” imitation: Ability to learn new actions not part
of the usual repertoire
• Humans only, and possibly great apes
![Page 25: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/25.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Developmental Stages of Imitation
• Newborns to 3 months infants
Innate facial imitation: Tongue and lips protrusion,
mouth-opening, head movements, cheek and brow
motion, eye blinking
• Delayed imitation up to 24 hours
→ Imitation is mediated by a stored representation
![Page 26: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/26.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Developmental Stages of Imitation
• 18 months (Piaget) or 9-12 months (Meltzoff) infants
• Deferred and delayed imitation of novel behavior 67% of the infants who saw the display reproduced the act
after the week's delay, as compared to 0% of the control
infants who had not seen the novel display.
![Page 27: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/27.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Goals and Intentions
• 14 months infants
• They imitate new action to achieve the same
goal only if they consider it to be the most rational
alternative.
![Page 28: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/28.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Goals and Intentions
• 18 months infants
• Differentiate between human and machine
demonstration
→Attribute intentions only to the human
• Learn from unsuccessful examples
![Page 29: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/29.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Goals and Intentions
• Imitation is hierarchical and goal-directed
• Single-hand motions: accurate ipsilateral imitation,
48% subsitution for crosslateral imitation
• Two-hand motions: only 10% substitution for
crosslateral imitation.
• Two-phase motion eliminates mistakes
• Adding constraints of hand gestures increases mistakes
![Page 30: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/30.jpg)
Learning by ImitationBiological
Inspiration
Gesture Recognition
Imitation in adults
• Reaches highest level of complexity
• Is present in all activities:
Social influence in establishing group norms; collective
frame of reference, transmission of phoebias
![Page 31: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/31.jpg)
• Range of imitative behaviors in animals
→ Increasing in complexity across species
• Stages of development in children imitation
→ facial and motion imitation
→ inferring goals
→ hierarchy of imitation
• Imitation in adulthood is influenced by mvmt observation,
handedness, orientation of the demonstrator
• Adaptation and reinforcement (including learning from
failures) comes hand by hand with imitation learning
• The underlying neural mechanisms are not yet completely
deciphered
Imitation Learning in Animals
![Page 32: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/32.jpg)
Advantages: When is Imitation useful?
• It is a powerful means of transferring skills
• It speeds up the learning process by showing
possible solutions or conversely by showing bad
solutions
Imitation Learning in Animals
![Page 33: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/33.jpg)
Disadvantages:
When is Imitation not useful?
• Not appropriate: When a good solution for the
teacher is not a possible solution for the learner (when not considering adaptation and reinforcement)
• Disadvantageous: When it induces you in error -
bad teacher
Imitation Learning in Animals
![Page 34: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/34.jpg)
34
![Page 35: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/35.jpg)
Programming by Demonstration (PbD)
35How to integrate multiple demonstrations?
![Page 36: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/36.jpg)
Technical Details of PbD
36
• Classification vs regression
➢ Classification: the output variables take
discrete class labels
➢ Regression: the output variable takes
continuous variables
Regression can be used to integrate multiple different demonstrations
![Page 37: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/37.jpg)
Technical Details of PbD
37
• Gaussian (normal) distribution➢ Gaussian is a characteristic symmetric bell curve
that quickly falls off towards 0 (practically)
![Page 38: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/38.jpg)
Technical Details of PbD
38
• Multivariate Gaussian distributions in the n-D space
![Page 39: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/39.jpg)
Technical Details of PbD
39
• Gaussian Mixture Models (GMM)➢ Mixture model is a probabilistic model which
assumes the underlying data belongs to a mixture distribution
![Page 40: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/40.jpg)
Technical Details of PbD
40
• Gaussian Mixture Regression (GMR)➢ Mixture model is a probabilistic model which
assumes the underlying data belongs to a mixture distribution
![Page 41: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/41.jpg)
Technical Details of PbD
41
• Input trajectories from humandemonstrations
![Page 42: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/42.jpg)
Technical Details of PbD
42
• Trajectories are modeled as GMMs
The trajectory p(s,t) is encoded using a GMM, which is a continuous model.
![Page 43: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/43.jpg)
Technical Details of PbD
43
• Trajectories are modeled as GMMs
GMR is used to retrieve p(s|t), namely the expected position at each time step.
![Page 44: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/44.jpg)
Technical Details of PbD
44
• Other examples
GMM is used to model the trajectoryGMR is used to retrieve the trajectory
![Page 45: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/45.jpg)
Technical Details of PbD
45
• Robustness to perturbation
GMM is used to model the trajectoryGMR is used to retrieve the trajectory
![Page 46: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/46.jpg)
46
![Page 47: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/47.jpg)
Technical Details of PbD
47
Have we solved the problem?
• How to estimate the parameters of a Gaussian or GMM?
• How to estimate the number of Gaussian component in a GMM?
• How to align the demonstrated trajectories with different speed?
![Page 48: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/48.jpg)
Technical Details of PbD
48
Estimate parameters of a Gaussian
Maximum-likelihood estimation (MLE): find the parameters under which the data is most likely for that model• Likelihood function:
• The likelihood is thought of as a function of the parameters where the data is fixed.
![Page 49: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/49.jpg)
Technical Details of PbD
49
Estimate parameters of a Gaussian
Maximum-likelihood estimation (MLE): find the parameters under which the data is most likely for that model• In the maximum likelihood problem, our goal is to find
the that maximizes :
• Often we maximize instead because it is analytically easier.
![Page 50: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/50.jpg)
Technical Details of PbD
50
Estimate parameters of a Gaussian
Does EM work for GMMs?• The answer is no…• Since the data points are
not from the identical Gaussian components
![Page 51: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/51.jpg)
Technical Details of PbD
51
Estimate parameters of a GMM
Expectation–maximization (EM): an iterative method to find maximum likelihood estimates of parameters in statistical models, where the model depends on unobserved latent variables.
![Page 52: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/52.jpg)
Technical Details of PbD
52
Estimate GMM parameters using Expectation–maximization (EM)
![Page 53: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/53.jpg)
Technical Details of PbD
53
Estimate GMM parameters using Expectation–maximization (EM)
![Page 54: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/54.jpg)
Technical Details of PbD
54
Have we solved the problem?
• How to estimate the parameters of a Gaussian or GMM?
• How to estimate the number of Gaussian component in a GMM?
• How to align the demonstrated trajectories with different speed?
![Page 55: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/55.jpg)
Technical Details of PbD
55
Estimate # Gaussian component in GMMs
Model Selection:• Given different models (defined by different hyper-
parameter values), select the best model (i.e., the hyper-parameter resulting in best performance).
• Many methods exist based on different criteria:• Cross-validation• Others such as
Bayesian information criterion, and structural risk minimization
![Page 56: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/56.jpg)
Technical Details of PbD
56
Have we solved the problem?
• How to estimate the parameters of a Gaussian or GMM?
• How to estimate the number of Gaussian component in a GMM?
• How to align the demonstrated trajectories with different speed?
![Page 57: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/57.jpg)
Technical Details of PbD
57
Trajectory Alignment
Dynamic Time Warping (DTW): aims at aligning two sequences by warping the time axis iteratively until an optimal match between the two sequences is found• DTW is a time series alignment algorithm developed
originally for speech recognition.• Consider two trajectories (sequences of data points)
![Page 58: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/58.jpg)
Technical Details of PbD
58
Trajectory Alignment based on DTW• The two sequences are arranged on the sides of a grid,
with one on the top and the other up the left hand side.• Both sequences start on the bottom left of the grid.• Inside each cell a distance measure
can be placed, comparing the corresponding elements of the two sequences.
• To find the best match or alignment between these two sequences, one need to find a path through the grid,which minimizes the total distance between them.
• This shortest path can be found using dynamic programming.
![Page 59: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/59.jpg)
Technical Details of PbD
59
Trajectory Alignment based on DTW
![Page 60: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/60.jpg)
Technical Details of PbD
60
Trajectory Alignment based on DTW
1 3 1 2 1 1 1
1 1 3 1 2 1 11
1
2
1
3
1
1
1 3 1 2 1 1 1
0 8 0 3 0 0 0
0 8 0 3 0 0 0
3 5 3 0 3 3 3
0 8 0 3 0 0 0
8 0 8 5 8 8 8
0 8 0 3 0 0 0
0 8 0 3 0 0 0
![Page 61: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/61.jpg)
Technical Details of PbD
61
Trajectory Alignment based on DTW
1 3 1 2 1 1 1
1 1 3 1 2 1 11
1
2
1
3
1
1
1 3 1 2 1 1 1
0 8 0 3 0 0 0
0 8 0 3 0 0 0
3 5 3 0 3 3 3
0 8 0 3 0 0 0
8 0 8 5 8 8 8
0 8 0 3 0 0 0
0 8 0 3 0 0 0
1 3 1 2 1 1 1
1 1 3 1 2 1 1
![Page 62: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/62.jpg)
Technical Details of PbD
62
Have we solved the problem?
• How to estimate the parameters of a Gaussian or GMM?
• How to estimate the number of Gaussian component in a GMM?
• How to align the demonstrated trajectories with different speed?
Other general data-related issues also exist in PbD, for example, the curse of dimensionality
![Page 63: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/63.jpg)
Big Dog by Boston Dynamics
63
![Page 64: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/64.jpg)
64
![Page 65: Learning by Demonstration](https://reader030.fdocuments.in/reader030/viewer/2022012801/61bd08fa61276e740b0eb076/html5/thumbnails/65.jpg)
65