Chelsea Finn, Pieter Abbeel, Sergey Levine
Lifelong Few-Shot Learning
Lifelong Learning Goal: learn many tasks, typically in sequence
Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks
Chelsea Finn, UC Berkeley
Lifelong Learning Goal: learn many tasks, typically in sequence
Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks
Typical Approach: Run optimization algorithm (e.g. policy gradient)a.k.a. continual fine-tuning
Chelsea Finn, UC Berkeley
Fine-tuning for lifelong learning
Suppose, we have trained a model on N tasks.
Now, we want to learn a new task with little data
Why is fine-tuning not effective?
None of these are satisfactory :(
Chelsea Finn, UC Berkeley
…1 2 3 N
Option 0: collect a lot more data Option 1: Finetune on (N+1)’th task Option 2: Replay old task data, including new data
Chelsea Finn, UC Berkeley
Lifelong Meta-Learningcontinuously learn to learn sequence of tasks
[D1 for learning, D2 for meta-learning]
non-stationary task distribution
Sample a task
Sample N datapoints from
Result: Can reuse prior learning experience to quickly learn new tasks
meta-learning
, split into datasets D1 and D2
Update model so that learning using D1 enables effective performance on D2
How can we do better? Learn how to learn efficiently
Key idea: Train over many tasks, to learn parameter vector θ that transfers
Fine-tuning:task
pretrained parameters
Our method:
[test-time]
Background: Model-Agnostic Meta-Learning fine-tuning from ImageNet-trained features (Deng et al. ’09, Donahue et al. ’14)
+ simple, works well, same learning rule- no ImageNet for non-vision domains, won’t extend to few-shot setting
Chelsea Finn, UC Berkeley
One slice of lifelong learning:
Preliminary Investigation into Lifelong Meta-Learning
Chelsea Finn, UC Berkeley
- given experience with N tasks - evaluate ability to efficiently learn (N+1)’th
task from different distribution
Illustrative Regression Example
Adaptation in Reinforcement Learning
RegressionQuickly learn a new real-valued function
meta-learning
training data evaluation data
… …
Chelsea Finn, UC Berkeley
Tasks: sine function with varied amplitude & frequency
x y
new task
Comparison: task-conditioning
Chelsea Finn, UC Berkeley
x
y
Condition model on task information
Train with standard supervised learning on training tasks. Fine-tune on new task.
In-distribution task performance
Chelsea Finn, UC Berkeley
10 datapoints used for all fine-tuning steps
Out-of-distribution task performance
Chelsea Finn, UC Berkeley10 datapoints used for all
fine-tuning steps
Task representation out-of-distribution
Chelsea Finn, UC Berkeley
10 datapoints used for all fine-tuning steps
the importance of task representation
Meta-Learning: Half-Cheetah
Chelsea Finn, UC Berkeley
Training task distribution:
Quickly adapt behavior to run at a goal velocity
sa
Comparison to task-conditioning
Evaluate on:
Half-Cheetah in-distribution performance
Chelsea Finn, UC Berkeley
20 trajectories used for each fine-tuning step
Half-Cheetah out-of-distribution performance
Chelsea Finn, UC Berkeley
20 trajectories used for each fine-tuning step
Conclusion:- lifelong meta-learning enables faster learning of new tasks
Chelsea Finn, UC Berkeley
CollaboratorsSergey Levine Pieter Abbeel
Top Related