Collaborative Reinforcement Learning
Presented by Dr. Ying Lu
Credits
Reinforcement Learning: A User’s Guide. Bill Smart at ICAC 2005
Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill, "Collaborative Reinforcement Learning of Autonomic Behaviour", 2nd International Workshop on Self-Adaptive and Autonomic Computing Systems, pages 700-704, 2004. [Winner Best Paper Award].
What is RL?
“a way of programming agents by reward and punishment without needing to specify how the
task is to be achieved”
[Kaelbling, Littman, & Moore, 96]
Basic RL Model
1. Observe state, st
2. Decide on an action, at
3. Perform action
4. Observe new state, st+1
5. Observe reward, rt+1
6. Learn from experience7. Repeat
Goal: Find a control policy that will maximize the observed rewards over the lifetime of the agent
AS R
World
An Example: Gridworld
Canonical RL domain• States are grid cells• 4 actions: N, S, E, W• Reward for entering top right cell• -0.01 for every other move
Maximizing sum of rewards Shortest path• In this instance
+1
The Promise of RL
Specify what to do, but not how to do it• Through the reward function• Learning “fills in the details”
Better final solutions• Based on actual experiences, not programmer
assumptions
Less (human) time needed for a good solution
Mathematics of RL
Before we talk about RL, we need to cover some background material
• Some simple decision theory• Markov Decision Processes• Value functions
Making Single Decisions
Single decision to be made• Multiple discrete actions• Each action has a reward associated
with it
Goal is to maximize reward• Not hard: just pick the action with the largest reward
State 0 has a value of 2• Sum of rewards from taking the best action from the
state
0
1
2
A
B2
1
Markov Decision Processes
We can generalize the previous example to multiple sequential decisions
• Each decision affects subsequent decisions
This is formally modeled by a Markov Decision Process (MDP)
0
1
2
A
B2
1
5
3
4
AA
-1000
1
A A10
1B
1
Markov Decision Processes
Formally, an MDP is• A set of states, S = {s1, s2, ... , sn}
• A set of actions, A = {a1, a2, ... , am}
• A reward function, R: SAS→• A transition function,
We want to learn a policy, : S →A• Maximize sum of rewards we see over our lifetime
aai,s|jsPP tt1taij
Policies
There are 3 policies for this MDP1. 0 →1 →3 →5
2. 0 →1 →4 →5
3. 0 →2 →4 →5
Which is the best one?
0
1
2
A
B2
1
5
3
4
AA
-1000
1
A A10
1B
1
Comparing Policies
Order policies by how much reward they see1. 0 →1 →3 →5 = 1 + 1 + 1 = 3
2. 0 →1 →4 →5 = 1 + 1 + 10 = 12
3. 0 →2 →4 →5 = 2 – 1000 + 10 = -988
0
1
2
A
B2
1
5
3
4
AA
-1000
1
A A10
1B
1
Value Functions
We can define value without specifying the policy• Specify the value of taking action a from state s and
then performing optimally• This is the state-action value function, Q
0
1
2
A
B2
1
5
3
4
A
-1000
1
A10
1B
1
Q(0, A) = 12 Q(0, B) = -988
Q(3, A) = 1
Q(4, A) = 10
Q(1, A) = 2Q(1, B) = 11
Q(2, A) = -990
A
A
How do you tell whichaction to take from
each state?
Value Functions
So, we have value function• Q(s, a) = R(s, a, s’) + maxa’ Q(s’, a’)
In the form of• Next reward plus the best I can do from the next state
These extend to probabilistic actions•
s’ is thenext state
a' ,s'Q maxs' a, s,RPas,Q a's'
as's,
Getting the Policy
If we have the value function, then finding the best policy is easy
• (s) = arg maxa Q(s, a)
We’re looking for the optimal policy, (s)• No policy generates more reward than
Optimal policy defines optimal value functions•
The easiest way to learn the optimal policy is to learn the optimal value function first
a' ,s'Qmaxs' a, s,Ras,Q *a'*
Collaborative Reinforcement Learningto Adaptively Optimize MANET Routing
Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill
Overview
Building autonomic distributed systems with self* properties
• Self-Organizing• Self-Healing• Self-Optimizing
Add collaborative learning mechanism to self-adaptive component modelImproved ad-hoc routing protocol
Introduction
Autonomous distributed systems will consist of interacting components free from human interference
• Existing top-down management and programming solutions require too much global state
• Bottom up, decentralized collection of components who make their own decisions based on local information
• System wide self* behavior emerges from interactions
Self-* Behavior
Self-adaptive components that change structure and/or behavior at run-time, adapt to
• discovered faults• reduced performance
Requires active monitoring of component states and external dependencies
Self-* Distributed Systems using Distributed (collaborative) Reinforcement Learning
For complex systems, programmers cannot be expected to describe all conditions
• Self-adaptive behavior learnt by components• Decentralized co-ordination of components to
support system-wide properties• Distributed Reinforcement Learning (DRL) is
extension to RL and uses neighbor interactions only
Model-Based Reinforcement Learning
( , ) ( , ) ' | , . 's
Q s a R s a P s s a V s
S
MDPAdaptationContract
AMM
action (at)rt+1
st+1
reward (rt)
state (st)
Component
1.Action Reward 2. State Transition Model
3. Next State Reward
Markov Decision Process = {States }, {Actions}, R(States,Actions), (States, Actions, States)
Decentralised System Optimisation
Coordinating the solution to a set of Discrete Optimisation Problems (DOPs)
• Components have a Partial System View• Coordination Actions
• Actions ={delegation} U {DOP actions} U {discovery}
• Connection Costs
A
B
Causally-Connected
States
CDelegation
Collaborative Reinforcement Learning
Advertisement• Update Partial Views of Neighbours
Decay• Negative Feedback on State Values in the Absence of
Advertisements
( , ) ( , ) ' | , . ' ' | ,i i i is
Q s a R s a P s s a Decay V s D s s a
S
Action Reward State Transition Model
CachedNeighbour’s V-value
ConnectionCost
Adaptation in CRL System
A feedback process to
• Changes in the optimal policy of any RL agent• Changes in the system environment• The passing time
SAMPLE: Ad-hoc Routing using DRL
Probabilistic ad-hoc routing protocol based on DRL• Adaptation of network traffic around areas of congestion
• Exploitation of stable routes
Routing decisions based on local information and information obtained from neighbors
Outperforms Ad-hoc On Demand Distance Vector Routing (AODV) and Dynamic Source Routing (DSR)
SAMPLE: A CRL System (I)
SAMPLE: A CRL System (II)
Instead of always choosing the neighbor with the best Q value, i.e., taking the delegation action
a= arg maxaQi(B, a),
a neighbor is chosen probabilistically
SAMPLE: A CRL System (III)
Pi(s’|s, aj) = E(CS/CA)
SAMPLE: A CRL System (IV)
Performance
Metric:• Maximize
• throughput
• ratio of delivered packets to undelivered packets
• Minimize• number of transmission required per packet sent
Figures 5-10
Questions/Discussions
Top Related