May 18,2004A Reinforcement Learning Method Based on Adaptive Simlated Annealing1 A Reinforcement...
-
date post
21-Dec-2015 -
Category
Documents
-
view
226 -
download
3
Transcript of May 18,2004A Reinforcement Learning Method Based on Adaptive Simlated Annealing1 A Reinforcement...
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 1
A Reinforcement Learning Method Based on Adaptive Simulated Annealing
Authored by:Amir F. Atiya
Department of Computer EngineeringCairo University, Giza, Egypt
Alexander G. ParlosDept. mechanical Engineering
Texas A&M University, College Station, Texas
Lester IngberLester Ingber Research
Ingber.com
September 13, 2003
Presented by Doug Moody, May 18, 2004
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 2
Glass-Blowing and its Impact on Reinforcement Learning
• Considering the whole piece while focusing on a particular section
• Slow cooling to relieve stress and gain consistency• Use of “annealing”
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 3
Paper Approach
• Review the reinforcement learning problem, and introduce the use of function approximation to determine state values
• Briefly review the use of an adaptation of “annealing” algorithms to find functions that will determine a state’s value
• Use this approach on a straight forward decision-making problem.
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 4
Function ApproximationIntroduction
• Much of our emphasis in reinforcement learning has treated a value function as one entry for each state-action pair
• Finite Markov Decision processes have a fixed number of states and actions
• This approach can, in some problems, introduce limitations when there are many states , insufficient samples across all states or a continuous state space.
• These limitations can be addressed by “generalization”
• Generalization also can be referred to as “function approximation”
• Function approximation has been widely studied in many fields (think regression analysis!)
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 5
Function ApproximationCharacteristics
• A “batch” or “supervised learning” approach versus the on-line approach we have encountered
• Requires a “static” training set from which to learn
• Can not handle dynamically changing target functions, which may have been bootstrapped.
• Hence , function approximation is not suitable for all types of reinforcement learning
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 6
Function ApproximationGoals
• Requires a “static” training set from which to learn
• Can not handle dynamically changing target functions, which may have been bootstrapped.
• Hence , function approximation is not suitable for all types of reinforcement learning
• The value function is dependent upon a parameter vector which could be the vector of connection in a network
• Typically function approximation wants to minimize:
• P(s) are weights of the errors
MSE( t) P(s) V (s) Vt (s) s S 2
MSE: Mean Squared Error : vector of function parameters
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 7
Function ApproximationMethods
• Step by Step Approach : Gradient Descent - move slowly toward optimal “fit”
• Linear Approach: Special case of Gradient where parameters are a column vector
• Coding Methods
– Coarse
– Tile
– Radial Basis Functions
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 8
COARSE CODING
features should relate to the characteristics of the stateFor instance for a robot, the location, remaining power may be usedFor chess, the number of pieces, moves for pawn queen, etc.. Slide from Sutton and Barto textbook
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 9
LEARNING AND COARSE CODING
Slide from Sutton and Barto textbook
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 10
TILE CODING
• Binary feature for each tile
• Number of features present at any one time is constant
• Binary features means weighted sum easy to compute
• Easy to compute indices of the features present
Slide from Sutton and Barto textbook
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 11
RADIAL BASIS FUNCTIONS (GAUSSIAN)
s (i) exp s ci
2
2 i2
reflects degrees which feature is presentLook to variance to show relationship of feature in the state space
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 12
Paper’s Description of the Reinforcement Learning Model
Basic System
Value Definition
Policy Definition
Optimal Policy
Maximal Value
Eq. 4
Eq. 5
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 13
Value Function to Optimize
basis function
weight parameter
GOAL: find the optimal set of that will lead to the most accurate evaluation
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 14
Use Simulated Annealing to find best set of Wk
• Annealing algorithms seek to search the entire state space and slowing “cool” to appropriate local minima
• Algorithms trade off between fast convergence and continuous sampling of the entire
• Used typically to find the optimization of a combinatorial problem
• Requirements:
– Concise definition of the system
– Random generator of moves
– Objective function to be optimized
– Temperature schedule
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 15
Example of Simulated Annealing
• Problem - find the lowest valley in a mountainous region• View the problem as having two directions - North-South and East-West• Use a bouncing ball to explore the terrain at high temperature• The ball can make high bounces exploring many regions• Each point in the terrain has a “cost function” to optimize• As the temperature cools, the ball’s range and exploration decreases as it
focuses on a smaller region of the terrain• Two distributions are used: generating distribution (for each parameter),
acceptance distribution• Acceptance distribution determines whether to stay in the valley or
bounce out.• Both distributions are affected by temperature
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 16
Glass-Blowing Example
• Larger changes are made to the glass piece at higher temperatures• As glass is cooled, the piece is still scanned (albeit more quickly) for stress
points• Can not be “heated” up again and keep previous results
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 17
Adaptive Simulated Annealing (ASA)
• Has some approach as “simulated annealing”• Uses a specific distribution with a wider tail• Does not rely on “quenching” to achieve quick convergence• Has been available as a C programming system• Relies heavily upon a large set of tuning options:
– scaling of temperatures , probabilities– limitation on searching in regions with certain parameters– linear vs. non-linear vector
• Supports re-annealing - time is wound back ( and hence temperature) after some results are achieved to take example of found sensitivities
• Good for non-linear functions
More information and software available at www.ingber.com
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 18
Reinforcement learning with ASA Search
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 19
Sample Implementation
• Problem: Choose the highest number from a sequence of numbers
– Numbers are generated from an unknown source, with a normal distribution having a mean between 0 and 1 and a standard deviation between 0 and .5
– As time passes the reward is discounted
– Hence the tradeoff: more waiting provides more information, but a penalty is incurred
• Paper used 100 sources, with each generating 1000 numbers for a given sequence as the training set.
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 20
Solution Approach
• Define a state space as a combination of the following:
– time t
– the current mean at time t of observed numbers
– the current standard deviation’
– the highest number chosen thus far
• Place 10 Gaussian basis functions throughout the State Space
• Use the algorithm to optimize a vector of weight parameters to the basis functions
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 21
RESULTS
•ASA achieved an overall reward value•Q-Learning found the standard deviation•Improvement is substantial given that picking the first number in each set would yield .5
May 18,2004 A Reinforcement Learning Method Based on Adaptive Simlated Annealing 22
Paper Comments
• Pros
– Looked to use existing reinforcement taxonomies to discuss the problem
– Selected a straight forward problem
• Negative
– Did not fully describe the basis function placement
– Insufficient parameter for Q-Learning used
– Did not show an non-linear example
– Could have provided more information on ASA Options used for results duplication