Machine Learning for Control
of Morphing Air Vehicles
John Valasek, Amanda Lampton, Adam Niksch
Aerospace Engineering DepartmentTexas A&M University
SAE Aerospace Control and Guidance Systems Committee Meeting 99
1 March 2007Boulder, CO
Valasek, Lampton, Niksch - 2
Student Research Team
2006 - 2007
Valasek, Lampton, Niksch - 3
Briefing Agenda
Shape Changing / Morphing for Micro Air Vehicles
Learning
Control
Learning and Control– Adaptive-Reinforcement Learning Control
Some Results
Some Conclusions
Research Issues
Extensions
Valasek, Lampton, Niksch - 4
Biomimetic Example 1
Valasek, Lampton, Niksch - 5
300 million year old design
97% success rate in capturing prey
Flight relies entirely on 3-D unsteady aerodynamics
Unstable in all axes
Can hover as well as fly from a forward speed of +100 body
length/sec (~ 60 mph) to - 3 bl/sec within 5 bl
Can alter heading by 90 degs in less than 0.1 secs
Dragonflies and other flying insects have the fastest visual processing
system in the animal kingdom
Biomimetic Example 2
These feats are accomplished with the neuro-circuitryof a brain smaller than a sesame seed.
Andres Meade, Rice University
Valasek, Lampton, Niksch - 6
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
Cossack
Valasek, Lampton, Niksch - 7
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
Valasek, Lampton, Niksch - 8
Biomimetic Example 3Cossack sysid using OKID
Valasek, Lampton, Niksch - 9
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
Kinematic Engineering Model of Eagle Wing
Valasek, Lampton, Niksch - 10
Reconfigurable; shape changing (morphing); rigid or elastic body plants Multi-Input Multi-Output (MIMO)systems Distributed sensing, actuation, and propulsion
– Multiple, possibly non co-located sensors– Can have large number of actuators control allocation problem– High bandwidth actuators gust alleviation
Robust Adaptive Control– Flow control– Exogenous inputs (turbulence and gusts)– Damage and fault tolerance
Distributed and limited processing capability– Volumetric limit– Mass limit– Power limit– Real-time operation
Integrated guidance, navigation, and control– Self learning, self adapting, self tuning
Biomimetic Flight Summary
Information processing system
Valasek, Lampton, Niksch - 11
A Multi-Role Platform that:
Changes its state substantially to adapt to changing mission environments.
Provides superior system capability not possible without reconfiguration.
Uses a design that integrates innovative combinations of advanced materials, actuators, flow controllers, and mechanisms to achieve the state change.
Morphing Aircraft
Aerospace America, Feb 2004
DARPA’s definition
Changes On The Order Of 50% More Wing Area Or Wing Span And Chord
Valasek, Lampton, Niksch - 12
Morphing for Mission Adaptation
– Large scale, relatively slow, in-flight shape change to enable a single vehicle to perform multiple diverse mission profiles
or:
Morphing for Control
– In-flight physical or virtual shape change to achieve multiple control objectives (maneuvering, flutter suppression, load alleviation, active separation control)
MissionAdaptation
Control
John Davidson, NASA Langley, AFRL Morphing Controls Workshop – Feb 2004
Which Morphing?
Valasek, Lampton, Niksch - 13
Lockheed Martin
NextGen&
Barron Associates
Large Morphing Air Vehicle Models
Valasek, Lampton, Niksch - 14
Small Morphing Air Vehicle Models
Cornell (collaborator) Garcia & Lipson
– Morphing dynamical model and simulation that incorporates aerodynamic and structural effects
– Validated with experimental data– Basic morphing parameters (incidence angle, dihedral angle)
Maryland Hubbard
– Actuating a flapping wing structure with SMA’s– Structure, material, distribution of actuators
Florida Lind
– Morphing flight demonstrator vehicle– Basic morphing parameters (dihedral angle, sweep angle)– H-inf control
lots of modeling, very little control
Valasek, Lampton, Niksch - 15
• Use biologically inspired approach to understand and control the: • morphing (shape changing),
• flight control (stability, flight path, gust tolerance, stall)
• maneuvering (perching and hovering)
of multi-mission micro air vehicles.
• Control the physics, in concert with nonlinearities.• Not as a robustness bandaid for aerodynamic and structural
uncertainties & lack of understanding
• Shape change the entire vehicle, not a component on the vehicle• DARPA definition
Technical Approach and Scope
Valasek, Lampton, Niksch - 16
• Address:
•WHEN to morph, perch, etc.
•HOW to morph, perch, etc.
•LEARNING to morph, perch, etc.
All while keeping the (pointy?) end facing forward.
Big Picture Research Goals
Valasek, Lampton, Niksch - 17
Valasek, Lampton, Niksch - 18
Common Mathematical Domain Helps
To Avoid Ad-hoc Approaches
??
Machine Learning
Reconfiguration Policy
State based methods
learning
Adaptive Control
Parameters in a Known
Functional Relationship
State based methods
adapting
The Mathematical Domain Issue
Valasek, Lampton, Niksch - 19
Adaptive-ReinforcementLearning Control (A-RLC)
Conceptual Control Architecture
for Reconfigurable or Morphing Aircraft
SAMIStructured Adaptive Model Inversion
(Traditional Control)
Flight controller to handle wide variation in dynamic properties
due to shape change
ML Machine Learning(Intelligent Control)
Learn the morphing dynamics and the optimal shape at every flight
condition in real-time
Valasek, Lampton, Niksch - 20
Synthetic Jets for Virtual Shaping and Separation Control
MultiSensor MEMS Arrays for Flow Control Feedback
Adaptive Controller
Sensed Information Aggregation
Control Information Distribution
Reconfiguration Command
Generation
System Performance Evaluation
Knowledge
Base
Environment
Control Architecture
Valasek, Lampton, Niksch - 21
2-D Plate Rectangular Block Ellipsoid Delta Wing Final 2004 2005 2006 2007 Objective
Morphing Air Vehicle Evolution
Valasek, Lampton, Niksch - 22
Morphing Air Vehicle Model - TiiMY
Shape
Ellipsoidal shape with varying axis dimensions.
Constant volume (V) during morphing
2 independent variables: y and z, dependent dimension
Morphing Dynamics
Smart material: shape memory alloy (SMA)
Morphing Dynamics : Simple Nonlinear Differential Equations
6Vx
yz
y-dimension 2
z-dimension 3y
z
y yy V
z zz V
Valasek, Lampton, Niksch - 23
Morphing Dynamics
Y-morphing Z-morphing
Valasek, Lampton, Niksch - 24
Shape Morphing Simulation TiiMY
Valasek, Lampton, Niksch - 25
Optimal Shapes atVarious Flight Conditions
Optimality is defined by identifying a cost function.
J=J (Current shape, Flight condition)
2 2
0.5
( ( )) ( ( ))
3 cos( ) and 2 22
y z y z
Fy z
J J J y S F z S F
S F S e
Valasek, Lampton, Niksch - 26
6-DOF Mathematical Modelfor Dynamic Behavior
Variables
Nonlinear 6–DOF Equations
– Kinematic level:
– Acceleration level:
Drag Force
– Function of air density, square of velocity along axis, and projected area of the vehicle perpendicular to the axis
[ ] [ ]
[ ] [ ]
T Tc x y z c
T T
p d d d v u v w
p q r
;
c l c a
c c d
d
p J v J
mv mv F F
I I I M M
�
�
additional dynamics due to morphing
Valasek, Lampton, Niksch - 27
Learning
Valasek, Lampton, Niksch - 28
– Machine Learning
• Learn the optimal control mechanism by wind-tunnel experiments & flight tests
• Possible learning algorithms include Artificial Neural Networks (ANN) , Explanation-Based Learning (EBL), and Reinforcement Learning (RL)
Candidates to develop inference mechanism– Rules-Based Expert System
• Model the knowledge of human experts
• Imitate the natural behaviour of birds
Question: How Many Control Theorists Does It Take To Change A Using
Artificial Neural Networks ?
Knowledge Based Control
– Biologically inspired control process• Mimic the behaviour of birds
Question: How Many Control Theorists Does It Take To Change A Using
Reinforcement Learning ?
Valasek, Lampton, Niksch - 29
Simple Example
Valasek, Lampton, Niksch - 30
Reinforcement Learning - 1
Supervised or unsupervised learning? Sequential decision making.– Knowledge is based on experience and interaction with the environment, not
on input-output data supplied by an external supervisor
Achieves a specific goal by learning from interactions with the environment.– Considers state information (s)– Performs sequences of actions, (a), observing the consequences– Attempts to maximize rewards (r) over time
• These specify what is to be achieved, not how to achieve it
– Constructs a state value function (V)• Learns an optimal control policy
Memory is contained in the state value function
* *arg max[ ( , ) ( ( , ))]a
r s a V s a
21 2 3
0
... it t t t t i
i
R r r r r
*( ) max ( )V s V s for all s
Valasek, Lampton, Niksch - 31
Reinforcement Learning - 2
Learning is done repetitively, by subjecting to different scenarios
Learning is cumulative and lifelong
Formulations are generally based on Finite Markov Decision Processes (MDP)
• 3 major candidate algorithms: – Dynamic programming
– Monte Carlo methods
– Temporal Difference Learning
Valasek, Lampton, Niksch - 32
1. Actor takes action based upon states and preference function2. Critic updates state value function, and evaluates action3. Actor updates preference function
Learning is done repetitively, by subjecting to different scenarios
Reinforcement Learningself training
Valasek, Lampton, Niksch - 34
Two Illustrative Examples
Valasek, Lampton, Niksch - 35
State: – Gain for δa = Kφ (φcmd - φ)
Actions: – Increase gain by small amount– Decrease gain by small amount
Constraints/Boundaries– Max overshoot– Rise time– Settling time
Interesting Features– e-greedy policy incorporated– Upward annealing of γ
incorporated
max os = 2%Tr = 8sTs = 10s
Cessna 208B Super Cargomaster
3 2
4 3 2
3 15 9.0 6.0 19.7
4.8 5.2 11.1 0.01a
e S S SS
S S S S
Matlab: ~250 sec real-time for 1000 learning episodes
familiar example
Reinforcement Learning
Valasek, Lampton, Niksch - 36
Reinforcement Learningfamiliar example
Valasek, Lampton, Niksch - 37
Reinforcement Learningfamiliar example
Valasek, Lampton, Niksch - 38
0
20
40
60
80
100
120
-4-2024
-505
x
z
y
Start
Finish
Smart Block Demo 1
aerial obstacle course
Valasek, Lampton, Niksch - 40
Smart Block: First Try
Valasek, Lampton, Niksch - 41
Smart Block: Second Try
Valasek, Lampton, Niksch - 42
Smart Block: New Course
Valasek, Lampton, Niksch - 51
Adaptive–Reinforcement Learning Control
Valasek, John, Tandale, Monish D., and Rong, Jie, "A Reinforcement Learning Adaptive Control Architecture for Morphing,” Journal of Aerospace Computing, Information, and Communication, Volume 2, Number 4, pp. 174-195, April 2005.
Valasek, John, Doebbler, James, Tandale, Monish D., and Meade, Andrew J., "Improved Adaptive-Reinforcement Learning Control for Morphing Unmanned Air Vehicles,”Journal of Aerospace Computing, Information, and Communication (in review).
Tandale, Monish D., Rong, Jie, and Valasek, John, "Preliminary Results of Adaptive-Reinforcement Learning Control for Morphing Aircraft,” AIAA-2004-5358, Proceedings of the AIAA Guidance, Navigation, and Control Conference, Providence, RI, 16-19 August 2004.
Valasek, Lampton, Niksch - 52
A-RLC Architecture
Valasek, Lampton, Niksch - 53
Air Vehicle Example
Valasek, Lampton, Niksch - 54
Objective– Demonstrate optimal
shape morphing for multiplespecified flight conditions
Method– For every flight condition, learn
optimal policy that commands
voltage producing the optimal shape– Minimize total cost over the entire flight trajectory– Evaluate the learning performance after 200 learning episodes
Air Vehicle Example
RL Module is Completely Ignorant of Optimality Relations and Morphing Control Functions:
It Must Learn On Its Own, From Scratch
Valasek, Lampton, Niksch - 55
Agent: Morphing Air Vehicle Reinforcement Learning Module
Environment: Various flight conditions
Goal: Fly in optimal shape that minimizes cost
States: Flight condition; shape of vehicle
Actions: Discrete voltages applied to change shape of vehicle– Action set:
Rewards: Determined by cost functions
Optimal control policy: Mapping of the state to the voltage leading to the optimal shape
Timmy Demo
reinforcement learning definitions
( , ) [2,4] [0,5]
( , ) [2,4] [0,5]
y FS
z F
( , ) [0, 0.5, 1, 5] ; [0, 0.5, 1, 4]y zA V V
Valasek, Lampton, Niksch - 56
Episodic Learning
Unsupervised learning episode
Single pass through 100 meter flight path in 200 seconds
Reference trajectory is generated arbitrarily
The flight condition changes twice during each episode
Shape change iteration after every 1 second
Exploration-exploitation dilemma:– Explorative early, exploitative later -policy with decreasing
Limited training examples– Only 6 discrete flight conditions:
2000 samples for KNNPI) rand(aa
asQ a
i
a
else
),(maxarg
) -1 ty (probabili if policy greedy-ε
Valasek, Lampton, Niksch - 57
Demo
Valasek, Lampton, Niksch - 58
Comparison of True Optimal Shape and Learned Shape
KNN learns poorly for
several flight conditions
Valasek, Lampton, Niksch - 59
Function Approximation– Errors remained which could not be eliminated with additional training.
– Use Galerkin-based Sequential Function Approximation (SFA) to approximate the action-value function Q(s,a)
What Happened?
Valasek, Lampton, Niksch - 60
New SFA approach learns optimal shape well
Comparison of True Optimal Shape and Learned Shape
Valasek, Lampton, Niksch - 61
Comparison
Normalized RMS error
Y dimension Z dimension
K N N 1.42 0.821
S F A 1.27 0.661
10% reduction 20% reduction
Valasek, Lampton, Niksch - 68
Reinforcement Learning successfully learns the optimal control policy that results in the optimal shape at every flight condition. – Can function in real-time, leading to better performance as system
operates over the long term.
Adaptive-Reinforcement Learning Control is a promising candidate for control of Mission Morphing.– Maintains asymptotic tracking in the presence of parametric
uncertainties and initial condition errors.
Shape Changes for “Mission Morphing” can be treated as piecewise constant parameter changes– SAMI is a favorable method for trajectory tracking control
“Morphing for Control” will require different control strategy– Piecewise constant approximation no longer valid
What Does This Show?
Valasek, Lampton, Niksch - 69
Issues & Future Directions
Realistic structural response effects – Aeroelastic behaviour– SMA models and hysteretic behaviour
• Priesach model is algebraic, only has major hysterisis loops• Solution: roll your own with R-L
SMA Hysteresis
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0 20 40 60 80 100
Temperature (C)
Str
ain
Experimental Major Math Model Experimental Minor
Valasek, Lampton, Niksch - 70
Issues & Future Directions
Time scale problem: control methodologies to handle faster shape changes– Hovakimyan’s Adaptive Control– Linear Parameter Varying (LPV) control
Novel distributed sensing and distributed actuation on a large(!) scale– Insect and avian inspired sensing
Learning on a continuous domain– Continuous versus discrete
Modify the simulation to include a more advanced aircraft model– Wing-Body, Wing-Body-Empennage, etc.
Build and fly R/C class morphing demonstrator UAV
Valasek, Lampton, Niksch - 71
Issues & Future Directions
Incorporate aerodynamic and structural effects due to large shape changing
Cost Function– Potential components
• Specified CL
• Minimum drag• Minimum peak stress
Degrees of Freedom– Thickness
• 6% to 24%– Camber
• 0% to 10%– Max camber location
• 0.2c to 0.8c– Chord
• 1 unit to ? units– Angle-of-attack
• -5° to 10 °– Within linear range
R-L For Morphing Airfoils & Wings
Valasek, Lampton, Niksch - 72
Morphing Airfoil Demonstration
Valasek, Lampton, Niksch - 73
Questions?
John Valasek
Aerospace Engineering Department
Texas A&M University
3141 TAMU
College Station, TX 77843-3141
(979) 845-1685
FSL Web Page
– http://flutie.tamu.edu/~fsl
Top Related