Distributed Control in Multi-agent Systems: Design and Analysis
description
Transcript of Distributed Control in Multi-agent Systems: Design and Analysis
Distributed Control in Multi-agent Distributed Control in Multi-agent Systems: Design and AnalysisSystems: Design and Analysis
Kristina LermanAram Galstyan
Information Sciences InstituteUniversity of Southern California
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Design of Multi-Agent SystemsDesign of Multi-Agent Systems
Multi-agent systems must function inDynamic environmentsUnreliable communication channelsLarge systems
SolutionSimple agents
No reasoning, planning, negotiation
Distributed controlNo central authority
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Advantages of Distributed Advantages of Distributed ControlControl
• Robust• tolerant of agent error and failure
• Reliable• good performance in dynamic environments
with unreliable communication channels• Scalable
• performance does not depend on the number of agents or task size
• Analyzable• amenable to quantitative analysis
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Analysis of Multi-Agent SystemsAnalysis of Multi-Agent Systems
Tools to study behavior of multi-agent systems• Experiments
• Costly, time consuming to set up and run• Grounded simulations: e.g., sensor-based simulations of
robots• Time consuming for large systems
• Numerical approaches• Microscopic models, numeric simulations
• Analytical approaches • Macroscopic mathematical models• Predict dynamics and long term behavior • Get insight into system design
• Parameters to optimize system performance• Prevent instability, etc.
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
DC: Two Approaches and DC: Two Approaches and AnalysesAnalyses
• Biologically-inspired approach• Local interactions among many simple agents
leads to desirable collective behavior• Mathematical models describe collective
dynamics of the system• Markov-based systems
• Application: collaboration, foraging in robots• Market-based approach
• Adaptation via iterative games• Numeric simulations • Application: dynamic resource allocation
Biologically-Inspired ControlBiologically-Inspired Control
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Analysis of Collective BehaviorAnalysis of Collective Behavior
Bio control modeled on social insects• complex collective behavior arises in simple,
locally interacting agents
Individual agent behavior is unpredictable• external forces – may not be anticipated• noise – fluctuations and random events • other agents – with complex trajectories• probabilistic controllers – e.g. avoidance
Collective behavior described probabilistically
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Some Terms DefinedSome Terms Defined
• State - labels a set of agent behaviors• e.g., for robots Search State = {Wander,
Detect Objects, Avoid Obstacles}• finite number of states• each agent is in exactly one of the states
• Probability distribution• = probability system is in
configuration n at time t • where Ni is number of agents in
the i’ th of L states
),( tnP
),,( 1 LNNn
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Markov SystemsMarkov Systems
• Markov property: configuration at time t+t depends only on configuration at time t
• also, • change in probability density:
.),(),|,(),(
n
tnPtnttnPttnP
n
n
tnPtnttnP
tnPtnttnPtnPttnP
),(),|,(
),(),|,(),(),(
n
tnttnP 1),|,(
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Stochastic Master EquationStochastic Master Equation
In the continuum limit,
with transition rates
nn
tnPtnnWtnPtnnWdt
tndP),();|(),();|(
),(
ttnttnP
tnnWt
),|,(lim);|(
0
0t
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Rate EquationRate Equation
Derive the Rate Equation from the Master Eqn
• describes how the average number of agents in state k changes in time
• Macroscopic dynamical model
k
kk
kk tNtkkWtNtkkW
dt
tNd)();|()();|(
)(
Collaboration in RobotsCollaboration in Robots
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Stick-Pulling Experiments Stick-Pulling Experiments (Ijspeert, Martinoli & Billard, 2001)(Ijspeert, Martinoli & Billard, 2001)
• Collaboration in a group of reactive robots• Task completed only through collaboration• Experiments with 2 – 6 Khepera robots• Minimalist robot controller
A. Ijspeert et al.
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Experimental ResultsExperimental Results
Key observations• Different
dynamics for different ratio of robots to sticks
• Optimal gripping time parameter
Flowchart of robot’s controller
start look for sticks
object detected?
obstacle?
gripped?
grip & wait
time out?
teammatehelp?
release
obstacleavoidance
success
Ijspeert et al.
Y
N
Y
N
N
NN
Y
Y
Y
State diagram for amulti-robot system
search
grip
s u
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Model VariablesModel Variables
• Macroscopic dynamic variables Ns(t) = number of robots in search state at time t
Ng(t) = number of robots gripping state at time t
M(t) = number of uncollected sticks at time t• Parameters
• connect the model to the real system
= rate of encountering a stick
RG = rate of encountering a gripping robot
= gripping time
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Mathematical Model of Mathematical Model of CollaborationCollaboration
);()()()(
)()()()()()(
ttNtMtN
tNtNRtNtMtNdt
tdN
gs
gsGgss
Initial conditions: 00 )0(,0)0(,)0( MMNNN gs
find & grip stickssuccessful collaboration
unsuccessful collaboration0NNN gs
consttM )( for static environment
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Dimensional AnalysisDimensional Analysis
• Rewrite equations in dimensionless form by making the following transformations:
• only the parameters and appear in the eqns and determine the behavior of solutions
• Collaboration rate• rate at which robots pull sticks out
)()(1)(~),,( 00 NfNtntntR
G
s
RMN
MtMtNtNtn
~
,/
,,/)()(
00
000
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Searching Robots vs TimeSearching Robots vs Time
=5
=0.5
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Collaboration Rate vs Collaboration Rate vs
Key observations• critical • optimal gripping
time parameter
=1.0
=1.5
=0.5
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Comparison to Experimental Comparison to Experimental ResultsResults
=1.0
=1.5
=0.5
Ijspeert et al.
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Summary of ResultsSummary of Results
• Analyzed the system mathematically• importance of • analytic expression for c and opt
• superlinear performance • Agreement with experimental data and
simulations
Foraging in RobotsForaging in Robots
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Robot ForagingRobot Foraging
• Collect objects scattered in the arena and assemble them at a “home” location
• Single vs group of robots• no collaboration• benefits of a group
• robust to individual failure• group can speed up collection
• But, increased interference
Goldberg & Matarić
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Interference & Collision Interference & Collision AvoidanceAvoidance
• Collision avoidance
• Interference effects• robot working alone is more efficient• larger groups experience more interference• optimal group size: beyond some group size,
interference outweighs the benefits of the group’s increased robustness and parallelism
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
State DiagramState Diagram
start look for pucks
object detected?
obstacle?avoid
obstacle
grab puck
go home
searching homing
avoidingavoiding
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Model VariablesModel Variables
• Macroscopic dynamic variables Ns(t) = number of robots in search state at time t
Nh(t) = number of robots in homing state at time t
Nsav(t), Nh
av(t) = number of avoiding robots at time t
M(t) = number of undelivered pucks at time t
• Parametersr = rate of encountering a robot
p = rate of encountering a puck
= avoiding time
h0 = homing time in the absence of interference
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Mathematical Model of ForagingMathematical Model of Foraging
savh
hsSr
havhsp
s NNNNNNNMNdt
dN
11][ 0
hh
Ndt
dM1
Initial conditions: 00 )0(,)0( MMNNs
Average homing time:
00 1 Nrhh
hh
havhhr
havhsp
h NNNNNNNMNdt
dN
11
][][ 0
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Searching Robots and Pucks vs Searching Robots and Pucks vs TimeTime
robots
pucks
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Group Efficiency vs Group SizeGroup Efficiency vs Group Size
=1
=5
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Sensor-Based SimulationsSensor-Based Simulations
Player/Stage simulatornumber of robots = 1 - 10number of pucks = 20arena radius = 3 mhome radius = 0.75 mrobot radius = 0.2 m robot speed = 30 cm/s puck radius = 0.05 m rev. hom. time = 10 s
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Simulations ResultsSimulations Results
0
200
400
600
800
1000
1200
1400
1600
0 2 4 6 8 10
number of robots
tim
e (s
)avoid time = 3s
avoid time = 1s
model (3 s)
model (1 s)
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Simulations ResultsSimulations Results
0
0.002
0.004
0.006
0 2 4 6 8 10
number of robots
eff
icie
nc
yt=3 s
model 3 s
t=1 s
model 1 s
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
SummarySummary
• Biologically inspired mechanisms are feasible for distributed control in multi-agent systems
• Methodology for creating mathematical models of collective behavior of MAS
• Rate equations • Model and analysis of robotic systems
• Collaboration, foraging• Future directions
• Generalized Markov systems – integrating learning, memory, decision making
Market-Based ControlMarket-Based Control
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Distributed Resource AllocationDistributed Resource Allocation
• N agents use a set of M common resources with limited, time dependent capacity LM(t)
• At each time step the agents decide whether to use the resource m or not
• Objective is to minimize the waste
where Am(t) is the number of agents utilizing resource m
t m
mm tLtAw 2))()((
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Minority GamesMinority Games
• N agents repeatedly choose between two alternatives (labeled 0 and 1), and those in the minority group are rewarded
• Each agent has a set of S strategies that prescribe a certain action given the last m outcomes of the game (memory) 00
0001
010
011
100
101
110
111
0 1 1 0 0 1 0 1
strategy with m=3
• Reinforce strategies that predicted the winning group
• Play the strategy that has predicted the winning side most often
inputaction
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
MG as a Complex SystemMG as a Complex System
• Let be the size of the group that chooses ”1” at time t• The “waste” of the resource is measured by the standard
deviation - average over
time• In the default Random Choice Game (agents take either action
with probability ½) , the standard deviation is
2/N
)(tA
,)()( 22 tAtA
0
0.25
0.5
0.75
1
1.25
1.5
2 4 6 8 10 12 14
memory length
sta
nd
ard
de
via
tio
nFor some memory size the waste is smaller than in the random choice game
Coordinated phase
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Variations of MGVariations of MG
• MG with local information Instead of global history agents may use local interactions (e.g., cellular automata)
• MG with arbitrary capacitiesMG with arbitrary capacities The winning choice is “1” if where is the capacity, is the number of agents that chose “1”
LtA )( )(tAL
To what degree agents (and the system as a whole) can coordinate in externally changing environment?
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
MG on Kauffman NetworksMG on Kauffman Networks
Set of N Boolean agents: Nisi ..1},1,0{
Kjk j ..1},{
)()(),()( 01
tLLtLtstAN
ii
Each agent hasA set of K neighbors A set of S randomly chosen Boolean functions of K variables
))(),...(()1(1
tstsFtsK
MAX
kkj
ii
SjF ji ..1,
Dynamics is given by
The winning choice is “1” if where)()( tLtA
Global measure for optimality:
T
ttLtA
T 1
22 )]()([1
For the RChG (each agent chooses “1” with probability )
NtLt /)(
T
ttdtT
N0
2 ]1[1
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Simulation ResultsSimulation Results
K=2 networks show a tendency towards self-organization into a coordinated phase characterized by small fluctuations and effective resource utilization
K=2 Traditional MG m=6
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Results (continued)Results (continued)
Coordination occurs even in the presence of vastly different time scales in the environmental dynamics
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
ScalabilityScalability
For K=2 the “variance” per agent is almost independent on the group size,
constN
In the absence of coordination N
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Phase Transitions in Kauffman Phase Transitions in Kauffman NetsNets
Kauffman Nets: phase transition at K=2 separating ordered (K<2) and chaotic (K>2) phases
For K>2 one can arrive at the phase transition by tuning the homogeneity parameter P (the fraction of 0’s or 1’s in the output of the Boolean functions)
K=3
78.0cP
The coordinated phase might be related to the phase transition in Kauffman Nets.
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
Summary of ResultsSummary of Results
• Generalized Minority Games on K=2 Kauffman Nets are highly adaptive and can serve as a mechanism for distributed resource allocation
• In the coordinated phase the system is highly scalable
• The adaptation occurs even in the presence of different time scales, and without the agents explicitly coordinating or knowing the resource capacity
• For K>2 similar coordination emerges in the vicinity of the ordered/chaotic phase transitions in the corresponding Kauffman Nets
01/23/2002
ISI
USC Information Sciences Institute K. LermanDistributed Control in MAS
ConclusionConclusion
• Biologically-inspired and market-based mechanisms are feasible models for distributed control in multi-agent systems
• Collaboration and foraging in robots
• Resource allocation in a dynamic environment
• Studied both mechanisms quantitatively
• Analytical model of collective dynamics
• Numeric simulations of adaptive behavior