Necessary and Sufficient Conditions for ParetoOptimality in Infinite Horizon Cooperative

Differential Games

Puduru Viswanadha Reddy Jacob Engwerda

Department of Econometrics and Operations Research

Tilburg University, P.O.Box: 90153, 5000LE Tilburg, The Netherlands

{P.V.Reddy, J.C.Engwerda}@uvt.nl

May, 2010

AbstractIn this article we derive necessary and sufficient conditions for the existence of Pareto

optimal solutions for an N player cooperative infinite horizon differential game. Firstly, wewrite the problem of finding Pareto candidates as solving N constrained optimal controlsubproblems. We derive some weak conditions which entail one to find all Pareto candi-dates by solving a weighted sum optimal control problem. We observe that these conditionsare related to transversality conditions of the associated subproblems. Furthermore, we de-rive sufficient conditions under which candidate Pareto solutions are indeed obtained bysolving a weighted sum optimal control problem. We consider games defined by nonau-tonomous and discounted autonomous systems. The obtained results are used to analyzethe regular indefinite linear quadratic infinite horizon differential game. For the scalar casewe devise an algorithm to find all the Pareto optimal solutions.

Keywords Pareto Efficiency, Cooperative Differential Games, Infinite HorizonOptimal Control, LQ theory.

JEL-Codes C61, C71, C73.

1 IntroductionCooperative differential games involve situations where several players decide to cooperate whiletrying to realize their individual objectives, with players acting in a dynamic environment. We con-sider cooperative games with no side payments and open loop information structure. The dynamicenvironment is modelled as:

x(t) = f (t,x(t),u1(t),u2(t), · · · ,uN(t)), x(t0) = x0 ∈ Rn, (1)

where ui(t) is the control strategy/action of player i, (u1,u2, · · · ,uN) ∈ U , with U being the set ofadmissible controls. Each player tries to minimize the objective function

Ji(t0,x0,u1,u2 · · · ,uN) =∫ ∞

t0gi(t,x(t),u1(t),u2(t), · · · ,uN(t))dt, (2)

1

for i = 1,2, · · · ,N. For the above problem to be well-defined we assume that f : Rn ×U → Rn

and gi : Rn ×U → R, i = 1,2, · · ·N, are continuous and all the partial derivatives of f and gi w.r.tx and ui are continuous. Further, we assume that the integrals involved in the player’s objectivesconverge. If the integrals do not converge there exist many notions of optimality [18, 9] and theanalysis becomes complicated.

Pareto optimality plays a central role in analyzing these problems. Since we are interested in thejoint minimization of the objectives of the players at an optimal point, the cost incurred by a singleplayer cannot be minimized without increasing the cost incurred by other players. So, we consider(Pareto optimal) solutions which cannot be improved upon by all the players simultaneously. Wecall the controls (u∗1,u

∗2, · · · ,u∗N) ∈ U Pareto optimal if the set of inequalities Ji(u1,u2, · · · ,uN) ≤

Ji(u∗1,u∗2, · · · ,u∗N), i = 1,2, · · · ,N, (with at least one of the inequalities being strict) does not allow

for any solution in (u1,u2, · · · ,uN) ∈ U .A well known way to find Pareto optimal controls is to solve a parameterized optimal control

problem [23, 14]. However, it is unclear whether all Pareto optimal solutions are obtained using thisprocedure. The closest references we could track are [4, 3] and [21]. [21] and the affiliated papers[20] and [15] discuss necessary conditions for Pareto solutions for dynamic systems where costfunctions are just functions of the terminal state. In [21] geometric properties of Pareto surfaceswere used to derive necessary conditions which are in the spirit of maximum principle. Somedifference with our work are: they assume that the admissible controls are feedback type and theterminal state should belong to some n−1 dimensional surface. Recently, [7] gives necessary andsufficient conditions for Pareto optimality for finite horizon cooperative differential games.

We notice that almost all of the earlier works address the problem of finding Pareto solutions inthe finite horizon case. In this article we focus on infinite horizon cooperative differential games.We follow a similar approach as mentioned in [7]. In section 2 we present a necessary and sufficientcharacterization of Pareto optimality. In section 3 the problem of finding all Pareto candidates ofan N player cooperative differential game is transformed as solving N constrained optimal controlsubproblems. Further, we show that by making an assumption on the transversality conditions(Lagrange multipliers), associated with the subproblems, one can find all the Pareto candidates bysolving the weighted sum optimal control problem. The transversality conditions for the optimalcontrol problems in the finite horizon, in general, do not extend naturally to the infinite horizon case.This behavior was first analyzed by [10] using a counterexample. For optimal control problems,[18, 2, 16, 1, 22] give conditions under which the finite horizon transversality conditions extendnaturally to the infinite horizon case. We give some weak conditions under which these conditionsextend naturally within this setting too. These conditions are related to growth conditions whichnaturally extend the finite horizon transversality conditions to the infinite horizons. We consider thegames defined by nonautonomous and discounted autonomous systems. For discounted autonomoussystems, [16] derives necessary conditions for free endpoint optimal control problems. We extendthese results for the constrained subproblems and derive weak conditions for the above assumptionto hold true. In section 4 we derive sufficient conditions for Pareto optimality in the similar lines ofArrow’s sufficiency results in optimal control.

In section 5 we consider regular indefinite infinite planning horizon linear quadratic differentialgames. That is the linear quadratic case, where the cost involved for the state variable has anarbitrary sign and the use of every control is quadratically penalized. This problem was recentlysolved for both a finite and infinite planning horizon in [6] assuming that the problem is a convexfunction of the control variables and the initial state is arbitrary. In this article we concentrate onthe general case and where the initial state is fixed. We provide an algorithm to compute all Paretooptimal solutions when the system is scalar. In section 6 we give conclusions.

This paper addresses just the issue of finding Pareto optimal candidates that can be realized by

2

a grand coalition. We do not discuss cooperative situations that can arise while sharing the jointPareto payoff. For a discussion of some frequently used cooperative agreements, related to bargain-ing, one may consult e.g., chapter 6 of [5].

Notation: We use the following notation. Let SN = {1,2, · · · ,N} denote the grand coalition andlet Si

N = SN/{i} denote the coalition of all players excluding player i. Let PN denote the N di-mensional unit simplex. RN

+ denotes a cone consisting of N dimensional vectors with nonnegativeentries. 1N denotes a vector in RN with all its entries equal to 1. y′ represents the transpose of thevector y ∈ RN . |x| represents the absolute value of x ∈ R. ||y|| represents the Euclidian norm of thevector y ∈ RN . |y|i represents the absolute value of the ith entry of the vector y. ||u||κ representsnorm of u(t) ∈ κ, where κ is a function space. |A|(m,n) represents the absolute value of entry (m,n)of the matrix A. A > 0 denotes matrix A is strictly positive definite. fx(.) represents the partialderivative of the function f (.) w.r.t x. −→ω ∈ RN

+ denotes the vector whose entries wi are the weightsassigned to the cost function of each player. We define the weighted sum function G(.) as follows:

G(−→ω , t,x(t),u(t)) = ∑i∈SN

ωigi(t,x(t),u(t)).

2 Pareto OptimalityIn this section we state conditions to characterize Pareto optimal controls. We mention below a wellknown lemma from [19].

Lemma 2.1. Let αi ∈ (0,1), with ∑Ni=0 αi = 1. Assume u∗ ∈ U is such that

u∗ ∈ arg minu∈U

{N

∑i=1

αiJi(u)

}. (3)

Then u∗ is Pareto optimal.

The above lemma which involves minimizing the weighted sum is an easy way to find Paretooptimal controls. It is, however, unclear whether we obtain all Pareto optimal controls in this way.The above approach may not yield any Pareto solutions even though there exist an infinite numberof Pareto solutions. The following example (an infinite horizon counterpart of (see example 2.2,[7])) illustrates this point.

Example 1. Consider the following game problem:

(P) minJ1(x0,u1,u2) =∫ ∞

0e−ρt/2(u1(t)−u2(t))dt and

minJ2(x0,u1,u2) =∫ ∞

0e−3ρt/2x2(t)(u2(t)−u1(t))dt

sub. to x(t) =ρ2

x(t)+u1(t)−u2(t), x(0) = 0, t ∈ [0,∞)

ui(.) ∈ U , i = 1,2. (4)

By construction, |x(t)| ≤ ceρt/2 for some constant c≥ 0. Taking the transformations x(t)= e−ρt/2x(t)

3

and ui(t) = e−ρt/2ui(t), we transform the game (P) as a new game (P) given by:

(P) minJ1(x0, u1, u2) =∫ ∞

0(u1(t)− u2(t))dt and

minJ2(x0, u1, u2) =∫ ∞

0x2(t)(u2(t)− u1(t))dt

sub. to ˙x(t) = u1(t)− u2(t), x(0) = 0, t ∈ [0,∞) (5)ui(t) = e−ρt/2ui(t), ui(t) ∈ U , i = 1,2.

By construction |x(t)| ≤ deρt/2 for some constant 0 ≤ d < ∞, so limt→∞ |x(t)| exists. Furthermore,we have Ji(x0,u1,u2) = Ji(x0, u1, u2), i = 1,2. The player’s objectives can be simplified as:

J1(x0, u1, u2) =∫ ∞

0(u1(t)− u2(t))dt = lim

t→∞x(t)

J2(x0, u1, u2) =∫ ∞

0x2(t)(u2(t)− u1(t))dt =−1

3

(limt→∞

x(t))3

.

We notice that J2 = −13J3

1 for all (u1,u2) and choosing different values for the control functionsui(.), every point in the (J1,J2) plane satisfying J2 = −1

3J31 can be attained. Moreover, every point

on this curve is Pareto optimal. Now consider the minimization of Jα = (α)J1 +(1−α)J2 subjectto (4) with α ∈ (0,1). If we choose u1 = 0 and u2 = c, straightforward calculations result in Jα =

−2cαρ + 8(1−α)c3

3ρ3 . By choosing c arbitrarily negative Jα can be made arbitrarily small, i.e., J(α) doesnot have a minimum.

Lemma 2.2 mentioned below gives a both necessary and sufficient characterization of Paretosolutions. It states that every player’s Pareto optimal solutions can be obtained as the solution of aconstrained optimization problem. We omit the proof as it is along the lines of the finite dimensionalcase considered in chapter 22 of [19].

Lemma 2.2. u∗ ∈ U is Pareto optimal if and only if for all i, u∗(.) minimizes Ji(u) on the con-strained set

Ui ={

u|J j(u)≤ J j(u∗), j = 1, · · · ,N, j = i}, for i = 1, · · · ,N. (6)

We observe that for a fixed player the constrained set Ui defined in (6) depends on the entries ofthe Pareto optimal solution that represents the loss of the other players. Therefore this result mainlyserves theoretical purposes, as we will see in the proof of theorem 3.1 and theorem 3.3. Usingthe above lemma, we next argue that Pareto optimal controls satisfy the dynamic programmingprinciple.

Corollary 2.1. If u∗(t) ∈ U , t ∈ [0,∞) is a Pareto optimal control for x(0) = x0 in (1,2), thenfor any τ > 0, u∗ ([τ,∞)) is a Pareto optimal control for x(τ) = x∗(τ) in (1,2), where x∗(τ) =x(t,0,u∗([0,τ])) is the value of the state at τ generated by u∗([0,τ]).

Proof. Let Ui(τ), with x(τ) = x∗(τ), be the constrained set defined as:

Ui(τ) :={

u∣∣J j(x(τ),u)≤ J j(x(τ),u∗ ([τ,∞)) , j = 1, · · · ,N, j = i

}.

4

Consider a control u ∈ Ui(τ) and let ue ([0,∞)) be a control defined on [0,∞) such that ue ([0,τ)) =u∗ ([0,τ)) and ue ([τ,∞)) = u, then x(τ,0,ue([0,τ)) = x∗(τ). Further,

J j (0,x0,ue) =∫ ∞

0g j(t,x(t),ue(t))dt

=∫ τ

0g j(t,x∗(t),u∗(t)) dt +

∫ ∞

τg j(t,x(t),u(t)) dt

(as u ∈ Ui(τ), we have)

≤∫ τ

0g j(t,x∗(t),u∗(t)) dt +

∫ ∞

τg j(t,x∗(t),u∗(t)) dt

=∫ ∞

0g j(t,x∗(t),u∗(t)) dt = J j(0,x0,u∗).

The above inequality holds for all j = 1, · · · ,N, j = i. Clearly, ue ([0,∞))∈Ui(0) i.e., every elementu ∈ Ui(τ) can be viewed as an element ue ∈ Ui(0) restricted to the time interval [τ,∞). From thedynamic programming principle it follows directly that u∗ ([τ,∞)) has to minimize Ji(τ,x∗(τ),u) onUi(τ).

Another result that follows directly from lemma 2.2 is that if the argument at which some player’scost is minimized is unique, then this control is Pareto optimal too (for the proof see corollary 2.5in [7]).

Corollary 2.2. Assume J1(u) has a minimum which is uniquely attained at u∗. Then(J1(u∗),J2(u∗),

, · · · ,JN(u∗))

is a Pareto solution.

Again, from lemma 2.2 we give a result, though intuitively simple, which will be helpful to findPareto solutions of the cooperative game. Pareto optimality is preserved for every strictly monotonictransformation of the cost functions. So, if the player’s costs are modified as Ji(u) = Ji(u)− c, c ∈R, ∀i ∈ SN , then we have the following corollary.

Corollary 2.3. The set of Pareto optimal strategies for the games with player’s objectives as Ji(u)and Ji(u), i ∈ SN , is the same.

3 Necessary Conditions for the General CaseLet (P) be an N person infinite horizon cooperative differential game defined as follows:

(P) for each i ∈ SN minu∈U

∫ ∞

0gi(t,x(t),u(t))dt

sub. to x(t) = f (t,x(t),u(t)), x(0) = x0.

Let u∗(t) be a Pareto optimal strategy for the problem (P) and x∗(t) be the trajectory generated byu∗(t). Using lemma 2.2, (P) is equivalent to solving N constrained optimal control subproblems.Let (Pi) denote a constrained (w.r.t control space) optimal control subproblem for player i ∈ SN anddefined as:

(Pi) minu(t)∈Ui

∫ ∞

0gi(t,x(t),u(t))dt

sub. to x(t) = f (t,x(t),u(t)), x(0) = x0.

5

Introducing the auxiliary states xij(t) as

xij(t) =

∫ t

0gi(t,x(t),u(t))dt, xi

j(0) = 0,

the constrained set Ui can be represented by inequalities. Then (Pi) is equivalently reformulated as:

(Pi) minu(t)∈U

∫ ∞

0gi(t,x(t),u(t))dt

sub. to x(t) = f (t,x(t),u(t)), x(0) = x0

˙xij(t) = g j(t,x(t),u(t)), xi

j(0) = 0, limt→∞

xij(t)≤ xi∗

j , ∀ j ∈ SiN ,

where xi∗j =

∫ ∞0 g j(t,x∗(t),u∗(t))dt. We make the following assumption.

Assumption 1. For each subproblem (Pi), the Lagrange multipliers associated with the objectivefunction and the terminal state (xi

j) constraints are nonnegative with at least one of them strictlypositive.

Now we derive necessary conditions for (J1(u∗),J2(u∗), · · · ,JN(u∗)) to be a Pareto optimal solu-tion.

Theorem 3.1. If (J1(u∗),J2(u∗), · · · ,JN(u∗)) is a Pareto candidate for problem (P) and assump-tion (1) holds, then there exists an −→α ∈ PN , a costate function λ (t) : [0,∞) → Rn such that withH(−→α , t,x(t),u(t),λ (t)) = λ ′(t) f (t,x(t),u(t))+G(−→α , t,x(t),u(t)), the following conditions are sat-isfied.

H(−→α , t,x∗(t),u∗(t),λ (t))≤H(−→α , t,x∗(t),u(t),λ (t)), ∀u(t) ∈ U (7a)

H0(−→α , t,x∗(t),λ (t)) = minu(t)∈U

H(−→α , t,x∗(t),u(t),λ (t))

λ (t) =−H0x (−→α , t,x∗(t),λ (t)) (7b)

x∗(t) = H0λ (−→α , t,x∗(t),λ (t)) s.t x∗(0) = x0 (7c)(−→α ,λ (t)

)=0, ∀t ∈ [0,∞), −→α ∈ PN . (7d)

Proof. By assumption the objective functions Ji(u), i ∈ SN converge. We define the Hamiltonianassociated with (Pi) as:

Hi(t,x(t),u(t),λ (t)) = λ ′i (t) f (t,x(t),u(t))+λ 0

i gi(t,x(t),u(t))+ ∑

j∈SiN

µ ij(t)g j(t,x(t),u(t)). (8)

From Pontryagin’s maximum principle, if the pair (x∗(t),u∗(t)) is optimal for the problem (Pi) thenthere exist a constant λ 0

i and (continuous, piecewise continuously differentiable) costate functionsλi(t) ∈ Rn and µ i

j(t) ∈ R, j ∈ SiN such that:

(λ 0i ,λi(t),µ i

j(t)) = (0,0,0), j ∈ SiN , t ∈ [0,∞) (9a)

u(t) = arg minu∈U

Hi(t,x(t),u(t),λ (t)) (9b)

H0i (t,x(t),λ (t)) = min

u(t)∈UHi(t,x(t),u(t),λ (t))

λi(t) = −H0i x(t,x

∗(t),λ (t)) (9c)µ i

j(t) = −H0i xi

j(t,x∗(t),λ (t)). (9d)

6

Since (H0i )xi

j= 0, multipliers associated with the auxiliary variables µ i

j(t) are constants µ ij. We

define the set of multipliers as a vector−→λ i =

(µ i

1, · · · ,µ ii−1,λ

0i , µ i

i+1, · · · ,µ iN)′ and write the first

order conditions as:

λ ′i (t) f (t,x∗(t),u∗(t))+G(

−→λ i, t,x∗(t),u∗(t)) ≤ λ ′

i (t) f (t,x∗(t),u(t))+G(−→λ i, t,x∗(t),u(t))(10)

λi(t) =− f ′x(t,x∗(t),u∗(t))λi(t)−Gx(

−→λ i, t,x∗(t),u∗(t)). (11)

Summing (10) and (11) for all i ∈ SN yields

∑i∈SN

(λ ′

i (t) f (t,x∗(t),u∗(t))+G(−→λ i, t,x∗(t),u∗(t))

)≤ ∑

i∈SN

(λ ′

i (t) f (t,x∗(t),u(t))+G(−→λ i, t,x∗(t),u(t))

)(12)

∑i∈SN

λi(t) =− f ′x(t,x∗(t),u∗(t)) ∑

i∈SN

λi(t)− ∑i∈SN

Gx(−→λ i, t,x∗(t),u∗(t)). (13)

Let’s introduce d := ∑i∈SN

(λ 0

i +∑ j∈SiN

µ ij

). By assumption (1) we have d > 0. We define λ (t) :=

1d ∑i∈SN λi(t), αi := 1

d

(λ 0

i +∑ j∈SiN

µ ji

), i ∈ SN and a vector −→α := (α1, · · · ,αN)

′. Notice that −→α ∈PN by assumption (1). Dividing the equation (12) by d we have:

λ ′(t) f (t,x∗(t),u∗(t))+G(−→α , t,x∗(t),u∗(t))) ≤ λ ′(t) f (t,x∗(t),u(t))+G(−→α , t,x∗(t),u(t)) (14)

λ (t) =− f ′x(t,x∗(t),u∗(t))λ (t)−Gx(

−→α , t,x∗(t),u∗(t)). (15)

Next we define the modified Hamiltonian as

H(−→α , t,x(t),u(t),λ (t)) = λ ′(t) f (t,x(t),u(t))+G(−→α , t,x(t),u(t)).

The above necessary conditions for u∗(t) to be Pareto optimal control can be rewritten as (7).

Remark 1. Let K be a cone defined as K ={

α ∈ RN+ s.t α ′1N > 0

}. Clearly, if

−→λ i ∈K , ∀i ∈ SN ,

then all Pareto candidates of the problem (P) can be obtained by solving the necessary conditionsfor optimality associated with the corresponding weighted sum infinite horizon optimal controlproblem.

Remark 2. Generally transversality conditions are specified in addition to equations (7) to singleout optimal trajectories from the set of extremal trajectories satisfying (7). If the game problem(P) is finite horizon type, then the transversality conditions associated with the subproblem (Pi) areλi(T ) = 0, 0 < T < ∞ and µ i

j ≥ 0 for j ∈ SiN , i ∈ SN and the maximum principle holds in normal

form i.e., λ 0i = 1, i ∈ SN (refer to proposition 3.16, [9]). The assumption 1 holds true naturally

for the finite horizon case. So, for finite horizon games−→λi ∈ K , i ∈ SN , all Pareto candidates can

be obtained by solving the necessary conditions for optimality associated with the weighted sumfinite horizon optimal control problem (as shown in [7]). However, the transversality conditionsgenerally do not carry over to the subproblems (Pi) in infinite horizons. Refer to [10, 9, 18, 1] forcounterexamples to illustrate this behavior. A natural extension of transversality conditions, in theinfinite horizon, can be made by imposing certain restrictions on the functions f (.), gi(.), i ∈ SN ,(see [16, 1, 18, 10, 22]).

7

It is clear, from the above remarks, that an extension of finite horizon transversality conditions tothe infinite horizon case is sufficient to obtain all Pareto candidates from the necessary conditionsassociated with the corresponding weighted sum optimal control problem. So, for the analysisthat follows from now onwards we focus on the growth conditions which allow such an extension.Towards that end, we first consider non-autonomous systems. From [18, theorem 3.16] or [17,example 10.3], we have the following corollary:

Corollary 3.1. Suppose∫ ∞

0 gi(t,x(t),u(t))dt >−∞, i ∈ SN and there exist non-negative numbers a,b and c with c > Nb such that the following conditions are satisfied for t ≥ 0 and all x(t):

(|gix(t,x(t),u(t)|)m ≤ ae−ct , m = 1, · · · ,N, ∀i ∈ SN (16a)(| fx(t,x(t),u(t)|)(l,m) ≤ b, l = 1, · · · ,N, m = 1, · · · ,N. (16b)

Then assumption 1 is satisfied. Consequently, for every Pareto solution the necessary conditionsgiven by (7) hold true and in addition limt→∞ λ (t) = 0 is satisfied.

Proof. If conditions (16) hold true, then by [18, theorem 3.16], the finite horizon transversalityconditions do extend to the infinite horizon case. As a result, λ 0

i = 1, µ ji ≥ 0, ∀ j ∈ Si

N andlimt→∞ λi(t) = 0 are satisfied for the constrained optimal control problem (Pi) and

−→λ i ∈K , ∀i∈ SN .

Clearly assumption (1) is satisfied. So, the necessary conditions given by (7) hold true and in addi-tion limt→∞ λ (t) = 0.

Example 1 (Necessary Conditions): We consider the game problem (P) mentioned in example 1.It is easily verified that for (P), | fx(.)| = 0, |g1x(.)| = 0, |g2x(.)| ≤ 2|x(t)||u1(t)− u2(t)| ≤ c (thereexists a constant c > 0). Furthermore, we see that −∞ < Ji(x0, u1, u2)< ∞. The growth conditionsmentioned in corollary 3.1 hold true for the game problem (P). Then from theorem 3.1 there existsa costate function λ (t), with Hamiltonian defined as

H(.) := λ (t)(u1(t)− u2(t))+(α − (1−α)x2(t)

)(u2(t)− u1(t)) .

Further, H(.) attains a minimum w.r.t ui(.), i = 1,2 only if

λ (t)+(

α − (1−α)x∗2(t))= 0 for all t ∈ [0,∞). (17)

As the growth conditions are satisfied we have limt→∞ λ (t) = 0. The adjoint variable λ (t) satisfies(by differentiating (17))

˙λ (t) = 2(1−α)x∗(t)(u2(t)− u1(t)), ∀t ∈ [0,∞) limt→∞

λ (t) = 0.

We see that the necessary condition (7b) also results in the same differential equation for λ (t),∀ t ∈ [0,∞). Using (17) we can conclude that for arbitrary choices of u1(.) and u2(.), theorem 3.1holds true by choosing α such that limt→∞ x∗

2(t) = α

1−α .

3.1 Discounted Autonomous systemsMany economic applications/situations can be effectively modeled as discounted autonomous op-timal control problems. The growth conditions given in corollary 3.1 ensure that assumption 1 issatisfied. However, the conditions 16 are quite strict. In this section we analyze games defined byautonomous systems with exponentially discounted player’s costs. The discount factor ρ is assumedto be strictly positive, i.e., ρ > 0. We represent the game problem in this section as (Pρ) and the

8

related subproblem as (Pρi ). We notice that the subproblem (Pρ

i ) is a mixed endpoint constrained op-timal control problem. The cost minimizing efforts of players in the coalition Si

N result in terminalendpoint constraints in (Pρ

i ). For discounted autonomous systems, [16] proves necessary conditionsfor optimality for free endpoint infinite horizon optimal control problems. We first derive the nec-essary conditions for optimality for the mixed endpoint constrained problem (Pρ

i ) (in similar linesof [16]). If u∗(t) is a Pareto optimal strategy for the game problem (Pρ ), then from lemma 2.2 u∗(t)is optimal for each optimal control problem (Pρ

i ), i ∈ SN .

(Pρi ) min

u∈U

∫ ∞

0e−ρtgi(x(t),u(t))dt

sub. to x(t) = f (x(t),u(t)), x(0) = x0, u(t) ∈ U

˙x j(t) = e−ρtg j(x(t),u(t)), xij(0) = 0, lim

t→∞xi

j(t)≤ xi∗j , ∀ j ∈ Si

N .

Let (x∗(t),u∗(t)), 0 ≤ t ≤ ∞ be the optimal admissible pair for problem (Pρi ), we fix T > 0 and

define hi(z), i ∈ SN as:

hi(z) =∫ ∞

ze−ρtgi (x∗(t − z+T ),u∗(t − z+T ))dt. (18)

To derive the necessary conditions for optimality of u∗(t), we first consider the following truncatedand augmented problem (Pρ

iT ) (associated with the subproblem (Pρi )) defined as follows:

(PρiT ) min

u∈Uhi(z(T )−T )+

∫ T

0v(t)e−ρz(t)gi(Y (t),U(t))dt

sub. to Y (t) = v(t) f (Y (t),U(t)), Y (0) = Y0, Y (T ) = x∗(T ), U(t) ∈ U˙Y i

j(t) = v(t)e−ρz(t)g j(Y (t),U(t)),

Y ij(0) = 0, Y i

j(T )+h j(z(T )−T )≤ xi∗j , ∀ j ∈ Si

N

z(t) = v(t), z(0) = 0, v(t) ∈[1/2,∞

).

Remark 3. In problem (PρiT ), the constraint v(t) ∈ [1/2,∞) can be replaced by v(t) ∈ [v,∞) for

0 < v < 1.

Lemma 3.1. If (x∗(t),u∗(t)) is an optimal admissible pair for the problem (Pρi ) then (x∗(t), t,u∗(t),1),

t ∈ [0, T ], is an optimal admissible pair for the problem (PρiT ).

Proof. We prove the lemma using contradiction. Suppose (x∗(t), t,u∗(t),1), t ∈ [0, T ] is not optimalfor the problem (Pρ

iT ) then there exists an admissible pair (Y (t), z(t),U(t), v(t)), t ∈ [0, T ] such that

hi(z(T )−T )+∫ T

0v(t)e−ρ z(t)gi(Y (t),U(t))dt <

∫ ∞

0e−ρtgi(x∗(t),u∗(t))dt,

h j(z(T )−T )+∫ T

0v(t)e−ρ z(t)g j(Y (t),U(t))dt ≤

∫ ∞

0e−ρtg j(x∗(t),u∗(t))dt, ∀ j ∈ Si

N .

Since v(t) ∈ [1/2, ∞), z(t) is an increasing function defined on [0, T ] so by the inverse functiontheorem z(t) is invertible on [0, z(T )]. We define x(s) = Y (z−1(s)) and u(s) = U(z−1(s)) for s ∈[0, z(T )] and observe that x(0) = Y (z−1(0)) = x0 and x(z(T )) = Y (z−1(z(T ))) = Y (T ) = x∗(T ).Further, we have x(s) defined on s ∈ [0, z(T )] as:

x(z(t)) = x0 +∫ t

0

˙Y (t)dt = x0 +∫ T

0v(t) f (Y (t),U(t))dt.

9

Taking s = z(t) we have

x(s) = x0 +∫ s

0f (Y (z−1(s)),U(z−1(s)))ds = x0 +

∫ s

0f (x(s), u(s))ds,

for s∈ [0, z(T )]. Since x(s) satisfies the above integral equation we have ˙x(s)= f (x(s), u(s)), x(0)=x0, s ∈ [0, z(T )]. Next, for s > z(T ), we define x(s) = x∗(s− z(T )+T ) and u(s) = u∗(s− z(T )+T ).Then we observe that x(s) = f (x(s),u(s)) with x(z(T )) = x∗(T ). Clearly the pair (x(s), u(s)), s ∈[0, ∞

)is admissible for problem (Pρ

i ) and satisfies the following conditions∫ ∞

0e−ρtgi(x(t), u(t))dt <

∫ ∞

0e−ρtgi(x∗(t),u∗(t))dt∫ ∞

0e−ρtg j(x(t), u(t))dt ≤

∫ ∞

0e−ρtg j(x∗(t),u∗(t))dt, ∀ j ∈ Si

N ,

which clearly violates the optimality of (x∗(t),u∗(t)) for the problem (Pρi ).

Theorem 3.2. If (x∗(t),u∗(t)), t ∈[0, ∞

)is an optimal pair for the problem (Pρ

i ) then there exist−→λ i ∈ RN

+, l0i ∈ Rn and continuous functions λi(t) ∈ Rn and γi(t) ∈ R respectively such that(−→

λ i,λi(t),γi(t))= (0,0,0), ∀t ≥ 0,

∥∥∥(−→λ i,λi(0))∥∥∥= 1 (19a)

λi(t) =−e−ρtGx(−→λ i,x∗(t),u∗(t))− f ′x(x

∗(t),u∗(t))λi(t), λi(0) = l0i (19b)

γi(t) = ρ e−ρtG(−→λ i,x∗(t),u∗(t)), lim

t→∞γi(t) = 0 (19c)

H(−→λ i, t,x∗(t),u(t),λ (t)) = λ ′

i (t) f (x∗(t),u∗(t))+ e−ρtG(−→λ i,x∗(t),u∗(t))

H(−→λ i, t,x∗(t),u∗(t),λ (t))≤ H(

−→λ i, t,x∗(t),u(t),λ (t)) ∀u(t) ∈ U (19d)

H(−→λ i, t,x∗(t),u∗(t),λ (t)) =−γi(t). (19e)

Proof. (PρiT ) is a mixed endpoint constrained finite horizon optimal control problem. We first define

the Hamiltonian HiT

(−→λ iT ,z(t),Y (t),v(t),U(t)

)as:

HiT (.) = v(t)e−ρz(t)

λ 0iT gi(Y (t),U(t))+ ∑

j∈SiN

µ ijT (t)g j(Y (t),U(t))

+ v(t)λ ′

i (t) f (Y (t),U(t))+ v(t)γi(t). (20)

The necessary conditions are: there exist λ 0iT ∈R+, µ i

jT (t) ∈R, λiT (t) ∈Rn, γiT (t) ∈R such that foralmost every t ∈ [0,T ] (the partial derivatives of the Hamiltonian given below are evaluated at theoptimal pair ((x∗(t), t),(u∗(t),1))):

λiT (t) = −(HiT )Y

= − f ′x(x∗(t),u∗(t)z)λiT (t)−λ 0

iT e−ρtgix(x∗(t),u∗(t))

− ∑j∈Si

N

µ ijT (t)e

−ρtg jx(x∗(t),u∗(t)), λiT (T ) free

µ ijT (t) = −(HiT )Y i

j, µ i

jT (T )≥ 0, µ ijT (T )

(x j(T )+h j(0)− x∗j

)= 0, ∀ j ∈ Si

N

γiT (t) = −(HiT )z , γiT (T ) = λ 0iT hix(0)+µ i

jT (T )h jx(0)(λ 0

iT , {µ ijT (t), j ∈ Si

N},λiT (t),γiT (t))= (0, · · · ,0) .

10

Since (HiT )xij= 0, we have µ i

jT (t)= µ ijT ≥ 0, ∀ j ∈ Si

N . Let−→λ iT =

(µ i

1T, · · · ,µ i

i−1T, λ 0

iT ,µii+1T

, · · · ,µ iNT

).

Then−→λ iT ∈RN

+. Next we show by contradiction that also from the necessary conditions(−→

λ iT ,λiT (0))=

(0,0). For, if(−→

λ iT ,λiT (0))= (0,0) then the necessary conditions give that

λiT (t) =− f ′x(x∗(t),u∗(t))λiT (t)

which results in λiT (t) = 0 for t ∈ [0,T ]. Further,−→λ iT = 0 leads to γiT (t) = 0 for t ∈ [0,T ] which

violates the necessary condition(−→

λ iT ,λiT (t),γiT (t))= (0,0,0) for all 0 ≤ t ≤ T . Since λiT (T ) is

free, we choose (without loss of generality) λiT (0) such that∥∥∥(−→λ iT ,λiT (0)

)∥∥∥ = 1. The adjointvariable λiT (t) satisfies:

λiT (t) =− f ′x(x∗(t),u∗(t))λiT (t)− e−ρtGx(

−→λ iT ,x

∗(t),u(t)), λiT (0) = l0iT , (21)

whereas γiT (t) satisfies

γiT (t) =−(HiT )z = ρ G(−→λ iT ,x

∗(t),u∗(t)). (22)

From the definition of h(.) we have:

γiT (T ) = ρλ 0iT

∫ ∞

Te−ρtgi(x∗(t),u∗(t))dt +ρ ∑

j∈SiN

µ ijT

∫ ∞

Te−ρtg j(x∗(t),u∗(t))dt

= ρ∫ ∞

Te−ρtG(

−→λ iT ,x

∗(t),u∗(t))dt. (23)

The Hamiltonian is linear in v(t) and the minimum w.r.t (U(t),v(t)) on the set U × [1/2, ∞) isattained at (u∗(t),1). The minimum of the Hamiltonian w.r.t v(t) for U(t) = u∗(t) is achieved at aninterior point of [1/2, ∞), so we have:

e−ρtG(−→λ iT ,x

∗(t),u∗(t))+λ ′iT (t) f (x∗(t),u∗(t))+ γiT (t) = 0, ∀t ∈ [0,T ]. (24)

The minimum of the Hamiltonian w.r.t U(t) is independent of v(t) (positive scaling) and does notdepend on the term γi(t)v(t). So, the minimum of HiT w.r.t U(t) at v(t) = 1 is achieved at

u∗(t) = arg minU(t)∈U

HiT (−→λ iT , t,x

∗(t),z(t) = t,U(t),v(t) = 1,λiT (t),γiT (t))

= arg minu(t)∈U

(e−ρtG(

−→λ iT ,x

∗(t),u(t))+λ ′iT (t) f (x∗(t),u(t))

), ∀t ∈ [0,T ]. (25)

Now, consider an increasing sequence {Tk}k∈N such that limk→∞ Tk = ∞. We can associate anoptimal control problem (PiTk

) with each Tk such that the necessary conditions as discussed above

hold true. Then there exists a sequence{(−→

λ iTk, l0

iTk

)}such that

−→λ iTk

∈RN+ and

∥∥∥(−→λ iTk, l0

iTk

)∥∥∥= 1.We know from the Bolzano-Weierstrass theorem that every bounded sequence has a convergentsubsequence. Using the same indices for such a subsequence we infer that there exists

−→λ i ∈ RN

+

and l0i such that

limk→∞

−→λ iTk

=−→λ i ∈ RN

+ and limk→∞

l0iTk

= l0i such that

∥∥∥(−→λ i, l0i

)∥∥∥= 1. (26)

11

We observe that (29c) is a linear ODE. So we can write λiTk(t) as:

λiTk(t) = Φ− f ′x(t,0)l

0iTk

−∫ t

0Φ− f ′x(t,s)e

−ρsGx(−→λ iT ,x

∗(s),u∗(s))ds, (27)

where Φ− f ′x(t,s) is the fundamental matrix associated with z(t) = − f ′x(x∗(t),u∗(t)) z(t). Since the

weights of−→λ iTk

appear linearly in Gx(.) as k → ∞, λi(t) satisfies the differential equation (19b).A similar argument holds for γiTk

(t), condition (24) and (25) resulting in (19c), (19e) and (19d)respectively.

Remark 4. Though the approach in lemma 3.1 and theorem 3.2 is similar to the one given in [16],the main differences lie in problem formulation. In [16], the necessary conditions are obtained forthe free endpoint unconstrained infinite horizon optimal control problem. In our case the problem(Pρ ) is written as N mixed endpoint constrained optimal control subproblems (due to cost mini-mizing efforts of players). Furthermore, these constraints have a special structure. As a result theterm G(

−→λ i,x∗(t),u∗(t)), weighted instantaneous undiscounted cost of the players, appears in all the

subproblems (the choice of weights being Lagrange multipliers associated with the correspondingsubproblem).

For autonomous systems [16] gives assumptions under which limt→∞ λi(t) = 0 for the free end-point case with maximization problem. We show next, in corollary 3.2, that these conditions (formu-lated as assumption 2 below) also suffice to conclude that in our subproblem (Pρ

i ) limt→∞ λi(t) = 0.The proof is similar to the one given in [16]. We repeat it for the sake of completeness.

Assumption 2. gi(x(t),u(t)), ∀i ∈ SN is non positive and there exists a neighborhood V of 0 ∈ Rn

which is contained in the set of possible velocities f (x∗(t),u(t)) for all u(t) ∈ U if t → ∞.

Corollary 3.2. Let assumption 2 hold true. Then, an optimal solution for the problem (Pρi ) satisfies

in addition to the conditions (19), the following transversality condition: limt→∞ λi(t) = 0.

Proof. From the necessary conditions (19d) and (19e) of theorem 3.2 we have

e−ρtG(−→λ i,x∗(t),u(t))+λ ′

i (t) f (x∗(t),u(t)) ≥ e−ρtG(−→λ i,x∗(t),u∗(t))

+λ ′i (t) f (x∗(t),u∗(t)) =−γi(t). (28)

By assumption 2 we have λ ′i (t) f (x∗(t),u(t))≥−γi(t). Next define q(t) as follows:

q(t) =λi(t)

max{1, ||λi(t)||}. Since ||q(t)|| ≤ 1 we have limsup

t→∞||q(t)||= l.

If l = 0 there is nothing to prove. So assume l > 0 and consider a sequence {tn} converging toinfinity such that ||q(tn)|| > l/2. Since there exists u(t) ∈ U such that Bδ>0

1 ⊂ f (x∗(t),u(t)),there exists an ε > 0 such that 2ε

l < δ . So, there exists un(tn) ∈ U such that f (x∗(tn),un(tn)) =−(2ε/l)q(tn). Since, limt→∞ γi(t) = 0 we take the above sequence {tn} such that −lε/2≤−γ(tn)≤lε/2. Collecting all the above we have:

λ ′i (tn) f (x∗(tn),un(tn)) =−max{1, ||λi(tn)||}(2ε/l) ||q(tn)||2 ≥−γi(tn)

γi(tn)≥ max{1, ||λi(tn)||}(2ε/l) ||q(tn)||2 > lε/2.

Clearly, this is a contradiction and thus limt→∞ λ (t) = 0.

1Unit ball in Rnof radius δ > 0.

12

Remark 5. a) We can relax the nonpositivity restriction made in assumption 2. If the instanta-neous undiscounted costs of players gi(x(t),u(t)), i ∈ SN are bounded above for all pairs(x(t),u(t)), t ∈ [0,∞) then by assigning a new cost gi(x(t),u(t)) = g(x(t),u(t))− M withM = maxi∈SN supt∈[0,∞) gi(x(t),u(t)) leaves g(x(t),u(t)) nonpositive. Now, by defining anew game (Pρ ) with g(.) as the instantaneous undiscounted costs for player i we observe thatPareto optimal controls (if they exist) of (Pρ ) and (Pρ ) coincide. We will use this idea inexample 2 to find the Pareto optimal controls.

b) Notice that the second condition in assumption 2 is identical to the notion of state reachabilitywhen the state dynamics is described by a linear constant coefficient differential equation.

We introduce another assumption based on growth conditions [18, 1] which ensure the transversalitycondition i.e., limt→∞ λi(t) = 0 and as a result

−→λ i ∈ K for the subproblem (Pρ

i ).Assumption 3. a) There exist a s ≥ 0 and an r ≥ 0 such that

||gix(x(t),u(t))|| ≤ s(1+ ||x(t)||r) for all x(t) ∈ Rn,u(t) ∈ U and i ∈ SN .

b) There exist nonnegative constants c1, c2, c3 and λ ∈ R, such that for every admissible pair(x(t),u(t)), one has

||x(t)|| ≤ c1 + c2eλ t for all t ≥ 0,||Φ fx(t,0)|| ≤ c3eλ tfor all t ≥ 0.

c) For every admissible pair (x(t),u(t)) the eigenvalues of fx(x(t),u(t)) are strictly positive.

Corollary 3.3. Let the assumption 3 hold true. Then, an optimal solution for the problem Pi satisfiesin addition to the conditions (19), the following transversality condition limt→∞ λi(t) = 0 if ρ >(1+ r)λ .

Proof. From the assumption 3 there exist constants c4 ≥ 0, c5 ≥ 0, c6 ≥ 0 and c7 ≥ 0 such that:

e−ρt |gi(x(t),u(t))| ≤ e−ρt (c4 + c5||x(t)||r+1)≤ c6e−ρt + c7e−(ρ−(r+1)λ )t .

Since ρ > (1+ r)λ , the player’s costs Ji(u) converge for every admissible pair (x(t),u(t)). For thesubproblem (Pi) we rewrite λi(t) (from (19b)) as follows:

λi(t) = Φ− f ′x(t,0) l0i −

∫ t

0e−ρsΦ− f ′x(t,s) Gx(x∗(s),u∗(s))ds

= Φ− f ′x(t,0)(

l0i −

∫ t

0e−ρsΦ− f ′x(0,s) Gx(x∗(s),u∗(s))ds

)(

we know Φ− f ′x(t,s) =(

Φ−1fx (t,s)

)′= Φ′

fx(s, t))

= Φ− f ′x(t,0)(

l0i −

∫ t

0e−ρsΦ′

fx(s,0) Gx(x∗(s),u∗(s))ds).

The norm of λi(t) is bounded as:

||λi(t)|| ≤∥∥Φ− f ′x(t,0)

∥∥(||l0i ||+

∫ t

0e−ρs∥∥Φ fx(s,0)

∥∥∥Gx(x∗(s),u∗(s))∥ds)

(from (3b) there exist a c8 ≥ 0, a c9 ≥ 0 such that)

≤∥∥Φ− f ′x(t,0)

∥∥(||l0i ||+

∫ t

0e−ρs

(c8eλ s + c9e(1+r)λ s

)ds)

(l0i is bounded, so there exist a c10 ≥ 0, a c11 ≥ 0 and c12 ≥ 0) such that

≤∥∥Φ− f ′x(t,0)

∥∥(c10 + c11e−(ρ−λ )t + c12e−(ρ−(1+r)λ )t).

13

Let ϕ 0(t) and ϕ0(t) denote the largest and smallest eigenvalues of the Hermitian part2 of − f ′x(x∗(t),u∗(t)).

By assumption − f ′x(x∗(t),u∗(t)) is bounded and has strictly negative eigenvalues so we have −∞ <

ϕ0(t)≤ ϕ 0(t)< 0, ∀t ≥ 0. From [11, lemma 4.2] we have:

exp(∫ T

0µ0(s)ds

)≤∥∥Φ− f ′x(t,0)

∥∥≤ exp(∫ T

0µ0(s)ds

).

Since µ0(s) < 0 for all s ≥ 0 we have limt→∞∥∥Φ− f ′x(t,0)

∥∥ = 0. By assumption ρ > (1+ r)λ solimt→∞ λi(t) = 0 follows directly.

Remark 6. a) When the state evolution dynamics is linear and player’s objectives are convex inthe control variable, the growth conditions (a) and (b) given in assumption 3 are similar to theones given in [1] and [2].

b) In [1], the free endpoint infinite horizon optimal control problem is approximated with a seriesof free endpoint finite horizon problems whereas in the current approach (Pρ

i ) is approximatedwith fixed endpoint problems. As a result an additional term

∥∥Φ− f ′x(t,0)∥∥(||l0

i ||)

appears inthe estimate of ||λi(t)|| and condition (c) has to be included in assumption 3.

Theorem 3.3. Let assumption 2 or 3 hold true. If (J1(u∗),J2(u∗), · · · ,JN(u∗)) is a Pareto optimalsolution for problem (P) then there exists an −→α ∈ PN and a costate function λ (t) : [0,∞) → Rn

such that the following conditions are satisfied.

H(−→α , t,x(t),u(t),λ (t)) = λ ′(t) f (t,x(t),u(t))+ e−ρtG(−→α ,x(t),u(t)) (29a)

H(−→α , t,x∗(t),u∗(t),λ (t))≤ H(−→α , t,x∗(t),u(t),λ (t)), ∀u(t) ∈ U (29b)

H0(−→α , t,x∗(t),λ (t)) = minu(t)∈U

H(−→α , t,x∗(t),u(t),λ (t))

λ (t) =−H0x (−→α , t,x∗(t),λ (t)), lim

t→∞λ (t) = 0 (29c)

x∗(t) = H0λ (−→α , t,x∗(t),λ (t)), x∗(0) = x0 (29d)

γ(t) = ρ G(−→α ,x∗(t),u∗(t)), limt→∞

γ(t) = 0 (29e)

H0(−→α , t,x∗(t),λ (t)) =−γ(t) (29f)(−→α ,λ (t))= 0, ∀t ∈ [0,∞), −→α ∈ PN . (29g)

Proof. If assumption 2 holds true then for each subproblem (Pρi )

−→λ i ∈ K and limt→∞ λi(t) = 0.

We define d = ∑i∈SN

−→λ ′

i1N , αi =1d

(λ 0

i +∑ j∈SiN

µ ji

), i ∈ SN and a vector −→α = (α1, · · · ,αN)

′. We

notice that −→α ∈ PN . Taking the summation of equation (19b) for all i ∈ SN and defining λ (t) =1d ∑i∈SN λi(t) we observe that conditions (29b) and (29c) are satisfied. Taking the summation ofequation (19c) for all i ∈ SN and defining γ(t) = 1

d ∑i∈SN γi(t), the conditions (29e) and (29f) aresatisfied. Since, −→α ∈ PN ⊂ K and limt→∞ λ (t) = 0 we observe that (29g) is satisfied.

Remark 7. Since equations (29a-29g) are necessary conditions for a Pareto solution, solving themgives Pareto candidates.

We consider the example from [13] to illustrate usage of assumption 2 and necessary conditionsgiven in theorem 3.3.

2The Hermitian part of matrix A is defined here as AH = 12 (A+A′).

14

Example 2. Consider a fishery game with two players. The evolution of the stock of fish, in aparticular area, is governed by the differential equation

x(t) = ax(t)−bx(t) lnx(t)−u1(t)−u2(t), x(0) = x0 ≥ 2 (30)

where x(t) refers to the stock of fish, and a > 0, b > 0. It is assumed that x(t) ≥ 2, t ∈ [0,∞). In(30), the stock of fish x(t) depends upon ax(t) births, bx(t) lnx(t) deaths and the fishing efforts ofplayer i, ui(t) = wi(t)x(t), at each point in time t. Each fisherman tries to maximize his utility Ji(.),given by:

Ji(x0,u1,u2) =∫ ∞

0e−ρt lnui(t) dt.

We assume 0 < ε ≤ wi(t) < ∞ for the utility to be well defined. By taking the transformationy(t) = lnx(t) the system (30) is modified as:

y(t) = a−by(t)−w1(t)−w2(t), y(0) = lnx(0), (31)

and the player’s utility is transformed as:

Ji(x0,u1,u2) =∫ ∞

0e−ρt (y(t)+ lnwi(t)) dt. (32)

We notice that the instantaneous undiscounted reward is bounded below, by controllability of thesystem (31) and remark 5.a we notice that assumption 2 is satisfied. So all Pareto candidates canbe obtained by solving the necessary conditions associated with the weighted sum optimal controlproblem :

minw1,w2

{−∫ ∞

0e−ρt (y(t)+α lnw1(t)+(1−α) lnw2(t))dt

},

subject to (31). Defining the Hamiltonian as

H(α, t,y,w1,w2,λ ) = λ (a−by(t)−w1(t)−w2(t))− e−ρt (y(t)+α lnw1(t)+(1−α) lnw2(t)) .

Taking Hwi = 0, i = 1,2 gives w1(t) = − αλ (t)e

−ρt and w2(t) = −1−αλ (t) e−ρt . The adjoint variable is

governed by

λ (t) = bλ (t)+ e−ρt , limt→∞

λ (t) = 0,

and the solution is given as λ (t) =− e−ρt

ρ+b . The candidates for Pareto optimal strategies in open loopform are given by:

u∗1(t) = α(ρ +b)em(t,x0),

u∗2(t) = (1−α)(ρ +b)em(t,x0),

where m(t,x0) = e−bt lnx0 +a−(b+ρ)

b

(1− e−bt

). The candidates for Pareto solutions are given as

J∗1(x0,u1,u2) =ρ lnx0 +a− (ρ +b)

ρ(ρ +b)+

ln(α(ρ +b))ρ

,

J∗2(x0,u1,u2) =ρ lnx0 +a− (ρ +b)

ρ(ρ +b)+

ln((1−α)(ρ +b))ρ

. (33)

15

4 Sufficient Conditions for Pareto OptimalityIt is well known [8] that if the action spaces as well as the players objective functions are convexthen minimization of the weighted sum of the objectives results in all Pareto solutions. We give thefollowing theorem from [5].

Theorem 4.1. If U is convex and Ji(u) is convex for all i = 1,2, · · · ,N then for all Pareto optimalu∗ there exist −→α ∈ PN , such that u∗ ∈ argminu∈U ∑N

i=1 Ji(u).

Recently in [6], this property was used to obtain both necessary and sufficient conditions for theexistence of Pareto optimal solutions for regular convex linear quadratic differential games. Ingeneral it is a difficult task to check if the players objectives are convex functions of initial state andcontrols. However, under some conditions the solutions of (29) result in Pareto optimal strategies.In this section we derive sufficient conditions for a strategy to be Pareto optimal. The sufficientconditions given in the theorem below are inspired by Arrow’s sufficient conditions [18] in optimalcontrol. Further, these sufficient conditions are given for nonautonomous systems and they holdtrue for discounted autonomous systems as well.

Theorem 4.2. Assume that there exists −→α ∈PN and a costate function λ (t) : [0,∞)→Rn satisfying(29c). Introduce the Hamiltonian H(t,−→α ,x(t),u(t),λ (t)) := f (t,x(t), u(t)) +G(−→α , t,x(t),u(t)).Assume that the Hamiltonian has a minimum w.r.t u(t) for all x(t), denoted by

H0(−→α , t,x(t),λ (t)) = minu(t)∈U

H(−→α , t,x(t),u(t),λ (t)).

If H0(−→α , t,x(t),λ (t)) is convex in x(t) and liminft→∞ λ ′(t) (x∗(t)− x(t)) ≥ 0, then u∗(t) is Paretooptimal.

Proof. From the convexity of H0(−→α , t,x(t),λ (t)) we have:

H0(−→α , t,x(t),λ (t))−H0(−→α , t,x∗(t),λ (t))≥ H0x (−→α , t,x∗(t),λ (t))(x(t)− x∗(t))

Since, H(−→α , t,x(t),u(t),λ (t))≥H0(−→α , t,x(t),λ (t)) and H(−→α , t,x∗(t),u∗(t),λ (t))= H0(−→α , t,x∗(t),λ (t))we have:

H(−→α , t,x(t),u(t),λ (t))−H(−→α , t,x∗(t),u∗(t),λ (t))≥H0

x′(−→α , t,x∗(t),λ (t))(x(t)− x∗(t))

=−λ ′(t)(x(t)− x∗(t)) (by (29c)).

Using the definition of Hamiltonian the above inequality can be written as:

λ ′(t)( f (t,x(t),u(t)− f (t,x∗(t),u∗(t)))+G(−→α , t,x(t),u(t))−G(−→α , t,x∗(t),u∗(t))≥−λ ′(t)(x(t)− x∗(t)) ,

(G(−→α , t,x(t),u(t))−G(−→α , t,x∗(t),u∗(t))

)≥ λ ′(t)(x∗(t)− x(t))+λ ′(t)(x∗(t)− x(t))

=ddt

(λ ′(t)(x∗(t)− x(t))

).

Taking the integrals on both sides we have∫ T

0

(G(−→α , t,x(t),u(t))−G(−→α , t,x∗(t),u∗(t))

)dt ≥

(λ ′(t)(x∗(t)− x(t))

)∣∣∣T0.

16

As x∗(0) = x(0) = x0 and λ (0) is bounded the above inequality is given as:∫ T

0

(G(−→α , t,x(t),u(t))−G(−→α , t,x∗(t),u∗(t))

)dt ≥

(λ ′(T )(x∗(T )− x(T ))

).

Taking T → ∞

J(u)− J(u∗)≥ limt→∞

λ ′(t)(x∗(t)− x(t))≥ liminft→∞

λ ′(t)(x∗(t)− x(t))≥ 0.

Clearly, by lemma 2.1 u∗ is Pareto optimal.

Example 1 (sufficient conditions): We illustrate theorem 4.2 by considering example 1 again. First,we notice that H0(t, x∗, λ ) = 0 so H0(t, x∗, λ ) is convex in x(t). Next, limt→∞ x(t) exists and is finiteand limt→∞ λ (t) = 0, so liminft→∞ λ (t) (x∗(t)− x(t)) = 0. So, by theorem 4.2 every control (u1,u2)is Pareto optimal.

0 5 10 15 20 25 30 35 400

5

10

15

20

25

30

35

40

45

time units

u∗

2(t)

u∗

1(t)

x∗(t)

(a) Trajectories of optimal fish stock x∗(t) andfishing efforts of players u∗1(t) and u∗2(t)

0 5 10 15 20 25 300

5

10

15

20

25

30

J1

J2

(b) Pareto surface for 0.1714 ≤ α ≤ 0.8266

Figure 1: Fishery model with parameters a = 1, b = 0.2, ρ = 0.05 and x0 = 2.

Example 2 (sufficient conditions): For example 2 the candidates for Pareto solutions are given by(33). If the model in example 2 satisfies the sufficient conditions, mentioned in theorem 4.2, thenall Pareto solutions are indeed given by (33). The minimized Hamiltonian is given by:

H0(t,y(t),λ (t)) =−ρ y(t)ρ +b

e−ρt + e−ρt(

1− aρ +b

− ln(αα(1−α)1−α(ρ +b)

)).

Clearly, H0(t,y(t),λ (t)) is convex (linear here) in y(t). Since wi(t), i = 1,2, is bounded we have|y(t)| ≤ (c1 + c2e−bt). Further, λ (t) = − e−ρt

ρ+b , thus we have liminft→∞ λ (t)(y∗(t)− y(t)) = 0. Thefishery model satisfies the sufficient conditions as given by theorem 4.2. So, all the candidatesgiven by (33) are Pareto solutions. Figure 1(a) illustrates trajectories of optimal fish stock x∗(t)and fishing efforts of the players u∗1(t) and u∗2(t). Figure 1(b) illustrates the Pareto surface forα ∈ [0.1714, 0.8266] (Pareto solutions which ensure positive returns for the players are shown).

5 Linear Quadratic CaseIn this section we consider the discounted infinite planning horizon linear quadratic cooperativedifferential game (denoted as (PLQ)). The player i ∈ SN may choose his control trajectory, ui(.)

17

from the set of piecewise continuous functions and u(.) = (u1(.), · · · ,uN(.))T ∈ U (the specific

choice of control space will be clarified later). The problem is to determine the set of Pareto candi-dates/solutions for the cooperative game defined by

(PLQ) minu(.)∈U

Ji(x0,u) (34)

Ji(x0,u) =∫ ∞

0e−ρt(x′(t) u′(t))Mi(x′(t) u′(t))′dt,

sub. to x(t) = Ax(t)+ ∑i∈SN

Biui(t), x(0) = x0 and limt→∞

x(t) free. (35)

where ρ > 0, Mi =

(Qi ViV T

i Ri

)is symmetric, Ri > 0, i ∈ SN and we make no definiteness assumption

w.r.t Qi. x(t) is the solution of the linear differential equation. We define the weighted sum linearquadratic regulator problem (WSLQRP) as follows:

(WSLQRP) minu(.)∈U

∫ ∞

0e−ρt(x′(t) u′(t))Mα(x′(t) u′(t))′dt,

x(t) = Ax(t)+Bu(t), x(0) = x0 and limt→∞

x(t) free.

Where Mα = ∑i∈SN αiMi,−→α ∈ PN , B = [B1, B2, · · · , BN ] and U is a control space. We are

interested in finding weak conditions under which all Pareto candidates of (PLQ) are obtained bysolving the equations associated with the necessary conditions for optimality of (WSLQRP). Wedefine the following spaces:

a) LN2,loc := {u |

∫ ∞0 u′(t)u(t)dt < ∞}, i.e., the set of locally square-integrable functions.

b) Ubnd :={

u(.) ∈ LN2,loc

∣∣ ||u(t)||< m < ∞, ∀ t ∈ [0,∞)}

.

c) L+2,s(x0;x) :=

{u ∈ LN

2,loc| s.t limt→∞ x(t) = 0, x(t) = Ax(t)+Bu(t), x(0) = x0

}.

Remark 8. It can be proved, for instance [7, see lemma 2.1], that the control spaces mentionedabove are convex.

We take LN2,loc as the choice of control space U unless otherwise specified. In the following corollary

we give conditions under which assumption 2 is satisfied.

Corollary 5.1. Let the choice of control space be Ubnd. If the pair (A,B) is controllable and A stablethen all Pareto candidates of the game (PLQ) are obtained by solving the necessary conditions foroptimality of the associated (WSLQRP).

Proof. If A is stable, there exist constants a > 1 and σ > 0 such that ||eAt || ≤ ae−σt .

||x(t)|| ≤ ||eAt || ||x0||+∫ t

0||eA(t−τ)|| ||B|| ||u(τ)||dτ

≤ ae−σt ||x0||+ ||B||∫ t

0ρe−σ(t−τ) ||u(τ)|| dτ

≤ ae−σt ||x0||+a m ||B||∫ t

0e−σ(t−τ) dτ

=am ||B||

σ+ae−σ t

(||x0||−

m||B||σ

)18

We try to find an upper bound on the cost gi(x(t),u(t)) as follows:

gi(x(t),u(t)) = x′(t)Qix(t)+2x′(t)Siu(t)+u′(t)Riu(t)(from Cauchy-Schwarz inequality)

≤ ||x(t)|| ||Qix(t)||+2||x(t)|| ||Six(t)||+ ||u(t)|| ||Riu(t)||(with consistent matrix norm)

≤ ||Qi|| ||x(t)||2 +2||Si|| ||x(t)|| ||u(t)||+ ||Ri|| ||u(t)||2

≤

(||Qi||

(a m||B||

σ

)2

+2||Si||a m2||B||

σ+ ||Ri|| m2

)+[

2 ||Qi||a m||B||

σ+2 ||Si||m

](||x0||−

m||B||σ

)ae−σ t +(

||x0||−m||B||

σ

)2

||Qi||a2e−2σ t

Further, assuming that there exist positive constants b, q, r and s such that ||B|| < b, ||Qi|| < q,||Ri||< r and ||Si||< s for all i ∈ SN , the instantaneous cost gi(x(t),u(t)) is bounded as:

gi(x(t),u(t)) ≤ m2(

a2 b2 qσ2 +

2s a bσ

+ r)+2m

(a q b

σ+ s)(

||x0||−m bσ

)ae−σ t

+

(||x0||−

a bσ

)2

qa2e−2σt

= g0 +g1(||x0||)e−σ t +g2(||x0||)e−2σ t = gub(||x0||, t). (36)

We define a new game (PLQ) with player’s costs defined as:

Ji(x0,u(.)) =∫ ∞

0e−ρt (gi(x(t),u(t))−gub(||x0||, t))dt, i ∈ SN .

From straightforward calculations we have:

Ji(x0,u(.)) = Ji(x0,u(.))−(

g0

ρ+

g1(||x0||)ρ +σ

+g2(||x0||)ρ +2σ

), ∀i ∈ SN . (37)

We notice that the instantaneous undiscounted costs of the players in the game (PLQ) are nonpositivedue to (36). Further, the pair (A,B) is controllable. So, assumption (2) holds and by theorem 3.3 allPareto candidates of the game (PLQ) are obtained by solving the necessary conditions for optimalityof the associated (WSLQRP). From (37) and corollary 2.3 the Pareto optimal strategies for thegames (PLQ) and (PLQ) coincide. Intuitively, the Pareto surface (if exists) associated with (PLQ) is ashifted version of the Pareto surface associated with (PLQ).

Remark 9. By assumption 3 and corollary 3.3 we notice that if A is anti-stable, ρ > 2σmax(A)3 andu(.) ∈ LN

2,loc, then by theorem 3.3 all Pareto candidates of the game (PLQ) are obtained by solv-ing the necessary conditions for optimality of the associated (WSLQRP). Here, we do not assumecontrollability of the pair (A,B) and boundedness of ||u(t)||.Assumption 4. Either the pair (A,B) is controllable, A is stable and u(.) ∈ Ubnd or conditions givenin remark 9 hold true.

3σmax(A) represents the largest eigenvalue of A.

19

Let x(t) = e−ρt/2x(t), u(t) = e−ρt/2u(t), A = A− ρ2 I and Mα = ∑i∈SN αiMi , Qα = ∑i∈SN αiQi,

Sα = ∑i∈SN αiSi, Rα = ∑i∈SN αiRi,−→α ∈ PN . We specialize theorem 3.3 for the linear quadratic

case below by taking the assumption 4 to hold true.

Theorem 5.1. Let assumption 4 hold true. If (J1(x0, u∗), · · · ,JN(x0, u∗) is a Pareto candidate for theproblem (34,35) then there exists an −→α ∈ PN such that[

x∗′(t) u∗

′(t)]

Mα

[x∗(t)u∗(t)

]+ λ ′(t)Bu∗(t) ≤

[x∗

′(t) u

′(t)]

Mα

[x∗(t)u(t)

]+ λ ′(t)Bu(t), (38)

˙x(t) = Ax(t)+Bu(t), x(0) = x0, (39)˙λ (t) =−A′λ (t)−2Qα x(t)−2Sα u∗(t), lim

t→∞e−ρt/2λ (t) = 0. (40)

In case α is such that Rα > 0, the above equations can be equivalently rephrased as that everyPareto optimal control satisfies u∗(t) = −R−1

α

(Sα x(t)+B′λ (t)

), where λ (t) is the solution of the

set of linear differential equations:[˙x(t)˙λ (t)

]= Gα

[x(t)λ (t)

],

[x(0)

limt→∞ e−ρt/2λ (t)

]=

[x00

]. (41)

Proof. Follows directly from theorem 3.3.

Remark 10. Here Gα is a Hamiltonian matrix given by

Gα :=(

A−BR−1α Sα −BR−1

α B′

−(Qα −SαR−1

α Sα)

−(A−BR−1

α Sα)′) .

The solution of the above two point boundary value problem (41) depends on x0 and α . For agiven x0, (41) may or may not have a solution. The eigenvalues of the Hamiltonian matrix Gα aresymmetric w.r.t real and imaginary axis. We know that if there exists an n-dimensional stable graphsubspace [5, see section 2.7] for Gα then (41) has a solution for every initial state x0.

5.1 Fixed Initial StateIn this section we give additional properties of Pareto candidates (which includes Pareto solutions)due to linear quadratic nature of the game (PLQ). From the linearity of (41), if for some α and forboth initial states x0

0 and x10 (41) has a solution then for every initial state in the linear combination

of x00 and x1

0, (41) has a solution. This property leads to the conclusion that if there exists a Paretosolution for the initial states x0

0 and x10 (for some α), then there exists a Pareto solution for an initial

state that is a linear combination of x00 and x1

0 (with the same choice of α). Furthermore, independentof the assumption 4 the following properties hold true due to linearity of the game (PLQ):

Lemma 5.1. Assume u∗ is a Pareto optimal control for (34, 35). Then µu∗ is a Pareto optimalcontrol for (34, 35) with x(0) = µx0.

Proof. Refer [7, lemma 3.2]).

As a result of the above lemma, if for the initial state x0 = 0 there exists a Pareto solution differentfrom zero, then all points on the half-line connecting this point and zero are Pareto solutions too.We state this observation formally in the next corollary.

Corollary 5.2. Consider the cooperative game (34, 35) with x0 = 0. Assume u∗ is a Pareto optimalcontrol for this game yielding the Pareto solution J∗. Then for all µ ∈ R, µu∗ yields the Paretosolution µ2J∗.

20

5.2 Arbitrary Initial StateFrom theorem 5.1, if assumption 4 is satisfied then all Pareto candidates are obtained by solving theequation (41). In this section we consider the conditions under which (34, 35) has a Pareto candidatefor an arbitrary initial state. It is well known [24] that the solution of (41) is closely related to theexistence of the solution to the following algebraic Riccati equation (ARE):

A′X +XA− (XB+Sα)R−1α(B′X +Sα

)+Qα = 0 (ARE).

Since u(.) ∈ LN2,loc or Ubnd

4 we have∫ ∞

0u′(t)u(t)dt =

∫ ∞

0e−ρtu′(t)u(t)dt ≤

∫ ∞

0u′(t)u(t)dt ≤ c < ∞ with c > 0 being a constant.

Clearly u(.) ∈ LN2,loc. We define the linear mapping

L : u →∫ t

−∞eA(t−s)Bu(s)ds

and let L ∗ denote its adjoint operator. Using a result from [12, corollary 4.1.7], we state thefollowing proposition:

Proposition 5.1. Consider game problem (34, 35) with assumption 4 holding true (as a result A isstable) then we have the following assertions:

a) The operator Rα = Rα +S′αL +L ∗Sα +L ∗QαL is coercive if and only if Rα > 0 and the(ARE) has a stabilizing solution Xα (i.e., σ

(A−BR−1

α (B′Xα +Sα))⊂ C−).

b) If the operator Rα is coercive then the minimum of quadratic cost is

minu∈L2,loc

Jα(x0,u) = x′0Xαx0, ∀x0 ∈ Rn.

Lemma 5.2. Let assumption 4 hold true and Ri > 0, i ∈ SN . If the (ARE) has a solution for−→α 1,

−→α 2, · · · ,−→α k ∈ PN then (ARE) has a solution for all α ∈ K (−→α 1,−→α 2, · · · ,−→α k), where

K (−→α 1,−→α 2, · · · ,−→α k) :=

{−→α ∈ PN

∣∣−→α =k

∑i=1

κi−→α i, κi > 0, i = 1,2, · · · ,k

}.

Proof. From proposition 5.1.a, if the (ARE) has a solution for −→α i ∈ PN then the operator Rαi iscoercive i.e., ∃δi > 0 such that ⟨u,Rαiu⟩ ≥ δi||u||2LN

2,loc. For any −→α ∈PN with −→α = ∑k

i=1 κi−→α i, κi >

0, i= 1,2, · · · ,k we have Rα =∑ki=1 κiRαi . We notice that ⟨u,Rαu⟩=

⟨u,∑k

i=1 κiRαiu⟩=∑k

i=1 κi ⟨u,Rαiu⟩≥(∑k

i=1 κiδi)||u||2

LN2,loc

, as ∑ki=1 κiδi > 0, Rα is coercive. Again, from proposition 5.1.a the statement

of the lemma holds true.

Using the above lemma we state the following corollary about Pareto candidates (we do not givea proof as it is a direct consequence of lemma 5.2).

4If u(.) ∈ Ubnd we assume a sufficiently large M to avoid boundary optima.

21

Corollary 5.3. Assume Ri > 0, i ∈ SN and A be stable. Assume that (ARE) has a solution for−→α i ∈ PN , i = 1,2, · · · ,k. Then for any initial state x0 ∈ RN for all −→α ∈ K (−→α 1,

−→α 2, · · · ,−→α k),{(J1(u∗α),J2(u∗α), · · · ,JN(u∗α))} yield Pareto candidates. Here

u∗α(t) :=−R−1α(B′Xα +Sα

)e(A−BR−1

α (B′Xα+Sα))tx0,

where Xα solves the (ARE).

We discuss, in the remark given below, the conditions underwhich the necessary conditions aresufficient too, i.e., if the obtained Pareto candidates are indeed Pareto solutions.

Remark 11. a) In [6, see corollary 2.8 and theorem 2.9], with (A,B) stabilizable and u(.) ∈L+

2,s(x0; x), it is assumed that (ARE) has a stabilizing solution for all −→α =−→e i5, i= 1,2, · · · ,N.

This assumption leads to convexity of the player’s objectives. By invoking theorem 4.1 allPareto solutions are obtained by solving the associated (WSLQP). However, in corollary 5.3a more general result is obtained since it is not required that (ARE) has a stabilizing solutionfor every −→α =−→e i, i = 1,2, · · · ,N.

b) In corollary 5.3, the control space is either LN2,loc or Ubnd. We notice that by taking u(.) ∈

L+2,s(x0; x) the results obtained in [6] follow from corollary 5.3.

c) In assumption 4, if dominated discounting (i.e., remark 9) is considered, one could arrive ata more general result stated in corollary 5.3 relaxing the controllability condition on the pair(A,B). However, for dominated discounting, from assumption 3.c, A should be anti-stable.

5.3 The scalar caseThroughout this subsection we make a standing assumption that assumption 4 is satisfied. Notice,assming bi = 0, i = 1,2, · · · ,N, that controllability is trivially satisfied. Using lemma 5.1 someinteresting observations can be derived for the scalar case. Theorem 5.2 below states that if (34, 35)has a Pareto solution for all initial states, then it can be found using the weighting method.

Lemma 5.3. Consider the scalar system with Ri > 0, i ∈ SN . Let x0 = 0. If (34, 35) has a Paretooptimal control u∗(x0) then there exists an −→α ∈ PN such that min∑i∈SN αiJi subject to (35) has asolution for all initial states (including x0 = 0).

Proof. If assumption 4 holds true, from theorem 5.1 all Pareto candidates can be obtained us-ing the weighting method. Let x0 = 0 be fixed and

(−→α , x(t), λ (t), u∗(x0))

solve the two pointboundary value problem given by (41). Then from lemma 5.1 for the initial state µx0, µ ∈ R,(−→α ,µ x(t),µλ (t),µ u∗(x0)

)solves (41). In other words for this −→α , ∀x0 ∈ R (41) has a solution.

This implies that the optimization problem min∑i∈SN αiJi subject to (35) has a solution for all theinitial states x0.

Remark 12. a) As already noticed in the proof of lemma 5.3 it follows directly from lemma 5.1that, in case the scalar game has a Pareto solution for some initial state different from zero,the game (34, 35) has a Pareto solution for every initial state.

b) From the proof of lemma 5.3 we can in fact conclude the following result. If for x0 = 0 thereexists a Pareto solution and (41) has a solution with the choice of

−→α = (α1, · · · , αN) ∈ PN ,

5−→e i represents an N-dimensional vector with all zeros except the ith entry equal to 1.

22

then the scalar optimization problem min∑i∈SN αiJi subject to (35) has a solution for all initialstates x0 ∈ R. In other words all candidates obtained from theorem 5.1 are indeed Paretosolutions.

Theorem 5.2. Consider the scalar system with Ri > 0, i ∈ SN . Then for some x0 = 0/every x0(34, 35) has a Pareto optimal control u∗(x0) if and only if there exists −→α ∈ PN such that for everyx0 min∑i∈SN αiJi subject to (35) has a solution.

Proof. ⇒ In particular, it follows that if (34, 35) has a Pareto optimal solution for some x0 = 0 thenlemma 5.3 yields the advertized result.

⇐ Since Rα > 0, if there exists an −→α ∈ PN such that for every x0 min∑i∈SN αiJi has a solutionthen the associated (ARE) has a stabilizing solution. The control strategy thus obtained is unique.Further, this control is indeed Pareto optimal from theorem 5.1.

In other words, theorem 5.2 shows that to find all Pareto solutions of the game (34, 35), witharbitrary initial state (or for some initial state different from zero) one has to determine all −→α ∈PN for which (ARE) has a stabilizing solution. From remark 12.b and theorem 5.2 we have thefollowing algorithm to find all the Pareto solutions for a scalar game.

Algorithm 1 With assumption 4 holding true, consider the scalar system with Ri > 0, i ∈ SN .Let us define the index set

I(α) :={−→α ∈ PN

∣∣(ARE) has a stabilizing solution Xα}.

1. To find all Pareto solutions of the game (34, 35) with arbitrary initial state one has to deter-mine the set I(α). Then, the set of Pareto optimal controls is given by

u∗(t) =−R−1α(B′Xα +Sα

)e(A−BR−1

α (B′Xα+Sα))tx0 with −→α ∈ I(α), (42)

Where Kα solves the corresponding (ARE).

2. To find all Pareto solutions for which the game (34, 35) has a Pareto solution for a fixed initialstate x0 = 0 use step (1).

3. If x0 = 0 and I(α) is non empty, u∗ ≡ 0 is a Pareto optimal control. We know from corollary5.2 that if we find a Pareto solution different from zero, then all points on the half-line throughthis solution and zero are Pareto solutions too.

We consider the example from [5, example 6.2] to illustrate the usage of algorithm 1.

Example 3. Consider the situation in which there are two individuals who invest in a public stock ofknowledge. Let x(t) be the stock of knowledge at time t and ui(t) the investment of player i in publicknowledge at time t. Assume that the stock of knowledge evolves according to the accumulationequation

x(t) =−βx(t)+u1(t)+u2(t), x(0) = x0,

where β is the depreciation rate. We observe that assumption (4) holds true here. Assume that eachplayer derives quadratic utility from the consumption of the stock of knowledge and that the costof investment increases quadratically with the investment effort. That is, the cost function of bothplayers is given by

Ji =∫ ∞

0e−θ t {−qix2(t)+ riu2

i (t)}

dt.

23

Since the investment efforts are bounded and the system is controllable all Pareto solutions canbe obtained using the weighting method. It can be easily verified that the associated (ARE) has astabilizing solution if(

β +θ2

)2

− (αq1 +(1−α)q2)

(1

αr1+

1(1−α)r2

)> 0.

For the choice of model parameters β = 2, θ = 0.05, ri = qi = 1, i= 1,2 the above condition is writ-ten as α(1−α)> 0.2380,0 < α < 1. Applying the algorithm (1) we find I(α) = (0.3902, 0.6098)and all Pareto efficient controls are given by (42).

6 ConclusionsIn this paper, we studied the problem of existence of Pareto solutions of infinite horizon cooper-ative differential games. We assumed an open loop information structure. We considered nonau-tonomous and discounted autonomous systems for the analysis. Firstly, we gave a necessary andsufficient characterization of Pareto optimality which enables to view the problem of finding Paretocandidates as solving N constrained infinite horizon optimal control subproblems. We gave weakconditions under which all Pareto candidates can be obtained by solving a weighted sum opti-mal control problem. These weak conditions relate to conditions which extend the finite horizontransversality conditions to infinite horizons. We gave sufficient conditions under which all Paretosolutions are obtained by solving a weighted sum optimal control problem and provided examples.Later, the obtained results are used to analyze the regular indefinite infinite horizon linear quadraticdifferential game. We presented some properties of Pareto solutions due to the linear structure ofthe game. Assuming some weak conditions to hold true we showed that for the scalar case all Paretosolutions can be obtained by using the weighting method. We presented an algorithm to calculateall Pareto optimal solutions if the initial state differs from zero.

AcknowledgmentsThe authors like to thank Hans Schumacher for comments on a earlier draft which helped to improvethe readability of this paper.

References[1] S. Aseev and V. Kryazhimskiy. The Pontryagin maximum principle and transversality con-

ditions for a class of optimal control problems with infinite time horizons. SIAM Journal onControl and Optimization, 43(3):1094–1119, 2004.

[2] J. P. Aubin and F. H. Clarke. Shadow prices and duality for a class of optimal control problems.SIAM Journal on Control and Optimization, 17(5):567–586, 1979.

[3] A. Blaquiere, L. Juricek, and K. E. Wiese. Geometry of Pareto equilibria and maximumprinciple in n person differential games. Journal of Optimization Theory and Applications,38(1):223–243, 1972.

[4] S. S. L. Chang. General theory of optimal processes. SIAM Journal of Control, 4(1):46–55,1966.

24

[5] J. C. Engwerda. LQ dynamic Optimization and Differential Games. John Wiley and Sons,Chichester, 2005.

[6] J. C. Engwerda. The regular convex cooperative linear quadratic control problem. Automatica,44(9):2453–2457, 2008.

[7] J. C. Engwerda. Necessary and sufficient conditions for Pareto optimal solutions of coopera-tive differential games. SIAM Journal on Control and Optimization, 48(6):3859–3881, 2010.

[8] K. Fan and A. J. Hoffman. Systems of inequalities involving convex functions. AmericanMathematical Society Proceedings, 8(3):617–622, 1957.

[9] D. Grass, J. P. Caulkins, G. Feichtinger, G. Tragler, and D. A. Behrens. Optimal Control ofNonlinear Processes with Applications in Drugs, Corruption and Terror. Springer, 2008.

[10] H. Halkin. Necessary conditions for optimal control problems with infinite horizons. Econo-metrica, 42(2):267–272, 1974.

[11] P. Hartman. Ordinary Differential Equations. John Wiley & Sons, 1964.

[12] V. Ionescu, V. C. Oara, M. D. Weiss, and M. Weiss. Generalized Riccati Theory and RobustControl. A Popov Function Approach. Wiley: Chichester, New York, 1999.

[13] M. I. Kamien and N. L. Schwartz. Dynamic Optimization: The Calculus of Variations andOptimal Control in Economics and Management. Elsevier Science, 1991.

[14] G. Leitmann. Cooperative and Noncooperative Many Player Differential Games. SpringerVerlag, Berlin, 1974.

[15] G. Leitmann, S. Rocklin, and T. L. Vincent. A note on control space properties of cooperativegames. Journal of Optimization Theory and Applications, 9(2):379–390, 1972.

[16] P. Mitchel. On the transversality conditions in infinite horizon optimal problems. Economet-rica, 50(4):975–985, 1982.

[17] A. Seierstad. Necessary conditions for non-smooth, infinite-horizon, optimal control prob-lems. Journal of Optimization Theory and Applications, 103(1):201–229, 1999.

[18] A. Seierstad and K. Sydsaeter. Optimal Control Theory with Economic Applications. North-Holland, Amsterdam, 1986.

[19] C. P. Simon and L. Blume. Mathematics for Economists. Norton & Company: New York,1994.

[20] H. L. Stalford. Criteria for Pareto optimality in cooperative differential games. Journal ofOptimization Theory and Applications, 9(6):391–398, 1972.

[21] T. L. Vincent and G. Leitmann. Control space properties of cooperative games. Journal ofOptimization Theory and Applications, 6(2):91–104, 1970.

[22] T. A. Weber. An infinite-horizon maximum principle with bounds on the adjoint variable.Journal of Economic Dynamics and Control, 30(2):229–241, 2006.

25

[23] L. A. Zadeh. Optimality and non-scalar-valued performance criteria. IEEE Transactions onAutomatic Control, 8(1):59–60, 1963.

[24] K. Zhou, J. C. Doyle, and K. Glover. Robust and optimal control. Prentice-Hall, Inc., UpperSaddle River, NJ, USA, 1996.

26