Properties of the Horst Properties of the Horst Algorithm for the Algorithm for the

Multivariable Eigenvalue Multivariable Eigenvalue ProblemProblem

Michael SkalakNorthwestern University

Outline of ProblemOutline of Problem

nnm

i i 1

mmmm

m

m

AAA

AAA

AAA

A

21

22221

11211

with ii nnii RA

Given such that and a symmetric, positive definite matrix

mnn ,...,1

Multivariable Eigenvalue Problem• Find real scalars and a real column vector

such that

where is the identity matrix of size

and is partitioned into blocks

with

m ,...,1nRx

mix

xAx

i ,...,1,1

},...,{ ][][1

1 mnm

n IIdiag ][ inI in

ini Rx

TTm

T xxx ],...,[ 1

nRx

ExampleExampleGiven the symmetric and positive definite matrix , ,

the vector

is a solution, as

442.12638.4581.0

638.4367.10297.0

581.0297.0990.8

473.1645.3

162.2349.0

052.295.1

473.1162.2052.2

645.3349.0950.1

919.9821.3

821.3740.6

A

21 n 32 n2m

717.0684.0131.0994.0177.0 x

21 841.17405.12 xxAx

Statistical ApplicationStatistical ApplicationFind the maximum correlation coefficient of random variables, each of

size

Maximize

subject to

Hence the solution is the global maximum of for vectors in

, where is a ball of radius 1 centered at the origin in dimensions.

AxxT

mmini ,...,1

mnn BB ...1

AxxT

nBn

mixi ,...,11

Power MethodPower Method

The power method finds the eigenvector with the largest eigenvalue for the usual single-variate eigenvalue problem.

end

yx

y

Axy

kfor

k

kk

kk

kk

)(

)()1(

)()(

)()(

,...2,1

Horst AlgorithmHorst Algorithm

end

end

yx

y

xAy

mifor

kfor

ki

kik

i

ki

ki

m

j

kjij

ki

)(

)()1(

)()(

1

)()(

:

:

:

,...,1

,...2,1

Proven to converge monotonically by Chu and Watterson [SIAM J. Sci. Comput. (14), No. 5, pp. 1089-1106]

Finds the which maximizes x AxxT

ExampleExample

For that same matrix, consider the Horst algorithm with the

starting point

First iteration:

577.0577.0577.0707.0707.0)0( x

968.0252.0

511.3

398.3884.

)1(1

)1(1)2(

1

)1(1

)1(1

2

1

)0(11

)1(1

yx

y

xAyj

j

511.0548.0662.0

807.12

578.6059.7527.8

)1(2

)1(2)2(

2

)1(2

)1(2

2

1

)0(22

)1(2

yx

y

xAyj

j

Dependence on Initial ConditionsDependence on Initial Conditions

Convergence point can depend on initial conditions:

Like many other maximization algorithms, the Horst algorithm can converge to a local instead of global max.

885.0464.0023.0436.0900.0577.0577.0577.0707.0707.0)0( x

717.0684.0131.0994.0108.0267.0535.0802.0894.0447.0)0( x

414.31AxxT

284.30AxxT

ResultsResults

For any , can have at least as many convergent points

For any m, there can be at least convergence points, and as few as one.

In at least a nontrivial special case (two convergence points, ) the portion of the region which converges to the global max can not be arbitrarily small

in 1in

3

1

3m

3,2 nm

Number of Convergence PointsNumber of Convergence PointsThere exist 3 matrices, for all such that there exist

convergence points.

The block matrix, with meaning a matrix of size with convergence points

is symmetric, positive definite, and has convergence points

With a little manipulation, this proves that for any size there exist

matrices with at least convergence points.

m

13,2,1 inm i m

2

1

0

0

m

m

A

A

mA m

21mm

3

1

3m

m

Convergence to Global MaxConvergence to Global MaxSuppose there is some transformation on the matrix that can arbitrarily move eigenvectors arbitrarily. After the transformation, the matrix is rescaled so that the largest element remains constant.

fe

ed

c

bcba

321 dxfecdxedbdxcbadx

Axxd T

Case 1Case 1

The difference of the values between the local mins and the global max is bounded . Then the derivative must increase without bound. However, since all elements of the matrix are less than a constant, this cannot happen.

0

Case 2Case 2

The values of the local mins approach the global max as the vectors approach. Since one of the local mins is the global min, the function become closer and closer to constant, which cannot happen since the derivative is bounded below in at least one direction.