Nelder Mead Search Algorithm

Presented By: Ashish Khetan IIT Guwahati, India

Supervisor: Prof. Dr. Ian Cloete President IU in Germany

2

Numerical method for minimizing an objective function in a multi dimensional space

Is it like the linear programming ? Answer is NO !!! Only applicable to unconstrained problems. Other names – Downhill simplex method or

amoeba method.

What does algorithm do ?

3

Simplex◦ Affine space ◦ Convex hull ◦ Polytope

Prerequisites to learn the algorithm

4

An affine space is what is left of a vector space after you've forgotten which point is the origin.

An affine subspace of a vector space V is a subset closed under affine combinations of vectors in the space. For example, the set

is an affine space, where {vi}i is a family of vectors in V.

the sum of the coefficients is 1

Affine space

5

The convex hull for a set of points X in a real vector space V is the minimal convex set containing X.

For planar objects, it can be easily visualized by imagining an elastic band stretched open to encompass the given object; when released, it will assume the shape of the required convex hull.

Same concept can be extended to multi-dimensional space.

Convex hull

6

Polytope means, first, the generalization to any dimension of polygon in two dimensions, polyhedron in three dimensions, and polychoron in four dimensions.

One special kind of polytope is a convex polytope, which is the convex hull of a finite set of points.

A convex polytope can also be represented as the intersection of half-spaces. This intersection can be written as the matrix inequality:

Polytope

7

where A is an m by n matrix, m being the number of bounding half-spaces and n being the number of dimensions of the affine space Rn in which the polytope is contained; and b is an m by 1 column vector.

Polytope continued….

8

A polytope of a N+1 vertices in N dimensions.

A line segment on a line. A triangle on a plane. A tetrahedron in three dimensional space

and so forth.

Simplex

9

For a function or system withN variables, algorithm starts with N+1 points in a N dimensional space, defining an initial simplex.

Take a random initial starting point Po and then the other N points can be taken by -

Pi = Po + λei where ei are N unit vectors. λ is a constant, and its value depends upon

the problem’s characteristic length scale. Different λ’s can be chosen for each vector

direction.

Core of the algorithm

10

Calculate the function value(for a system calculate the value of objective function) at each of the vertices of the simplex.

Sort the points in order of the value of the function at that point.

Replace the worst point with a point reflected through the centroid of the remaining N points.

If this point is better than the best current point, then we can try stretching out/in along this line.

How the algorithm works ?

11

If this new point is not much better than the previous value, then we are steeping across a valley, so we shrink the simplex towards the best point.

It may get stuck in a rut. Then restart the algorithm with a new simplex starting at the current bets value.

Many variations exist depending upon the actual nature of the problem being solved.

Continued……

12

1. Sort the simplex vertices according to the function value at that point.

2. Compute the centroid Xo using all points except Xn+1 , the worst point.

3. Compute a reflection

Now there are three possibilities.

Steps

13

1. , use Xr and reject Xn+1 , go to step 1.

2. , then compute Xe = Xo + ρ(Xr - Xo ) (expansion)

if , then use Xe else use

Xr ,

reject Xn+1 , go to step 1.

Continue….

14

3. (contraction) If , then use Xc and reject Xn+1 ,

go to step 1. Else ….. Shrink step- compute new N vertices

keeping only the best one, X1.

,

go to step 1.

Continue…..

15

Reflection coefficient – α =1. Expansion coefficient – ρ =2. Contraction coefficient – γ = ½. Shrink coefficient – σ = ½.

Standard values of the algorithm parametrs

16

It can find the optimum value of any numbers of parameters involved in any system.

Optimum value of parameters means value of parameters while the value of objective function is minimum.

It has nothing to do with the system, what all it does is iteratively runs the system with different set of values of the parameters and eventually terminates at the minimum value of the objective function providing the optimum value of parameters.

Upside of the algorithm

17

It can find only local minimums. The local minima that the algorithm finds

entirely depends upon the initial simplex starting point.

To find the different local minimums, algorithm must be started with the different and appropriate initial simplex guess.

For a black box system where there is no knowledge of number of minimums, only hit and trial method works to find all the local minimums.

Downside of the algorithm

18

http://math.fullerton.edu/mathews/n2003/neldermead/NelderMeadMod/Links/NelderMeadMod_lnk_5.html

The above link provides the best illustrative example of this algorithm.

Illustrative example

19

In the same folder find a file Nelder Mead.java, it runs over a N dimensional algebraic function and takes function as a input from user and finds it’s one of the local minimum depending upon the initial simplex starting point guess provided by the user.

The same code can be modified to run over a system and to find the optimum value of parameters for a given objective function.

JAVA code

20

Numerical Recipes in C , The Art of Scientific Computing, Second Edition.

http://en.wikipedia.org/wiki/Affine_space http://en.wikipedia.org/wiki/Convex_hull http://en.wikipedia.org/wiki/Polytope

http://en.wikipedia.org/wiki/Nelder-Mead_method

References

21

Thank You