lec 18 unidirectional search.pdf
-
Upload
muhammad-bilal-junaid -
Category
Documents
-
view
252 -
download
2
Transcript of lec 18 unidirectional search.pdf
![Page 1: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/1.jpg)
Multi-variable Unconstrained Optimization:
Unidirectional Search
Dr. Nasir M Mirza
Optimization TechniquesOptimization Techniques
Email: [email protected]
![Page 2: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/2.jpg)
In this Lecture …• In this lecture we will discuss two important methods
that deal with Multi-variable Unconstrained Optimization:
• Multi – Dimensional Plots and
• Uni-directional Search
![Page 3: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/3.jpg)
Example 4.2The following function of two variables has distinct local minimum points and inflection point:
f = (x - 2)2 + (x - 2у2)2
Let us draw a mesh – grid graph in three dimensions using MATLAB.
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; mesh(X, Y, Z) %interpolatedaxis tight; hold onplot3(x,y,z,'.','MarkerSize',15)
![Page 4: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/4.jpg)
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phong
SURFACE PLOT using MATLAB
![Page 5: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/5.jpg)
Example 4.2 continued• Object function is: f = (x - 2)2+ (x - 2у2)2
⎟⎟⎠
⎞⎜⎜⎝
⎛
+−−+−
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂∂∂∂
=3
2
168444yxyyx
yfxf
vectorgradient
⎟⎟⎠
⎞⎜⎜⎝
⎛+−−−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂∂
∂∂∂
∂∂
= 2
2
22
2
2
2
488884
yxyy
yf
xyf
yxf
xf
H
32
22
168)2(8
444)2(2)2(2
yxyyxyyf
yxyxxxf
+−=−−=∂∂
−+−=−+−=∂∂
![Page 6: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/6.jpg)
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; contour(X,Y,Z, 3000)
Next step is to draw a contour graph and identify points of interest.
(Here we see (2, -1)T and (2, 1)T as important points.
CONTOUR PLOT using MATLAB
![Page 7: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/7.jpg)
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)
SURFACE PLOT using MATLAB
![Page 8: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/8.jpg)
Example 4.2 continuedTwo types of contour plots showing possible maximum or minimum points
![Page 9: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/9.jpg)
Example 4.2• Optimality condition (first order):
0168)2(8
0444)2(2)2(2
32
22
≡+−=−−=∂∂
≡−+−=−+−=∂∂
yxyyxyyf
yxyxxxf
Possible solutions (x, y) are following:
(1, 0) ; (2, -1); (2, 1)
Let us check with second optimality condition for each point:
For point (1, 0):
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=80
04H
Principal minors are:
A1 = a11= 4, A2 = det(A)= -32
It is then inflection point and Function value = 2.
![Page 10: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/10.jpg)
Example 4.2Let us check with second optimality condition for point (2, -1)
⎟⎟⎠
⎞⎜⎜⎝
⎛=
32884
HPrincipal minors are:
A1 = a11= 4, A2 = det(A)= 128 – 64 = 64
It is then minimum point and Function value = 0.
Let us check with second optimality condition for point (2, +1)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
32884
HPrincipal minors are:
A1 = a11= 4, A2 = det(A)= 128 – 64 = 64
It is then minimum point and Function value = 0.
![Page 11: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/11.jpg)
Example 4.3• The following function of two variables has only one minimum point
but two distinct local maximum points and two inflection points:
f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.
% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.05:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)
Let us first draw 3D graph:
![Page 12: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/12.jpg)
% matlab function for this graphclear all [X,Y] = meshgrid(-1.5:.1:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) - 12.*Y.^4. surf(X, Y, Z);colormap(jet);
Example 4.3
![Page 13: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/13.jpg)
Example 4.3
% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.01:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. contour(X, Y, Z, 200);
![Page 14: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/14.jpg)
Example 4.3• Object function is: f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.
⎟⎟⎠
⎞⎜⎜⎝
⎛
−−−−−−−−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂∂
∂∂∂
∂∂
=22
22
2
22
2
2
2
14448509669664814450
yxxyxyyx
yf
xyf
yxf
xf
H
32
23
4848506
4864850
yyxyxyf
xyyxxxf
−−+−=∂∂
−−−=∂∂
![Page 15: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/15.jpg)
Example 4.3• Optimality condition (first order):
Possible solutions (x, y) are following:
(-0.764, 0.764) ; (-0.677, -0.677); (0, 0); (0.764, -0.764) ; (0.677, 0.677);
Let us check with second optimality condition for each point:
For point (-0.764, 0.764):
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
62505062
H
Principal minors are:
A1 = a11= -62, A2 = det(A)= 1344
It is then maximum point and
Function value = 16.33
04848506
04864850
32
23
=−−+−=∂∂
=−−−=∂∂
yyxyxyf
xyyxxxf
![Page 16: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/16.jpg)
Example 4.3Let us check with second optimality condition for point (-0.677, -0.677)
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−
=38505038
HPrincipal minors are:
A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08
Let us check with second optimality condition for point (0, 0)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
506650
HPrincipal minors are:
A1 = a11= 50, A2 = det(A)= 2464
It is then minimum point and Function value = 0.
![Page 17: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/17.jpg)
Example 4.3Let us check with second optimality condition for point (+0.677, +0.677)
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−
=38505038
HPrincipal minors are:
A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08
Let us check with second optimality condition for point (0.764, -0.764)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
62505062
HPrincipal minors are:
A1 = a11= -62, A2 = det(A)= 1364
It is then maximum point and
Function value = 16.3
![Page 18: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/18.jpg)
Example 4.3
The graph shows these points on contour plot as maximum, minimum and inflection points;
Maximum
Maximum
Inflection
Inflection
Minimum
![Page 19: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/19.jpg)
Unidirectional Search Method• A unidirectional search is a one-dimensional search
performed by comparing function values only along a specific direction.
• Usually a unidirectional search is performed from a point x(t) in a specific direction s(t).
• Only points that lie on a line passing through x(t) in direction of s(t) are allowed.
• Any arbitrary point on that line can be expressed as
x(a) = x(t) + as(t).
• The parameter a is a scalar quantity and it is measure of distance of point x(a) from x(t).
![Page 20: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/20.jpg)
Unidirectional Search Method
Once optimal a is found the corresponding point is found.
• The above equation is a vector equation in multi-dimensional space.
• Hence when a is known the x(a) can be estimated.
• Any positive or negative value of a will create x(a) on the line for search.
• When a is zero, the current point x(t) is obtained.
• When we want minimum point, we convert objective function in terms of single variable a by putting all xi as
x(a) = x(t) + as(t).
![Page 21: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/21.jpg)
Example 3.2.1 Consider the objective function:
Minimize: f(x, y) = (x – 10)2 + (y – 10)2
• Step 1: First draw the contour plot of this function. The contour plot has lines where any two points on the line have the same function value.
• The optimal point is a minimum at (10, 10)T. The function value is zero at this point.
• Step 2: Let us say that a current point selected is x(t) = (2, 1)T. and the function value in a search direction s(t) = (2, 5)T from the current point. x(a) = x(t) + as(t) ; vector equation
x(a) = 2 + a2 ; y(a) = 1 + a5; scalar equations
![Page 22: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/22.jpg)
Example 3.2.1 • Now we find the equation for the straight line passing
through (2, 1)T and direction (2, 5)T as:(x – 2)/2 = (y – 1)/5 ; y – 1 = (5/2)(x – 2).
• Now let us find a point on this line as x = 3 then y = 3.5 and we can estimate a from above equations as a = 0.5
• If the point on line is with x = 9 and y = 18.5, then a = 3.5
• The function value for (3, 3.5)T is 91.25 • And at point (9, 18.5)T the value is 73.25.• Then at a = (3.5 + 0.5)/2 = 2, then x = 6, y = 11 and
function value is 17. So, approx. opt. is (6, 11)T.
![Page 23: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/23.jpg)
Example 3.3.1Consider the Himmelblau function: Minimize: f(x, y) = (x2 + y – 11)2 + (x + y2 –7)2
in the interval 0 < x < 5;
Step 1: This function is plotted as Figure. It appears from the plot that a minimum point is at (3, 2)T.
% matlab function for 3D plot clear all [X,Y] = meshgrid(0:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; surf(X, Y, Z);colormap(jet);
![Page 24: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/24.jpg)
% matlab function for contour plot clear all axis([0 5 0 5])hold on[X,Y] = meshgrid(-2:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; contour(X, Y, Z, 200)
Example 3.3.1: contour linesA contour line is a collection of points having identical function values.
The contour plot for the object function is shown here.
The continual decrease in the function value of the successive contour lines as they approach a point means that the point is minimum point.
![Page 25: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/25.jpg)
Example 3.3.1Step 1: We now choose an initial point x(0) = (1, 1)T
Size reduction parameter = (2, 2)T
Also choose e = 0.001
Step 2: We create a two dimensional hypercube (a square) around x(0) such that
X(1) = (0, 0)T , x(2) = (2, 0)T ,
X(3) = (0, 2)T , x(4) = (2, 2)T
Step 3: The function values at these points are:
f(0) = 106 ; f(1) = 170 ; f(2) = 74
f(3) = 90 ; f(4) = 26 The minimum is at (2, 2)T. Let us say this as x* =(2, 2)T
![Page 26: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/26.jpg)
Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (2, 2)T and got to step 2.
Step 2: We create a new square around this x(0) with Δ = 1 such that
X(1) = (1, 1)T , x(2) = (3, 1)T ,
X(3) = (1, 3)T , x(4) = (3, 3)T
Step 3: The function values at these points are:
f(0) = 26 ; f(1) = 106 ; f(2) = 10
f(3) = 58 ; f(4) = 26 The minimum is at (3, 1)T. The function value is 10.
Let us say this as x* =(3, 1)T
![Page 27: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/27.jpg)
Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (3, 1)T and go to step 2.
Step 2: We create a new square around this x(0) with Δ = 1 such that
X(1) = (2, 0)T , x(2) = (4, 0)T ,
X(3) = (2, 2)T , x(4) = (4, 2)T
Step 3: The function values at these points are:
f(0) = 10 ; f(1) = 74 ; f(2) = 34
f(3) = 26 ; f(4) = 50 The minimum is at (3, 1)T where function value is 10. It is same as x(0).
Let us say this as x* =(3, 1)T
![Page 28: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/28.jpg)
Example 3.3.1Step 4: Since x* is same as x(0), we now set x(0) = (3, 1)T and go to step 2 after reducing step size = 0.5
Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that
X(1) = (2.5, 0.5)T , x(2) = (3.5, 0.5)T
,
X(3) = (2.5, 1.5)T , x(4) = (3.5, 1.5)T
Step 3: The function values at these points are computed and minimum is at x(4) = (3.5, 1.5)T
And function value is 9.125.
The minimum is at (3.5, 1.5)T
Let us say this as x* =(3.5, 1.5)T
![Page 29: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/29.jpg)
Example 3.3.1Step 4: Since x* is not same as x(0), we now set x(0) = (3.5, 1.5)T
and go to step 2 with step size = 0.5
Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that
X(1) = (3, 1)T , x(2) = (4, 1)T ,
X(3) = (3, 2)T , x(4) = (4, 2)T
Step 3: The function values at these points are
f(0) =9.125; f(1) = 10 ;
f(2) = 40; f(3) = 0; f(4) = 50
the minimum is f(3) = 0.0
The minimum is at (3, 2)T
This is the best minimum so far as f=0.
![Page 30: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/30.jpg)
Example 3.3.1It is important to see that although we have found a minimum point but the algorithm does not terminate at this step. We may continue.
It is clear that the convergence depends upon the initial cube size and location and the chosen size reduction parameter.
Small size may lead to premature convergence and large may never lead to convergence.
The minimum is at (3, 2)T
This is the best minimum so far as f=0.
![Page 31: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/31.jpg)
Example: Suppose f(x, y) = 2xy + 2x – x2 – 2y2
Using the steepest ascent method to find the next point if we are moving from point (-1, 1).
yxyfxy
xf 42222 −=
∂∂
−+=∂∂
⎥⎦
⎤⎢⎣
⎡−
=⎥⎦
⎤⎢⎣
⎡−−
−−+=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−∂∂
−∂∂
=∇−6
6)1(4)1(2
)1(22)1(2
)1,1(
)1,1(),1,1(At
yfxf
f
)6161()(Let hh,-fhg −+=
Next step is to find h that maximize g(h)
![Page 32: lec 18 unidirectional search.pdf](https://reader031.fdocuments.in/reader031/viewer/2022020118/577cb9af1a28aba7118d835d/html5/thumbnails/32.jpg)
2.0036072yields0)('Setting
180727...)6161()(222)(
2
22
=⇒=−=
−+−==−+=
−−+=
hhhg
hhhh,-fhgyxxxyxf
If h = 0.2 maximizes g(h), then x = -1+6(0.2) = 0.2 and y = 1-6(0.2) = -0.2 would maximize f(x, y).
So moving along the direction of gradient from point (-1, 1), we would reach the optimum point (which is our next point) at (0.2, -0.2).