Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical...
Transcript of Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical...
![Page 1: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/1.jpg)
Introduction to Numerical Optimization
Biostatistics 615/815Lecture 14Lecture 14
![Page 2: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/2.jpg)
Course is More Than Half Done!
If you have comments…… they are very welcomey y• Lectures• Lecture notes• Weekly Homework • Midterm• Content
![Page 3: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/3.jpg)
Last Lecture
Computer generated “random” numbers
Linear congruential generators• Improvements through shuffling summingImprovements through shuffling, summing
Importance of using validated generatorsImportance of using validated generators• Beware of problems with the default rand()
functionu c o
![Page 4: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/4.jpg)
Today …
Root finding
Minimization for functions of one variable
Ideas:• Li it• Limits on accuracy• Local approximations
![Page 5: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/5.jpg)
Numerical Optimization
Consider some function f(x)• e.g. Likelihood for some model …
Find the value of x for which f takes a maximum or minimum valuemaximum or minimum value
Maximization and minimization are equivalentMaximization and minimization are equivalent• Replace f(x) with –f(x)
![Page 6: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/6.jpg)
Algorithmic Objectives
Solve problem…• Conserve CPU time• Conserve memory
Most often, the CPU time is dominated by the cost of evaluating f(x)• Minimize the number of evaluations
![Page 7: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/7.jpg)
The Minimization Problem
![Page 8: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/8.jpg)
Specific Objectives
Finding global minimum• The lowest possible value of the function• Extremely hard problem
Finding local minimum• Smallest value within finite neighborhood
![Page 9: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/9.jpg)
Typical Quality Checks
When solving an optimization problem it is good practice to check the quality of the solution
Try different starting values …
Perturb solution and repeat …
![Page 10: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/10.jpg)
A Quick Detour
Consider the problem of finding zeros for f(x)
Assume that you know:• Point a where f(a) is positive• P i t b h f(b) i ti• Point b where f(b) is negative• f(x) is continuous between a and b
How would you proceed to find x such that f(x)=0?
![Page 11: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/11.jpg)
Root Finding in Cdouble zero(double (*func)(double), double lo, double hi, double e)
{while (true)
{double d = hi – lo;double d = hi – lo;double point = lo + d * 0.5;double fpoint = (*func)(point);
if (fpoint < 0.0)( po t . ){ d = lo – point; lo = point; }
else{ d = point – hi; hi = point; }
if (fabs(d) < e || fpoint == 0.0) return point;
}}
![Page 12: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/12.jpg)
Improvements to Root Finding
Consider the following approximation:
abafbfaxafxf
−−
−+=)()()()()(*
Select new trial point such that f*(x) is zero.
![Page 13: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/13.jpg)
Improved Root Finding in Cdouble zero (double (*func)(double), double lo, double hi, double e)
{double flo = (*func)(lo);double fhi = (*func)(hi);while (1)while (1)
{double d = hi – lo;double point = lo + d * flo / (flo – fhi);double fpoint = (*func)(point);doub e po t ( u c)(po t);
if (fpoint < 0.0){ d = lo – point; lo = point; flo = fpoint; }
else{ d = point – hi; hi = point; fhi = fpoint; }
if (fabs(d) < e || fpoint == 0.0) return point;
}}}
![Page 14: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/14.jpg)
Performance Comparison
Find the zero for sin(x) • In the interval -π/4 to π/2• Accuracy parameter set to 10-5
Bi ti th d d 17 ll t i ( )Bisection method used 17 calls to sin(x)
Approximation used 5 calls to sin(x)
![Page 15: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/15.jpg)
Program That Uses Root Findingdouble zero (double (*func)(double), double lo, double hi, double e);
double my_function(double x){t (4* 3)return (4*x – 3);
}
int main(int argc, char ** argv){{double solution = zero(my_function, -5, +5, 1e-5);
printf(“Zero for my function is %.3f at %.3f\n”,f ti ( l ti ) l ti )my_function(solution), solution);
}
![Page 16: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/16.jpg)
Notes on Root Finding
The 2nd method we implemented is the False Position Method
In the bisection method, the bracketing i t l i h l d t h tinterval is halved at each step
For well-behaved functions, the False Position Method will converge faster, but there is no performance guaranteethere is no performance guarantee
![Page 17: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/17.jpg)
Questions on Root Finding
What care is required in setting precision?
How to set starting brackets for minimum?• If the function was monotonic?If the function was monotonic?• If there is a specific target interval?
What would happen for a function such as f(x) = 1 / (x – c)f(x) 1 / (x c)
![Page 18: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/18.jpg)
Back to Numerical Optimization
Consider some function f(x)• e.g. Likelihood for some model …
Find the value of x for which f takes a maximum or minimum valuemaximum or minimum value
Maximization and minimization are equivalentMaximization and minimization are equivalent• Replace f(x) with –f(x)
![Page 19: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/19.jpg)
Notes from Root Finding
Introduces two useful ideas that we’ll apply to function minimization
BracketingBracketing• Keep track of interval containing solution
Accuracy• Recognize that solution has limited precisionRecognize that solution has limited precision
![Page 20: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/20.jpg)
Note on Accuracy
When estimating minima and bracketing intervals, floating point accuracy must be considered
In general, if the machine precision is εthe achievable accuracy is no more than ysqrt(ε)
![Page 21: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/21.jpg)
Note on Accuracy II
The error results from the second term in the Taylor approximation:
22
1 ))(()()( bxbfbfxf −′′+≈
For functions where higher order terms are i t t ld b limportant, accuracy could be even lower. • For example, the minimum for f(x) = 1 + x4 is only
estimated to about ε1/4
![Page 22: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/22.jpg)
Outline of Minimization Strategy
Part I• Bracket minimum
Part II• Successively tighten bracketing intervalSuccessively tighten bracketing interval
![Page 23: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/23.jpg)
Detailed Minimization Strategy
Find 3 points such that • a < b < c• f(b) < f(a) and f(b) < f(c)
Then search for minimum by• Selecting trial point in intervalSelecting trial point in interval• Keep minimum and flanking points
![Page 24: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/24.jpg)
Minimization after Bracketing
1 2
4
3
5
6
![Page 25: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/25.jpg)
Part I: Part I: Finding a Bracketing Interval
Consider two points• a, b• f(a) > f(b)
Take successively larger steps beyond b until function starts increasing
![Page 26: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/26.jpg)
Bracketing in C##define SCALE 1.618
void bracket (double (*f)(double), double* a, double* b, double* c){double fa = (*f)( *a);double fa = (*f)( *a);double fb = (*f)( *b);double fc = (*f)( *c = *b + SCALE * (*b - *a) );
while (fb > fc)e ( b c){*a = *b; fa = fb;*b = *c; fb = fc;*c = *b + SCALE * (*b - *a);fc = (*f) (*c);}
}
![Page 27: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/27.jpg)
Bracketing in C++##define SCALE 1.618
void bracket (double (*f)(double), double & a, double & b, double & c){double fa = (*f)(a);double fa = (*f)(a);double fb = (*f)(b);double fc = (*f)(c = b + SCALE * (b - a) );
while (fb > fc)e ( b c){a = b; fa = fb;b = c; fb = fc;c = b + SCALE * (b - a);fc = (*f) (c);}
}
![Page 28: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/28.jpg)
Part II:Part II:Finding Minimum after Bracketing
Given 3 points such that • a < b < c• f(b) < f(a) and f(b) < f(c)
How do we select new trial point?
![Page 29: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/29.jpg)
Consider …
BA B C
What is the best location for a new point X?
![Page 30: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/30.jpg)
Consider …
BA B CX
We want to minimize the size of the next h i t l hi h ill b ithsearch interval which will be either
from A to X or from B to C
![Page 31: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/31.jpg)
Formulae …
acabw
−−
=
acbxz
−−
=
length have willSegments
soypossibilitcaseworstminimizewant toWe
or 1 zww +−
so...y possibilitcaseworst minimize want toWe
![Page 32: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/32.jpg)
Effectively …ii lTh
21iscaseoptimal The
−= wz
1=
−w
wz
givesThis
38197.05-3w
gives This
==2
![Page 33: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/33.jpg)
Golden Search
![Page 34: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/34.jpg)
The Golden Ratio
A B C
Bracketing Triplet
A B C
![Page 35: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/35.jpg)
The Golden Ratio
A B CX
New Point
A B CX
0.38196 0.38196
The number 0.38196 is related to the golden mean studied by Pythagoras
![Page 36: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/36.jpg)
The Golden Ratio
B
New Bracketing Triplet
A B X
0.38196
Alternative New Bracketing Triplet
B X C
0.38196
![Page 37: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/37.jpg)
Golden Search
Reduces bracketing by ~40% after each function evaluation
Performance is independent of the functionPerformance is independent of the function that is being minimized
In many cases, better schemes are available Any ideas?available… Any ideas?
![Page 38: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/38.jpg)
Golden Step##define GOLD 0.38196#define ZEPS 1e-10
double golden step (double a double b double c)double golden_step (double a, double b, double c){double mid = (a + c) * 0.5;
if (b > mid)return GOLD * (a - b);
elsereturn GOLD * (c - b);
}
![Page 39: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/39.jpg)
Golden SearchGolden Searchdouble golden_search(double (*func)(double),
double a, double b, double c, double e){d bl fb (*f )(b)double fb = (*func)(b);
while ( fabs(c - a) > fabs(b * e) + ZEPS){d bl b + ld t ( b )double x = b + golden_step(a, b, c);double fx = (*func)(x);
if (fx < fb) {{if (x > b) { a = b; } else { c = b; }b = x; fb = fx;}
lelseif (x < b) { a = x; } else { c = x; }
}return b;}}
![Page 40: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/40.jpg)
Further Improvements
As with root finding, performance can improve substantially when a local approximation is used …
However, a linear approximation won't do in this case!
![Page 41: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/41.jpg)
Approximating The Function
![Page 42: Introduction to Numerical Optimizationcsg.sph.umich.edu/abecasis/class/2008/615.14.pdfNumerical Optimization zConsider some function f(x) • e.g. Likelihood for some model … zFind](https://reader035.fdocuments.in/reader035/viewer/2022071215/60457752fcdaa142844522ad/html5/thumbnails/42.jpg)
Recommended Reading
Numerical Recipes in C (or C++)• Press, Teukolsky, Vetterling, Flannery• Ch 10 0 10 2• Chapters 10.0 – 10.2
Excellent resource for scientific computingExcellent resource for scientific computing
O li tOnline at• http://www.numerical-recipes.com/• http://www library cornell edu/nr/• http://www.library.cornell.edu/nr/