Parameter Estimation in Stochastic Differential Equations by Continuous Optimization

Gerhard-Wilhelm Weber *

Nüket Erbil, Ceren Can, Vefa Gafarova, Azer Kerimov

Institute of Applied Mathematics Middle East Technical University, Ankara, Turkey

Pakize Taylan Dept. Mathematics, Dicle University, Diyarbakır, Turkey

Parameter Estimation in

Stochastic Differential Equations

by Continuous Optimization

* Faculty of Economics, Management and Law, University of Siegen, GermanyCenter for Research on Optimization and Control, University of Aveiro, Portugal

Universiti Teknologi Malaysia, Skudai, Malaysia

5th International Summer School

Achievements and Applications of Contemporary Informatics,

Mathematics and Physics

National University of Technology of the Ukraine

Kiev, Ukraine, August 3-15, 2010

http://www.ntu-kpi.kiev.ua/

http://ssa.ntu-kpi.kiev.ua/

• Stochastic Differential Equations

• Parameter Estimation

• Various Statistical Models

• C-MARS

• Accuracy vs. Stability

• Tikhonov Regularization

• Conic Quadratic Programming

• Nonlinear Regression

• Portfolio Optimization

• Outlook and Conclusion

Outline

Stock Markets

http://en.wikipedia.org/wiki/File:Photos_NewYork1_032.jpg

http://en.wikipedia.org/wiki/File:Paternoster_Square.jpg

http://en.wikipedia.org/wiki/File:Shanghai_Stock_Exchange_Building.jpg

http://en.wikipedia.org/wiki/File:IE_Real_SandP_Prices,_Earnings,_and_Dividends_1871-2006.png

http://en.wikipedia.org/wiki/File:Boerse-ffm021.jpg

drift and diffusion term

Wiener process

( , ) ( , )t t t tdX a X t dt b X t dW

(0, ) ( [0, ])tW N t t T



Wiener process


(0, ) ( [0, ])tW N t t T


Ex.: price, wealth, interest rate, volatility

processes

Input vector and output variable Y ;

linear regression :

which minimizes

1 2, ,...,T

mX X X X

1 0

1

( ,..., ) ,m

m j j

j

Y E Y X X X

0 1, ,...,T

m

2

1

:N

T

i i

i

RSS y x

1ˆ ,T TX X X y

12ˆCov( ) Tβ X X

Regression

are estimated by a smoothing on a single coordinate.

Standard convention .

• Backfitting algorithm (Gauss-Seidel)

it “cycles” and iterates.

0ˆ ,i i kj

j

ik

k

r y f x

jf

: 0j ijE f x

Generalized Additive Models

1 2 0

1

, ,...,i i i i m ij j

m

j

E x fx x xY

• Given data

• penalized residual sum of squares

• New estimation methods for additive model with CQP :

2

2''

0 0

1 1 1

( , ,..., ) : ( ) ( )

bN m m

1 m i j ij j j j

i j j a

PRSS f f y f x f t dtjμ

0.j

( , ) ( = 1,2,..., ),i iy x i N


splines:

By discretizing, we get

0, ,

2

2

0

1 1

2''

min

subject to ( ) , 0,

( ) ( 1,2,..., ),

t β f

N m

i j ij

i= j

j j j j

t

y β f x t t

f t dt M j m

1

( ) ( ).jd

j j

j l l

l

f x h x

0, ,

2 2

0 2

2

0 2

min

subject to ( , ) , 0,

( , ) ( 1,..., ).

t β f

j j

t

W t t

V M j m


MARS

x

y

+( , )=[ ( )]c x x( , )=[ ( )]-c x x

x

y

+( , )=[ ( )]c x x( , )=[ ( )]-c x x r egression w ith

Tradeoff between both accuracy and complexity.

1 2

1 2

1 2 1 2

( ) : | 1, 2,...,

: ( , ,..., )

( , )

: , , 0,1

Km

m

j m

m T

m m m

V m j K

t t tt =

where

1 2

, ( ) : ( )m m m m

r s m m r sD t tt t

max

1 2

222 2

,

1 1 1, ( )( , )

: ( ) ( )MN

m m

i i m r s m

i m r sr s V m

PRSS y f D dmx t tμ

C-MARS

Tikhonov regularization:

2 2

22( )PRSS y d L

Conic quadratic programming:

,

2

2

subject to

min ,

( ) ,

tt

td y

ML

2L

2( )y d

C-MARS


Wiener process


(0, ) ( [0, ])tW N t t T

Stochastic Differential Equations Revisited

Ex.: price, wealth, interest rate, volatility,

processes


Wiener process


(0, ) ( [0, ])tW N t t T

bioinformatics, biotechnology

(fermentation, population dynamics)

Universiti Teknologi Malaysia

Ex.:



Wiener process


(0, ) ( [0, ])tW N t t T


Ex.: price, wealth, interest rate, volatility,

processes

Milstein Scheme :

and, based on our finitely many data:

2

1 1 1 1 1

1ˆ ˆ ˆ ˆ ˆ( , )( ) ( , )( ) ( )( , ) ( ) ( )2

j j j j j j j j j j j j j j j jX X a X t t t b X t W W b b X t W W t t

2( )( , ) ( , ) 1 2( )( , ) 1 .

j j

j j j j j j j

j j

W WX a X t b X t b b X t

h h


• step length

• (independent),

•

1 :j j j jh t t t

1

1

, if 1,2,..., 1

:

, if

j

j j

j

N N

N

X Xj N

hX

X Xj N

h

jW Var( )j jW t

21( , ) ( , ) ( )( , ) 1

2

j

j j j j j j j j

j

ZX a X t b X t b b X t Z

h

(0, ),tW N t

, (0,1)j j j jW Z t Z N


• More simple form:

where

• Our problem:

is a vector which comprises a subset of all the parameters.

2

: ( , ) , : ( , ),

: , : 1 2 1 .

j j j j j j

j j j j j

G a X t H b X t

c Z h d Z

y

( ) ,j j j j j j jX G H c H H d

2

21

min ( ( ) )N

j j j j j j jy

j

X G H c H H d


where

• k th order base spline : a polynomial of degree k − 1, with knots, say

2 2

0 , 0 ,

1 1 1

2 2

0 , 0 ,

1 1 1

2 2

0 , 0 ,

1 1 1

( , ) ( ) ( )

( , ) ( ) ( )

( , ) ( ) ( )

gp

hr

fs

d

l l

j j j p j p p p j p

p p l

dm m

j j j j j r j r r r j r

r r m

dn n

j j j j j s j s s s j s

s s n

G a X t f U B U

H c b X t c g U C U

F d b b X t d h U D U

,1 ,2, : , ;j j j j jU U U X t

,kB ,x

1

,1

, , 1 1, 1

1 1

1,( )

0, otherwise

( ) ( ) ( )k

k k k

k k

x x xB x

x x x xB x B x B x

x x x x


• penalized sum of squares PRRS

• (smoothing parameters),

• large values of yield smoother curves,

smaller ones allow more fluctuation

2

1

22 2 2

0 , 0 , 0 ,

1 1 1 1 1 1 1

( ) ( ) ( )

h fgp sr

N

j j j j j j

j

d ddNl l m m n n

j p p j p r r j r s s j s

j p l r m s n

X G H c F d

X B U C U D U

22 2

1 1

2 22 2

1 1

( , , ) : ( )

( ) ( )

N

j j j j j j p p p p

j p

r r r r s s s s

r s

PRSS f g h X G H c F d f U dU

g U dU h U dU

, , 0p r s

, ,p r s

( , , )

b

a

p r s


• Then,

• Furthermore,

1 2

0 1 2

1 2

0 1 2

1 2

0 1 2

,

, , , , ,..., ( 1,2),

, , , , ,..., ( 1

, ,

, 2),

, , , , ,..., ( 1,2).

gp

hr

fs

TT dT T

p p p p

TTdT T

r r r r

TTdT T

s

T T

s

T

s s

T

p

r

s

22

21

.N

j j

j

X A X A 1 2

1 2

, ,...,

, ,...,

TT T T

N

T

N

A A A A

X X X X

12 2

1,

1

21

1 1

( ) ( ) ( )

( ) .

gp

b N

p p p p jp j p jp

ja

dNl l

p p jP j

j l

f U dU f U U U

B U u


• If

where

2 2 22 2 2 2

2 2 22 1 1 1

( , , ) B C D

p p p r r r s s s

p r s

PRSS f g h X A A A A

2: :p r s

222

22

( , , ) ,PRSS f g h X A L

, , .T

T T T

1

2

1

2

1

2

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0: ,

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

B

B

C

C

D

D

A

A

AL

A

A

A


22

22

min X A Lμ Tikhonov regularization

,

2

2

subject to

min ,

,

tt

A X t

L M


Conic quadratic programming

Interior Point Methods


,

6( 1)6( 1)

1 6( 1) 1

min

0subject to : ,

1 0 0

00: ,

0 0

,

t

N

T

m

NN

T

m

N N

t

tA X

t

M

L L

L

1 1 2 2 2

1 2 1 1 2: ( , ,..., ) | ...N T N

N N+ NL x x x x x x x xR

6( 1) 1

1 6( 1) 2

6( 1)

1 2

1

1 2

max ( ,0) 0 ,

10 1 0 0subject to ,

00 0

, N

T T

N

T T

N N

T Tmm m

N

X M

A

L L

Ldual problem

primal problem

is a primal dual optimal solution if and only if 1 2( , , , , , )t

6( 1)6( 1)

6( 1)

1 2

1 2

1 6( 1) 1

1 2

1 6( 1) 1

0: ,

1 0 0

00:

0 0

10 1 0 0

00 0

0, 0

,

, .

N

T

m

NN

T

m

T T

N N

T Tmm m

T T

N N

N N

tA X

t

M

A

L L

L L

L

L



Ex.:

nonlinear regression

, , , ,t t t t t tdX t X Z dt t X Z dW .

( ) + ,T T

t t t t t t t t tdV V dt cr dr t V dWμ σ

,t t t t td R r dt rr dWτα

min ( ) ( ) ( )

Tf F F

1( ) : ( ),..., ( )T

NF f f

2

,

1

2

1

min

:

N

j j

j

N

j

j

f d g x

f

Nonlinear Regression

• Gauss-Newton method :

• Levenberg-Marquardt method :

( ) ( ) ( ) ( )T qF F F F

( ) ( ) I ( ) ( )T

p qF F F F

0

1 :k k kq


,

2

2

min ,

subject to ( ) ( ) I ( ) ( ) , 0,

|| ||

t

T

p

qt

F F F F

qL

q t t

M


interior point methods

alternative solution

conic quadratic programming

max utility ! or

min costs !

martingale method:

Optimization Problem

Representation Problem

or stochastic control

Portfolio Optimization

max utility ! or

min costs !

martingale method:

Optimization Problem

Representation Problem

or stochastic control

Parameter Estimation

Portfolio Optimization

Aster, A., Borchers, B., and Thurber, C., Parameter Estimation and Inverse Problems, Academic Press, 2004.

Boyd, S., and Vandenberghe, L., Convex Optimization, Cambridge University Press, 2004.

Buja, A., Hastie, T., and Tibshirani, R., Linear smoothers and additive models, The Ann. Stat. 17, 2 (1989)

453-510.

Fox, J., Nonparametric regression, Appendix to an R and S-Plus Companion to Applied Regression,

Sage Publications, 2002.

Friedman, J.H., Multivariate adaptive regression splines, Annals of Statistics 19, 1 (1991) 1-141.

Friedman, J.H., and Stuetzle, W., Projection pursuit regression, J. Amer. Statist Assoc. 76 (1981) 817-823.

Hastie, T., and Tibshirani, R., Generalized additive models, Statist. Science 1, 3 (1986) 297-310.

Hastie, T., and Tibshirani, R., Generalized additive models: some applications, J. Amer. Statist. Assoc.

82, 398 (1987) 371-386.

Hastie, T., Tibshirani, R., and Friedman, J.H., The Element of Statistical Learning, Springer, 2001.

Hastie, T.J., and Tibshirani, R.J., Generalized Additive Models, New York, Chapman and Hall, 1990.

Kloeden, P.E, Platen, E., and Schurz, H., Numerical Solution of SDE Through Computer Experiments,

Springer Verlag, New York, 1994.

Korn, R., and Korn, E., Options Pricing and Portfolio Optimization: Modern Methods of Financial Mathematics,

Oxford University Press, 2001.

Nash, G., and Sofer, A., Linear and Nonlinear Programming, McGraw-Hill, New York, 1996.

Nemirovski, A., Lectures on modern convex optimization, Israel Institute of Technology (2002).

References

Nemirovski, A., Modern Convex Optimization, lecture notes, Israel Institute of Technology (2005).

Nesterov, Y.E , and Nemirovskii, A.S., Interior Point Methods in Convex Programming, SIAM, 1993.

Önalan, Ö., Martingale measures for NIG Lévy processes with applications to mathematical finance,

presentation in: Advanced Mathematical Methods for Finance, Side, Antalya, Turkey, April 26-29, 2006.

Taylan, P., Weber G.-W., and Kropat, E., Approximation of stochastic differential equations by additive

models using splines and conic programming, International Journal of Computing Anticipatory Systems 21

(2008) 341-352.

Taylan, P., Weber, G.-W., and A. Beck, New approaches to regression by generalized additive models

and continuous optimization for modern applications in finance, science and techology, in the special issue

in honour of Prof. Dr. Alexander Rubinov, of Optimization 56, 5-6 (2007) 1-24.

Taylan, P., Weber, G.-W., and Yerlikaya, F., A new approach to multivariate adaptive regression spline

by using Tikhonov regularization and continuous optimization, to appear in TOP, Selected Papers at the

Occasion of 20th EURO Mini Conference (Neringa, Lithuania, May 20-23, 2008) 317- 322.

Seydel, R., Tools for Computational Finance, Springer, Universitext, 2004.

Stone, C.J., Additive regression and other nonparametric models, Annals of Statistics 13, 2 (1985) 689-705.

Weber, G.-W., Taylan, P., Akteke-Öztürk, B., and Uğur, Ö., Mathematical and data mining contributions

dynamics and optimization of gene-environment networks, in the special issue Organization in Matter

from Quarks to Proteins of Electronic Journal of Theoretical Physics.

Weber, G.-W., Taylan, P., Yıldırak, K., and Görgülü, Z.K., Financial regression and organization, to appear

in the Special Issue on Optimization in Finance, of DCDIS-B (Dynamics of Continuous, Discrete and

Impulsive Systems (Series B)).

References

I1

a b

(3a)

I1 I2 I3 I4

a bI5 I6 I7 I8

(3b)

I2 I4I3 I5 I6I1

a b

(3c)

a b

.

.

.. ...... ......

. .......

..

: ( ) ( )j j j j jInd = d D v V


Appendix

cluster

cluster

robust optimization

C-MARS

Appendix

,

2

2

min ,

subject to ( ) ( ) I ( ) ( ) , 0,

|| ||

t

T

p

qt

F F F F

qL

q t t

M


alternative solution

2

1min ( ) := ( ) + ( ) ( ) + ( ) ( )

2

subject to

T T T

q

Q q f q F F q F F q

q

trust region

Appendix

Parameter Estimation in Stochastic Differential Equations by Continuous Optimization

Education

Transcript of Parameter Estimation in Stochastic Differential Equations by Continuous Optimization