1

USING A PRIORI INFORMATION FOR CONSTRUCTING REGULARIZING ALGORITHMS

Anatoly Yagola

Department of Mathematics, Faculty of Physics,

Moscow State University, Moscow 119899 Russia

E-mail: yagola@inverse.phys.msu.ru

2

Main publications:1.   Tikhonov, A.N., Goncharsky, A.V., Stepanov, 1.   Tikhonov, A.N., Goncharsky, A.V., Stepanov,

V.V. and Yagola, A.G. (1995). V.V. and Yagola, A.G. (1995). Numerical Numerical methods for the solution of ill-posed problemsmethods for the solution of ill-posed problems . . Kluwer Academic Publishers, Dordrecht.Kluwer Academic Publishers, Dordrecht.

2.  Tikhonov, A.N., Leonov, A.S. and Yagola, 2.  Tikhonov, A.N., Leonov, A.S. and Yagola, A.G. (1998). A.G. (1998). Nonlinear ill-posed problemsNonlinear ill-posed problems. . Chapman and Hall, London.Chapman and Hall, London.

3.   Kochikov, I.V., Kuramshina, G.M., Pentin, 3.   Kochikov, I.V., Kuramshina, G.M., Pentin, Yu.A. and Yagola, A.G. (1999). Yu.A. and Yagola, A.G. (1999). Inverse Inverse problems of vibrational spectroscopyproblems of vibrational spectroscopy . VSP, . VSP, Utrecht, Tokyo.Utrecht, Tokyo.

3

Introduction(1)

is a linear operator, are linear normed spaces.The problem (1) is called well-posed on the class of

its “admissible” data if for any pair from the set of “admissible” data the solution of (1):

1) exists,2) is unique,3) continuously depends on errors in and (is

stable).

UuZzuzA ,UZA :

UZ ,

uA,

A u

4

Stability means that if instead of we are given “admissible” such that , , the approximate solution converges to the exact one as . The numbers and are error estimates for the approximate data of (1) with the exact data . Denote . If at least one of the mentioned requirements is not met, then the problem (1) is called ill-posed.

uA, uAh , hAAh uu

0, h h

,h uAh ,

uA,

5

As a generalized solution, it is often taken the so-called normal pseudosolution . It exists and is unique for any exact data of the problem (1) if , , . Here and denote the ranges of the operator and its orthogonal complement in , and stands for the operator pseudoinverse to . Below we find as a normal pseudosolution, i.e., .

z~

),( UZLA )()( ARARu uAz ~

)(AR )(AR

A UA

zzz ~

A

6

What is to solve an ill-posed problem?

Tikhonov answered: to solve an ill-posed problem means to produce a map (regularizing algorithm) . such that

1) brings an element into correspondence with any data , , . of the problem (1);

2) has the convergence property as , .

,,uAR h

,,uARz h ,,uAh ),( UZLAh

Uu

uAzz

0 )()( ARARu

7

All inverse problems may be divided into three groups:

1) well-posed problems,

2) ill-posed regularizable problems,

3) ill-posed nonregularizable problems.

8

Is it possible to construct a regularizing algorithm that does not depend on , ?Theorem 1: Let be a map of the set into

. If is a regularizing algorithm (not depending explicitly on ), then the map . is continuous on its domain . .

Proof The second condition in the definition of RA implies in valid for each . and the convergence as valid for . . The map is continuous on . .

h uAR h , UUZL ,

Z

uAuAP , )()(, ARARUZL

uAR h ,

uAPuAuAR ,,

)()(,, ARARUZLuA uAPuAuARuAP hh ,,,

0, h )()(,,,, ARARUZLuAuA h

uAP , UUZLARARUZL ),()()(,

9

It is clear from Theorem 1 that a regularizing algorithm not using and explicitly can only exist for problems (1) well-posed on the set of the data . The theorem generalized the assertion proved by Bakushinskii. Tikhonov proved the similar theorem when was studying ill-posed SLAE. As result, L-curve and GCV methods cannot be applied for the solution of ill-posed problems.

h

UUZLARARUZL ,)()(,

10

It is very curious that the most popular error free methods cannot solve well-posed problems also! As the first example we consider so-called the “L-curve method” (P.C. Hansen). In this method the regularization parameter in Tikhonov functional is selected as a point maximum curvature of the L-curve

{(ln||Ahz - u||, ln||z||): 0}.

But this method cannot be used for the solution of ill-posed problems because the L-curve doesn’t depend on h and (see the theorem). Everybody can easily prove that this method is inapplicable to solving the simplest finite-dimensional well-posed problems.

11

Let us consider the equation:z = 1. Here Z = U = R1, A = I (unit operator), u = 1.

Let approximate data Ah = I and u = 1 for any h and .

Independently on h and , the regularization parameter selected by the L-curve method

L(Ah, u) = 1.

Therefore, the approximate solution zL = 0.5, and it doesn’t

converge to ze = 1 as h, 0.

Using L-curve method we’ve received 0.5 instead of 1 independently on errors!!!

12

For another popular form of L-curve

{(||Ahz - u||2, ||z||2): 0}

it is possible to prove that such method has systematic error for all well-posed systems of linear algebraic equations (A. Leonov, A. Yagola).Another very popular “error free” method is GCV – the generalized cross-validation method (G. Wahba), where (Ah, u) is found as the point of the global minimum of

the function G() = ||(AhAh

* + I)-1u|| [tr(AhAh* + I)-1]-1, 0.

This method is not applicable for the solution of ill-posed problems including ill-posed systems of linear algebraic equations (see the theorem above). It is possible construct well-posed systems of linear algebraic equations the GCV method failed for their solution.

13

Is it possible to estimate an error of an approximate solution of an ill-posed problem?The answer is negative. The main and very important

result was obtained by Bakushinskii. Assume . Let be a RA. Denote by .

the error of a solution of (1) at the point using the algorithm . If (1) is regularizable by a continuous map and there is an error estimate, which is uniform on

then the restriction of to is continuous on . .

AAh ,uR uzAUuzuRzR ,:,sup,,

zR

RD

00:,,sup asDzzR1A UAD

AD

14

The accuracy of the approximate solution . of the problem (1) could be estimated as , where does not depend on and the function defines the convergence rate of to .

Pointwise and uniform error estimations should be distinguished.

,uRz Kzz

K

z z

15

Consider the results obtained by Vinokurov.

Let be a linear continuous injective operator acting in Banach space and the inverse operator . be unbounded on . Suppose that . is an arbitrary positive function such that . as , and is an arbitrary method to solve the problem.

The following equality holds for elements except maybe for a first category set in :

A uniform error estimate can only exist on a first category subset in .

AZ

1A 1AD

R

z

Z

0 0

zR ,,suplim

0

Z

16

A compact set is a typical example of the first category set in a normed space . For this set special regularizing algorithms may be used and a uniform error estimation may be constructed.

Clearly, a uniform error estimate exists only for well-posed problems.

Z

17

A posteriori error estimation

For some ill-posed problems it is possible to find a so-called a posteriori error estimation.

Let be an exact injective operator with closed graph and be a -compact space.

Introduce a function such that . , , , :

The function is an a posteriori error estimation for the problem (1), if as .

AZ

,u Zz 0 z ],0( z Uu uu

,, uuRz

,u 0, u

0

18

The generalized discrepancy method

Let be Hilbert spaces, be a closed convex set of a priori constraints such that , . , be linear operators. On a set introduce the Tikhonov's functional:

where is a regularization parameter.

(2)

For any , and bounded linear operator . the problem (2) is solvable and has a unique solution .

UZ , ZD D0

A hA ,,uAh

22zuzAzM h

0

DzzM :inf

Uu 0hA

Dz

19

A priori choice of .

A regularizing algorithm using the extreme problem (2) for : to construct such that as .

If is an injective operator, and , . as , then as , i.e., there is the a priori choice of .

zM zz

0

A Dz 0

02

h 0 zz

0

20

A posteriori choice of .

The incompatibility measure of (1) on :

Let it can be computed with an error , i.e., instead of there is such that

The generalized discrepancy:

The generalized discrepancy is continuous and monotonically non-decreasing for .

D

DzuzAAu hh :inf,

0 hAu , hAu ,

hhh AuAuAu ,,,

222, hh AuzhuzA

0

21

The generalized discrepancy principle to choose the regularization parameter:

1) If the condition is not just, then is an approximate solution of (1);

2) If the condition is just, then the generalized discrepancy has a positive zero and .

If is an injective operator, then . Otherwise, , where is the normal solution of (1), i.e., . .

222, hAuu

0z 222

, hAuu

**

zz A zz

0lim

*

0lim zz

*z uAzDzzz ,:inf*

22

If are bounded linear operators, is a closed convex set, , , the generalized discrepancy principle are equivalent to the generalized discrepancy method:

find

hAA, DD0 Dz

222,,: hh AuzhuzADzz

23

Inverse problem for the heat conduction equation.

There is a function , we want to find such that as .

We may write that

0,

0,0

,0,02

tlw

tw

Tltxwaw xxt

lLTwu ,0, 2

lWxwxz ,00, 21 xzxz

0

l l

dxx

xzxzxzduu

0 0

22222

,

24

The problem may be written in the form of integral equation

where is the Green function:

The problem is solved for the parameters . , the function is taken such that .

l

dxxzTxGu0

,,

txG ,,

1

2

2 expsinsin,,n

l tl

na

l

nx

l

ntxG

0.1,1.0,0.1 lTa u

u 05.0

25

The exact solution ( ) and the approximate solution ( ).

)(xz xz

26

The Euler equation

The Tikhonov's functional is a strongly convex functional in a Hilbert space.

The necessary and sufficient condition for to be a minimum point of on a set of a priori constraints is

If is an interior point of , then , or

We obtain the Euler equation.

zM

z

zM D

DzzzzM

0,

z D 0

zM

uAzzAA hhh

**

27

Sourcewise represented sets(1)

is a linear injective operator.

Assume the next a priori information: is sourcewise represented with a linear compact operator :

(3)

Here is a reflexive Banach space.

Suppose is injective, is known exactly, .

uzA UZA :

z

ZVB :

vBz V

B A uu

28

Set and define the set

Minimize the discrepancy on .

If , then the solution is found. Denote . Otherwise, we change to and reiterate the process.

If is found, then we define the approximate solution of (1) as an arbitrary solution of the inequality

1n nvVvBvzZzZn ,,:

uAzzF

nZzuAz :min

nZ

n

1n

nn

n

nz

nZzuAz

29

Theorem 2: The process described above converges: . . There exists (generally speaking, depending on ) such that for . Approximate solutions strongly converge to . as .

Proof The ball is a bounded closed set in . The set is a compact in for any , since is a compact operator. Due to Weierstrass theorem the continuous functional attains its exact lower bound on .

Clearly, , where

. is the integer part of a number.

n 00 z 0 nn 0,0

nzz 0

nvVvVn :V nZ Z nB

zFnZ

NZvBz

otherwise1

integerpositiveais

v

vvN

30

Therefore is a finite number and there is such that for any . The inequality for any is evident. Thus, for all the approximate solutions . belong to the compact set , and the method coincides with the quasisolutions method for all sufficiently small positive . The convergence follows from the general theory of ill-posed problems.

Remark: The method is a variant of the method of extending compacts.

n0 0 nn 0,0

Nn 0 0,0

nz 0nZ

zzn

31

Theorem 3: For the method described above there exists an a posteriori error estimate. It means that a functional exists such that as and at least for all sufficiently small positive .

Remark 2: The existence of the a posteriori error estimation follows from the following. If by . we denote the space of sourcewise represented with the operator solutions of (1), then . Since is a compact set, then . is a -compact space.

,u 0, u0 ,uzzn

ZZ B

1n nZZ nZZ

32

An a posteriori error estimate is not an error estimate in general meaning that is impossible in principle for ill-posed problems. But it becomes an upper error estimate of the approximate solution for “small” errors , where depends on the exact solution .

0 0z

33

The operators and are known with errors. Let there be linear operators , such that . , . Denote the vector of errors by . For any integer define a compact set .

Find a minimal positive integer number such that the inequality

has a nonempty set of solutions.

Then the a posteriori error estimation is

A B

AhABhB

Ah hAAA

Bh hBBB

BA hh ,, n nvVvvBzZzZ

BB hhn ,,:,

nn

nhhAhBhuzA BAhBhAh ABA

},:max{,,, ,

nhhAhBhuzA

ZzzznhBAu

BAhBhAh

hnnBhh

ABA

BBA

34

Inverse problem for the heat conduction equation

For any moment of time there is

where . Suppose .

We solve the problem using the method of extending compacts.

Let , , , , .

dxxvtxGxBvzl

0

,, 0t

0,xwxv lLUZV ,02

0.1a 0.1l 02.0t 1.0T u 03.0

otherwise0

8.05.04

5.03.010

x

x

xv

35

The approximate solution and its a posteriori error estimation. We obtain .

xz

5n

36

Compact sets

There is the additional a priori information: the exact solution of (1) belongs to a compact set and is a linear continuous injective operator.

As a set of approximate solutions of (1) it is possible to accept

Then as in for any .

zAM

zhuzAMzZ hM :

zz 0 Z MZz

37

After finite dimensional approximation we obtain that , where is a convex polyhedron for convex or monotonic functions and

. is a matrix, and are vectors.

To find it is possible to use the method of conditional gradient or the method of projection conjugated gradients.

ZMZMˆˆˆ M

uzAZzZ ˆˆˆ:ˆˆˆ

A z u

z

38

Error estimation

1) Find the minimum and the maximum values for each coordinate of . Denote them by , , . .

2) Secondly, using the found we construct functions and close to such that . for each .

Therefore, we should minimize a linear function on a convex set. We may approximate the set by a convex polyhedron and solve a linear programming problem. The simplex-method or the method to cut convex polyhedrons may be used.

MZ

liz u

izni ,1

ul zz ˆ,ˆ xz l xzu

MZ xzxzxzZz ul

M : bax ,

39

Inverse problem for the heat conduction equation.

Let be a set of convex upward functions such that . Assume that , , . , , the number of nodes 20.

M xz Cxz 0 0.1a 0.1l

0.1T 2.1C

40

The exact solution ( ), the functions , . xz xz l xzu

41

We shall formulate now general conditions for constructing of regularizing algorithms for the solution of nonlinear ill-posed problems in finite-dimensional spaces. These conditions could be easily checked for an inverse vibrational problem which we consider as a problem to find the normal pseudosolution of nonlinear ill-posed problem on a given set of constraints. We shall discuss typical a priori constraints.

42

In this section the main problem for us is an operator equation 

(1) where D is a nonempty set of constraints, Z and U are finite-dimensional normed spaces, is a class of operators from D into U. Let us give a general formulation of Tikhonov's scheme of constructing a regularizing algorithm for solving the main problem: for the operator Eq. (1) on D find an element z* for which 

(2)

UuZDzuAz ,,

DzuAzuAz : ,inf *,

43

We assume that to some element there corresponds the nonempty set Z* in D of quasisolutions and that Z* may consist of more than one element. Furthermore, we suppose that a functional is defined on D and bounded below: 

 The -optimal quasisolution problem for Eq. (1) is

formulated as follows: find a such that  

uu

z

*Zz

.* : inf Zzzz

0 : inf* Dzzz

44

We suppose that instead of the unknown exact data (A, u), we are given approximate data which satisfy the following conditions:

  

Here the function represents the known measure of approximation of precise operator A by approximate operator .We are given also numerical characterizations of the “closeness” of to (A, u). The main problem is to construct from the approximate data an element  

 which converges to the set -optimal pseudosolutions as

.

,,,,;,,

Dz

zhzAAzAuuUu hh

hA 0, h

),( uAh

DhuAzz h ,,,,

Z

.0, h

),( uAh

45

Let us formulate our basic assumptions.

1) The class consists of the operators A continuous from D to U.

2) The functional is lower semicontinuous on D.3) If K is an arbitrary number such that

then the setis compact in Z.

4) The measure of approximation is assumed to be defined for , to depend continuously on all its arguments, to be monotonically increasing with respect to for any h 0, and satisfy the equality

z*K

KzDzK :

),( h*,0 h

* ,0,0

46

Conditions 1)-3) guarantee that Tikhonov’s scheme for constructing regularizing algorithms is based on using the smoothing functional    in the conditional extreme problem: for fixed 0, find an element such that  

.Z

0, ,, DzzuzAfzM h

Dz

.: inf DzzMzM

47

Here f[x] is an auxiliary function. A common choice isWe denote the set of extremals of (5) which correspond to a given 0 by Conditions 1)-3) imply that

.2, mxxf m

.Z.Z

48

The scheme of construction of an approximation to the set includes:

(i) the choice of the regularization parameter ;

(ii) the fixation of the corresponding to , and a special selection of an element in this set:

Z

,,, huAhZ

z

as 0. Zz

49

It is in this way that the generalized analogs of a posteriori parameter choice strategies are used. They were introduced by A.S. Leonov. We define for their formulation some auxiliary functions and functionals: 

 

Here    is a generalized measure of incompatibility for nonlinear problems having the properties [2]:

.

DzzhuzAh : ,,inf

0 , as

.0 ; ,

,,

,I, ,

Zzz

zhf

zuzAfz h

50

All these functions are generally many-valued. They have the following properties.

Lemma. The functions are single-valued and continuous everywhere for >0 except perhaps not more a countable set of their common points of discontinuity of the first kind, which are points of multiple-valuedness, then there exists at least two elements in the set such that . The functions , are monotonically nondecreasing and , are nonincreasing. The generalized discrepancy principle (GDP) for nonlinear problems consists of the following steps.

,,,

zz , Z

zz 0 ,0

51

(i) The choice of the regularization parameter as a generalized solution > 0 of the equation 

.

 Here and in the sequel we say that is the generalized

solution for a monotone function if is an ordinary solution or if is a “jump”-point of this function over 0.

0

52

The method of selecting an approximate solution from the set by means of the following selection rule. Let q>1 and C>1 be fixed constants, are auxiliary regularization parameters, and let and be extremals of (4) for =1,2.

 If the inequality holds for and , then any elements subject to the condition can be taken as the approximate solution. For instance we can take . But if 

 then we choose so as to have , for example .

zZ

qq 21 ,: 1z 2z

fCzCz 112 1z 2z Zz

0 z

zz

fCzCz 112

z 0 z

zz

53

Theorem. Suppose that for any quasisolution the inequality holds. Then (a) has positive generalized solution; (b) for any sequence such as , the corresponding sequence of approximate solutions, which is found by GDP has the following properties: 

Dz *

Dzzz : inf**

nnn h , 0 nz

. as , nzzz nn

0

54

In many practical cases it is very convenient to take (r is a constant, r >1). If it is known in addition that the operator equation has a solution on D, then the value can be omitted. GDP in linear and nonlinear cases has some optimal properties.

rzz

55

References1. Hadamard, J. (1923). Lectures on Cauchy's problem in linear partial

differential equations, Yale Univ. Press, New Haven.2. Tikhonov, A. N., Leonov, A. S. and Yagola, A. G. (1998). Nonlinear

ill-posed problems, Chapman and Hall, London.3. Tikhonov, A. N. (1963). Solution of incorrectly formulated problems

and the regularization method, Sov. Math., Dokl., 5, 1035-1038.4. Tikhonov, A. N. (1963). Regularization of incorrectly posed

problems, Sov. Math., Dokl., 4, 1624-1627.5. Leonov, A. S. and Yagola, A. G. (1995). Can an ill-posed problems

be solved if the data error is unknown? Moscow Univ. Physics Bull., 50(4), 25-28.

6. Bakushinskii, A. B. (1984). Remark about the choice of the regularization parameter by the quasioptimality criterion and the propertion criterion, Comput. Math. Math. Phys., 24(8), 1258-1259.

7. Bakushinskii, A. B. and Goncharskii, A. V. (1994). Ill-posed problems: theory and applications, Kluwer Academic Publishers, Dordrecht.

56

8) Tikhonov, A. N., Goncharsky, A. V., Stepanov, V. V. and Yagola, A. G. (1995). Numerical methods for the solution of ill-posed problems, Kluwer Academic Publishers, Dordrecht.

9) Vinokurov, V. A. (1979). Regularizable functions in topological spaces and inverse problems, Sov. Math., Dokl., 20, 569-573.

10) Vinokurov, V. A. and Gaponenko, Yu. L. (1982). A posteriori estimates of the solutions of ill-posed inverse problems, Sov. Math., Dokl., 25, 325-328.

11) Yagola, A. G. and Dorofeev, K. Yu. (2000). Sourcewise representation and a posteriori error estimates for ill-posed problems, In: Ramm, A. G. et al., eds., Fields Institute Communications: Operator Theory and Its Applications, 25, 543-550, AMS, Providence, RI.

12) Ivanov, V. K., Vasin, V. V. and Tanana, V. P. (1978). The theory of linear ill-posed problems and its applications, Nauka, Moscow (in Russian).

13) Dombrovskaya, I. N. and Ivanov, V. K. (1965). Some questions to the theory of linear equations in abstract spaces, Sibirskii Mat. Zhurnal, 16, 499-508 (in Russian).

57

14) Riesz, F. and Sz.-Nagy, B. (1990). Functional analysis, Dover Publications Inc., New York.

15) Vainikko, G. M. (1982). Methods for the Solution of Linear Incorrectly Formulated Problems in Hilbert Spaces. Textbook, Tartu University Press, Tartu.

16) Leonov, A. S. and Yagola, A. G. (1998). Special regularizing methods for ill-posed problems with sourcewise represented solutions, Inverse Problems, 14(6), 1539-1550.

17) Titarenko, V. N. and Yagola, A. G. (2000). A method to cut convex polyhedrons and its applications to ill-posed problems, Numerical Methods and Programming, 1, section 1, 8-13, www.http://num-meth.srcc.msu.su.

18) Goncharsky, A. V., Cherepashchuk, A. M., and Yagola, A. G. (1985). Ill-posed problems of astrophysics, Nauka, Moscow (in Russian).

19) Goncharsky, A. V., Cherepashchuk, A. M., and Yagola, A. G. (1978). Numerical methods for the solution of inverse problems in astrophysics, Nauka, Moscow (in Russian).

58

20) Rusov, V. D., Babikova, Yu. F. and Yagola, A. G. (1991). Image restoration in electronic microscopy autoradiography of surfaces, Energoatomizdat, Moscow (in Russian).

21) Yagola, A. G., Kochikov, I. V., Kuramshina, G. M., and Pentin, Yu. A. (1999). Inverse problems of vibrational spectroscopy, VSP, Zeist.