Solving Scalar Linear Systems Iterative approach

Lecture 15

MA/CS 471

Fall 2003

Some matlab scripts to construct various types of random circuit loop matricesare available at the class website:

The Sparsity Pattern of a Loop Circuit Matrix for a Random Circuit (with 1000 closed loops)

b

gridcircuit.m

An array of current loops with random resistors (resistors on circuit boundary not shown)

b

Matrix Due To Random Grid Circuit

Note the large amount of structure in the loop circuit matrix

The Limits of Factorization

• In the last class/lab we began to see that there are limits to the size of linear system solvable with matrix factorization based methods.

• The storage cost for the loop current matrix built on a Cartesian circuit stored as a sparse NxN matrix is ~ 5*N

• However, using LU (or Cholesky) and symmetric RCM the storage requirement is b*N which is typically at least an order of magnitude larger than the storage required for the loop matrix itself.

• Also – memory spent on storing the matrices is memory we could have used for extra cells…

Alternative Approach

• We are going to pursue iterative methods which will satisfy the equationin an approximate way without an excessive amount of extra storage.

• There are a number of different classes of iterative methods, today we will discuss an example from the class of stationary methods.

Ax b

http://www.netlib.org/linalg/html_templates/node9.html

Jacobi Iteration• Example system:

• Initial guess:• Algorithm:

• i.e. for the I’th equation compute the I’th degree of freedom using the values computed from the previous iteration.

5 1 3 2

3 6 2 3

2 3 6 1

x y z

x y z

x y z

1

1

1

5 1 3 2

3 6 2 3

2 3 6 1

n n n

n n n

n n n

x y z

x y z

x y z

0 0 0 0x y z

Cleaning The Scheme Up

1

1

1

12 1 3

51

3 3 261

1 2 36

n n n

n n n

n n n

x y z

y x z

z x y

1

1

1

5 1 3 2

3 6 2 3

2 3 6 1

n n n

n n n

n n n

x y z

x y z

x y z

Couple of Iterations

1 0 0

1 0 0

1 0 0

1 22 1 3

5 51 1

3 3 26 21 1

1 2 36 6

x y z

y x z

z x y

1st iteration:

2nd iteration:

2 1 1

2 1 1

2 1 1

1 1 1 1 12 1 3 2 1. 3.

5 5 2 6 5

1 1 2 1 113 3 2 3 3. 2.

6 6 5 6 45

1 1 2 1 131 2 3 1 2. 3.

6 6 5 2 60

x y z

y x z

z x y

Pseudo-Code For Jacobi Method

1) Build A, b2) Build modified A with diagonal zero Q3) Set initial guess x=04) do{

a) compute:

b) compute error:

c) update x: }while err>tol

1

1 j N

i i ij jjii

x b x

Q

A

1,..,

1,..,

max

max

i ii N

ii N

x xerr

b

i ix x

Matlab To The Rescue

Running The Jacobi Iteration Script

• I ran the script with the stopping tolerance (tol) set to 1e-8:

• Note that the error is of the order of 1e-8 !!!

• i.e. the Jacobi iterative method computes an approximate solution to the system of equations!.

Try The Larger Systems

• First I made the script into a function which takes the rhs vector, system matrix and stopping tolerance as input.

• It returns the approximate solution and the residual history.

Run Time!

• First I built a circuit matrix using the gridcircuit.m script (with N=100)

• Then I built a random source vector for the right hand side.

• Then I called the jacobisolve routine with:

• [x,residuals] = jacobisolve(mat,b,1e-4);

Convergence History

Stoppingcriterion..

Increasing System Size

Notice how the number of iterations required grew with N

Ok – so I kind of broke the rules• We set up the Jacobi iteration but did not ask the

question “when will the Jacobi iteration converge and how fast?”

Definition: A matrix A is diagonally dominant if

Theorem: If the matrix A is diagonally dominant then Ax=b has a

unique solution x and the Jacobi iteration produces a sequence which converges to x for any initial guess

Informally:The “more diagonally dominant” a matrix is the faster it

will converge… this holds some of the time.

nx 0x

1,

N

ij iij j i

A A

Going Back To The Circuit System

• For the gridcircuit code I set up the resistors so that all the internal circuit loops shared all their resistors with other loops.

• the current balance law => for internal cells the total cell resistance = sum of of resistances shared with neighboring cells…

• i.e. the row sums of the internal cell loop currents is zero

• i.e. the matrix is weakly diagonally dominant – and does not exactly satisfy the convergence criterion for the Jacobi iterative scheme.

1,

N

ij iij j i

A A

Slight Modification of the Circuit

• I added an additional random resistor to each cell (i.e. increased the diagonal entry and did not change the off-diagonal entries).

• This modification ensures that the matrix is now strictly diagonally dominant.

Convergence history for diagonally dominant system – notice dramaticreduction in iteration count

Gauss-Seidel

• Example system:

• Initial guess:• Algorithm:

• i.e. for the I’th equation compute the I’th degree of freedom using the values computed from the previous iteration and the new values just computed

5 1 3 2

3 6 2 3

2 3 6 1

x y z

x y z

x y z

1

11

1 1 1

3

5 1 3 2

6 2 3

3 6 12

n

n n

n n n

n n

n

x

x y

x y z

y z

z

0 0 0 0x y z

Cleaning The Scheme Up

1

1 1

1 1 1

12 1 3

51

3 3 261

1 2 36

n n n

n n n

n n n

x y z

y x z

z x y

1

1 1

1 1 1

5 1 3 2

3 6 2 3

2 3 6 1

n n n

n n n

n n n

x y z

x y z

x y z

First Iteration

1 0 0

1 0

1

1

1 1

12 1 3

51

3 3 261

1 2 36

x y

x

x y

z

y z

z

1st iteration:

As soon as the 1 level values are computed, we use them in the next equations..

Theorem First This Time!.• So we should first ask the questions

1) When will the Gauss-Seidel iteration converge?2) How fast will it converge?

Definition:A matrix is said to be positive definite if

Theorem:If A is symmetric and positive definite, then

the Gauss-Seidel iteration converges for any initial guess for x

Unoficially:Gauss-Seidel will converge twice as fast in some

cases as Jacobi.

0 for all t N x Ax x

Gauss-Seidel Algorithm

• We iterate:

0

11 1

1 1

0

1

i

j i j Nn n ni i ij j ij j

j j iii

x

x b x x

A A

A

Comparing Jacobi and Gauss-Seidel

Same problem – Gauss-Seidel takes almost half the work

The Catch• Ok – so it looks like one would always use

Gauss-Seidel rather than Jacobi iteration.

• However, let us consider the parallel implementation.

11

1 1

11 1

1 1

1:

1:

j i j Nn n ni i ij j ij j

j j iii

j i j Nn n ni i ij j ij j

j j iii

J x b x x

GS x b x x

A AA

A AA

Volunteer To Design A Parallel Version

1) Decide which cells go where2) Decide how much information each

process needs to keep locally3) Decide what information needs to be

communicated among processes.4) Are there any intrinsic bottlenecks in

Gauss-Seidel or Jacobi?.5) Can we devise a hybrid version of GS

which avoids the bottlenecks?.

Project 3 (serial part)

In C:

1) Build a sparse matrix based on one of the random circuits (design the storage for it yourself or use someone else’s sparse storage structure or class)

2) Write a sparse matrix times vector routine.

3) Implement the Jacobi iterative scheme