AmesosSparse Direct Solver Package

Ken Stanley, Rob Hoekstra, Marzio Sala, Tim Davis, Mike Heroux

Trilinos Users GroupTrilinos Users GroupAlbuquerque Albuquerque

3 Nov 20043 Nov 2004

What’s new in Amesos

• Release - in Trilinos 4.0• Solvers - Klu, Mumps, SuperLU• Interface - Base Solver and Factory Method• Epetra_RowMatrix support

• Future – Distributed Memory Solver

Amesos Interface to sparse direct solvers

•Common clean interface •Pick a solver!•Any solver on any (Epetra) matrix•Trilinos package –Autotooled configuration & build

–Nightly testing

–Bug Tracking

Epetra_LinearProblem Problem( A, x, b);

Amesos Afact;

Amesos_BaseSolver* Solver = Afact.Create( “Klu”, Problem ) ;

Solver->SymbolicFactorization( ) ;

Solver->NumericFactorization( ) ;

Solver->Solve( ) ;

Calling Amesos

Amesos Solvers

•Klu: Built-in. Serial unsymmetric [Davis] • SuperLU: Serial unsymmetric [Li et al.]•UMFPACK: Serial unsymmetric [Davis]• SuperLUdist: Parallel unsymmetric [Li et al.]•MUMPS: Parallel unsymmetric [Amestoy et al.]•DSCPACK: Parallel Symmetric [Ragavan]

• Amesos User Guide:

http://software.sandia.gov/trilinos/packages/amesos• Build Amesos

– CONFIGURE:./configure –enable-amesos

– BUILD:make

– TEST:cd amesos/test

source AmesosKlu.exe

Getting started: Building Amesos

Epetra_LinearProblem Problem( A, x, b);

Amesos Afact;

Amesos_BaseSolver* Solver = Afact.Create( “Klu”, Problem ) ;

Solver->SymbolicFactorization( ) ;

For () {

// Changes to the data, but not the non-zero pattern

Solver->NumericFactorization( ) ;

Solver->Solve( ) ;

}

Calling AmesosSame Structure Different Data

Epetra_LinearProblem Problem( A, x, b);

Amesos Afact;

Amesos_BaseSolver* Solver = Afact.Create( “Klu”, Problem ) ;

Solver->SymbolicFactorization( ) ;

Solver->NumericFactorization( ) ;

For ( ) {

// Changes to b

Solver->Solve( ) ;

}

Calling AmesosMultiple Solves, Changing RHS

Epetra_Multivector x, b;

Epetra_LinearProblem Problem( A, x, b);

Amesos Afact;

Amesos_BaseSolver* Solver = Afact.Create( “Klu”, Problem ) ;

Solver->SymbolicFactorization( ) ;

Solver->NumericFactorization( ) ;

Solver->Solve( ) ;

Calling AmesosBlocked Right Hand Sides

Epetra_LinearProblem Problem( A, x, b);Amesos Afact;Amesos_BaseSolver* Solver = Afact.Create( “Klu”, Problem ) ; ParamList.SetParam( “Refactorize”, true );Solver->SetParameters( ParamList );

Solver->SymbolicFactorization( ) ; For () { // Small changes to the data, Same non-zero pattern Solver->NumericFactorization( ) ; Solver->Solve( ) ; }

Calling AmesosPivotless Re-factorization

Amesos Switching solvers

• Afact.Create( “Klu”, Problem ) ;

-> Afact.Create( “SuperLUdist”, Problem ) ;

•Amesos redistributes data (serial to parallel, etc.)•Heuristics consistent across solvers• Parameter List – Allows individual control over each solver

•Amesos fills in capabilities– Transpose

– Blocked right hand sides

• Amesos User Guide: http://software.sandia.gov/trilinos/packages/amesos

• Download source from the authors’ website– Modify make.inc – “make” – Check output of example run

• Future Work – Work with authors to Autotool their libraries

Amesos Adding third party libraries

• Amesos User Guide:

http://software.sandia.gov/trilinos/packages/amesos• Build Amesos

– CONFIGURE: (e.g. DSCPACK )./configure –enable-amesos –enable-amesos-dscpack –with-libs=“-L/dir –ldscpack” -with-incdirs=-I/dir/SRC

– BUILD:make

– TEST:cd amesos/test

source AmesosDscpack.exe

Building Amesoswith additional third party libraries

Amesos Future Work

•Additional third party libraries •Allow any ordering to be used by any package• Extended precision iterative refinement• Enhance nightly testing– Performance

– Memory leaks

– Check code coverage

•Work with authors to autotool their libraries

Amesos [Davis, Stanley, Heroux]

Distributed memory solver goals:

•Built into Amesos in 2005• Simple design• Full partial pivoting– Allowing restrictions may improve performance

• Leverage existing technology– Epetra support

– Amesos serial solvers

– Partitioners

– Static Pivoting

AmesosDistributed memory solver overview

•Reduction to Block Triangular form•Partition each block•Use “enhanced” Amesos serial solver for local partition factorization– Restricted Pivoting

– Partial Factorization

•Add Schur Complements to root node•Use any Amesos solver on the root node

No Pivot Case 22 23

32 3

13

31 3

11

A A

A A

A

0

A

A0

Factor Leaves

3

22 22 2 22 2

2 2

1

3

11 11 13 11 1

31 3

3

3 3

L

L U U A A

L I A I

U U A A

L I A I

ComputeSchur Complements

1

2 3

31 3

2 3

1

2S

SC = L U

C = L U

Update Root Node

'33 3 213 -A SCA -SC=

Factor Root'

33 33 33L U = A

• Static pivoting [Li; Demmel]• Local pivoting

•Delayed pivoting

• Singly bordered

form [J. Scott]

augmented matrix

Pivoting Strategies

1 11 11 1 11P L U Q = A

11( )subset1 11 11 1P L U Q = A

1

2

11 13

3

2

33

2 23

3

A

-

A

0

A

I

A

I A

A

A

22 23

32 3

13

31 3

11

A A

A A

A

0

A

A0

Static and Local Pivoting

• Static pivoting [Li; Demmel]– Reorder and scale rows and columns to promote

diagonal dominance

– Imperfect but works well in practice

– Can reduce need for pivoting even if it doesn’t eliminate it

• Local pivoting – Requires no additional communication

1 11 11 1 11P L U Q = A

Delayed pivoting

•Delay columns

which do not allow

an acceptable pivot choice• Allow root node to grow• Threshold pivoting reduces communication• Requires:– A serial solver which supports

delayed pivoting

– Mutable root node size

22 23

32 3

13

31 3

11

A A

A A

A

0

A

A0

11( )subset1 11 11 1P L U Q = A

33A

•Augment matrix, adding

columns [-I;I] to split

and onto separate

rows•Allows full partial pivoting• If and are not used

as pivots, matrix can be

collapsed to original size

Singly bordered form

11 13

3

22 23

32 3

1

3

A

-

A

0

A

I

A

I A

A

A

22 23

32 3

13

31 3

11

A A

A A

A

0

A

A0

31A 23A

31A 23A

Amesos Interface to sparse direct solvers

•Simple interface •SuperLU, KLU, MUMPS, UMFPACK, DSCPACK, SuperLUdist•Built-in serial solver •Future:–Built-in distributed memory solver

–More third party solvers

–Extended precision iterative refinement