TSF and Abstract Interfaces in Trilinos TSFCore, TSFExtended … Roscoe A. Bartlett Department 9211:...

TSF and Abstract Interfaces in Trilinos

TSFCore, TSFExtended …

Roscoe A. Bartlett

Department 9211: Optimization and Uncertainty Estimation

Sandia National Laboratories

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

Outline

• Introduction of abstract numerical algorithms (ANAs)

• Background on Unified Modeling Language (UML) and “Design Patterns”

• Overview of TSF interfaces and software

• Discussion of different approaches for interoperability between Trilinos packages

• General recommendations and open questions related to TSF and Trilinos

Categories of Abstract Problems and Abstract Algorithms

Linear Problems:

Linear equations:

Eigen problems:

Nonlinear Problems:

Nonlinear equations:

Stability analysis:

Transient Nonlinear Problems:

DAEs/ODEs:

Optimization Problems:

Unconstrained:

Constrained:

Trilinos Packages

Belos

Anasazi

NOX

LOCA

Rhythms, TOX ???

MOOCHO

Common Environments for Scientific Computing

Serial / SMP (symmetric multi-processor)

• All data stored in RAM in a single local process Code

Data

Processor

Out of Core

• Data stored in file(s) (too big to fit in RAM)Processor Disk

Code Data SPMD (Single program multiple data)

• Same code on each processor but different data

Code

Data(0)

Proc 0

Code

Data(1)

Code

Data(N-1)

MPP (massively parallel processors)

…

Proc 1 Proc N-1

Introducing Abstract Numerical Algorithms

An ANA is a numerical algorithm that can be expressed abstractly solely in terms of vectors, vector spaces, and linear operators (i.e. not with direct element access)

Example Linear ANA (LANA) : Linear Conjugate Gradients

scalar product<x,y> defined by vector space

vector-vector operations

linear operator applications

Scalar operations

Types of operations Types of objects

What is an abstract numerical algorithm (ANA)?

Linear Conjugate Gradient Algorithm

Examples of Nonlinear Abstract Numerical Algorithms

Class of Numerical Problem Example ANA

Nonlinear equations Newton’s method (e.g. NOX, MOOCHO)

Initial Value DAE/ODE Backward Euler method (e.g. Rhythms, TOX???)

Linear equations GMRES (e.g. Belos)

Requires

Requires

Outline






Unified Modeling Language (UML)

Good tutorial on object-orientation development process and short (about 200 pages) introduction to the UML.

Unified Modeling Language

Graphical language for designing, documenting and communicating designs object-oriented software

Standardized by the Object Management Group (OMG) and backed by a large group of software companies

Allows the specification and description of object-oriented software at various levels of detail

Supported by many different CASE tools for

Designing OO software

Documenting OO software

Reverse engineering OO software

Creating skeleton code for OO software

Refactoring OO software

eFunc(in c: C)

class BaseA {public: virtual ~BaseA() {} virtual int aFunc() const = 0; virtual C& c() = 0; …};

class DerivedB : public BaseA {public: …private: std::vector<BaseD*> d_;};

class C { … };

class BaseD { … };

class DerivedE : public BaseE {public: void eFunc( const C& c ); …};

DerivedB

aFunc(): int

Overview of Basic UML Notation for Object Orientation (OO)

BaseA Cc

DerivedE

d

1…*

BaseD

Simple UML Class Diagram Compatible C++ declarations

Inheritance

Abstract class(italics)

Concrete class(non-italics)

Abstract member function(italics)

Association

Member function implementation

(non-italics)

Association

Multiplicity

Classes

Relationshipss

Dependency

Abstract classand inheritance

(italics)

Design Patterns : “Standard” Vocabulary for OO

Some of the Design Patterns used in TSF

Creational Patterns

“Abstract Factory”

“Prototype”

Structural Patterns

“Adapter”

“Composite”

“Decorator”

“Proxy”

Behavior Patterns

“Strategy”

“Visitor”

Design Patterns in UML Notation:

http://exciton.cs.oberlin.edu/javaresources/DesignPatterns/default.htm

Good tutorial on object-orientation and a catalog of 23 fundamental design patterns of object-oriented software

“Common problems have common solutions!”

http://exciton.cs.oberlin.edu/javaresources/DesignPatterns/default.htm

Outline






Software Componentization and TSF Interfaces

2) LAL : Linear Algebra Library (e.g. vectors, sparse matrices, sparse factorizations, preconditioners)

ANA

APP

ANA/APP Iterface

ANA Vector Interface

ANA Linear Operator Interface

1) ANA : Abstract Numerical Algorithm (e.g. linear solvers, eigen solvers, nonlinear solvers, stability analysis, uncertainty quantification, transient solvers, optimization etc.)

3) APP : Application (the model: physics, discretization method etc.)

Example Trilinos Packages:• Belos (linear solvers)• Anasazi (eigen solvers)• NOX (nonlinear equations)• Rhythms, TOX??? (ODEs,DAEs)• MOOCHO (Optimization)• …

Example Trilinos Packages:• Epetra/Tpetra (Mat,Vec)• Ifpack, AztecOO, ML (Preconditioners)• Meros (Preconditioners)• Pliris (Interface to direct solvers)• Amesos (Direct solvers)• Komplex (Complex/Real forms)• …Types of Software Components

TSFCoreANA Interfaces to Linear Algebra

TSFExtendedAPP to LAL Interfaces TSFExtended

LAL to LAL Interfaces

TSFCore::Nonlin ???

Examples:• SIERRA• NEVADA• Xyce• Sundance• …

LAL

Matrix Preconditioner

Vector

ANA software allows for a much more abstract interface than APP or LAL software!

Different Platform Configurations for Running ANAs

SPMDANA

LAL

Proc 0

APP

MPP

ANA

LAL

Proc 1

APP ANA

LAL

Proc N-1

APP

…

Master/SlaveANA

LAL

Proc 0 (master)

APP

MPP

LAL

Proc 1

APP

…

Client/ServerMaster/Slave

LAL

Proc N-1

APP

MPP Server

LAL

Proc 1

APP…

LAL

Proc N-1

APP

LAL

Proc 0

APP

Client

ANA

If a code can not run in these modes it is not an ANA code!

Requirements for Abstract Numerical Algorithms and TSF

An important consideration Scientific computing is computationally expensive!

Abstract Interfaces for Abstract Numerical Algorithms using TSF must:

• Be portable to all ASC (advanced scientific computing) computer platforms

• Provide for stable and accurate numerics

• Result in algorithms with near-optimal storage and runtime performance– Scientific computing is expensive!

An important ideal A customized hand-code algorithm in Fortran 77 should not provide significantsignificant improvements in storage requirements, speed or numerical stability!

Abstract Interfaces for Abstract Numerical Algorithms using TSF should:

• Be minimal but complete (good object-oriented design principle)

• Support a variety of computing platforms and configurations– i.e. Serial/SMP, Out-of-Core, SPMD, master/slave and client/server

• Be (relatively) easy to provide new implementations (Subjective!!!)

• Not be too complicated (Subjective!!!)

An important ideal Object-oriented “overhead” should be constant and not increase as the problem size increases.

LinearOpVectorSpace

OpBase

Vector

MultiVector

1

columns1..*

RTOpT

rangedomain

space

Fundamental TSFCore Linear Algebra Interfaces

An operator knows its domain and range VectorSpaces

A MultiVector has a collection of column vectors

A MultiVector is a linear operator

A Vector is a MultiVector and therefore is a linear operator

A Vector knows its VectorSpace

A linear operator can be applied to Vectors and MultiVectors

A linear operator is a kind of operator

Trilinos TSFCore website

Fundamental TSFCore Linear Algebra Interfaces

createMember() : VectorcreateMembers(in numMembers : int) : MultiVectorisCompatible(in vecSpc : VectorSpace) : boolscalarProd(in x : Vector, in y : Vector) : Scalar

dim : int

TSFCore::VectorSpace

applyOp(in op : RTOpT, inout ...)

TSFCore::Vector

apply_op(inout ...)reduce_reduct_objs(inout ...)

RTOpPack::RTOpT

applyOp(in op : RTOpT, inout ...)subView(in col_rng : Range1D) : MultiVectorsubView(in numCols : int, in cols[1..numCols] : int) : MultiVector

TSFCore::MultiVector

opSupported(in M_trans) : bool

TSFCore::OpBase

1

columns1..*

«create»

«create»

domain

space

apply(in M_trans, in x : Vector, inout y : Vector, in ...)apply(in M_trans, in X : MultiVector, inout Y : MultiVector, in ...)

TSFCore::LinearOp

createVecSpc(in dim : int) : VectorSpace

TSFCore::VectorSpaceFactory

«create»

smallVecSpcFcty

range

VectorSpaces create Vectors and MultiVectors!

Vector and MultiVector versions of apply(…) must agree with scalar product:

Adjoints supported but are optional!

Only one vector operation!

Scalar product <x,y> defined by vector space

Creates domain spaces for MultiVector objects

Interface that supports all element-wise vector reduction and transformation operations!

VectorSpace defines compatibility of vectors and multi-vectors

Creation of arbitrary subviews of columns

DRu,vAu,Av H

What’s the Big Deal about Vector-Vector Operations?

Examples from OOQP (Gertz, Wright)

y y x i ni i i / , ...1y y x z i ni i i i , ...1

yy y y yy y y y

y y yi ni

i i

i i

i

min min

max max

min max

, ...ififif0

1 dx:max

Example from TRICE (Dennis, Heinkenschloss, Vicente)

d

b u w bw b

u a w aw a

i ni

i i i

i i

i i i

i i

( ) and and

( ) and . and .

, ...

/

/

1 2

1 2

01 0

01 0

1

ifififif

Example from IPOPT (Waechter)

Ui

Li

Ui

Lii

U

Li

Li

Ui

Lii

L

iU

iiU

iL

iiL

iU

iL

Li

UiL

i

i

xxxxx

xxxxxwhere

ni

xxx

xxx

xxxx

x

x

,maxˆ

,minˆ:

...1,

îfˆ

îfˆ

ˆîf2 Currently in MOOCHO :

> 40 vector operations!

Many different and unusual vector operations are needed by interior point methods for optimization!

Vector Reduction/Transformation Operators Defined

Reduction/Transformation Operators (RTOp) Defined

z 1i … z q

i opt( i , v 1i … v p

i , z 1i … z q

i ) element-wise transformation opr( i , v 1

i … v pi , z 1

i … z qi ) element-wise reduction

2 oprr( 1 , 2 ) reduction of intermediate reduction objects

• v 1 … v p R n : p non-mutable (constant) input vectors

• z 1 … z q

R n : q mutable (non-constant) input/output vectors• : reduction target object (many be non-scalar (e.g. {yk ,k}), or NULL)

Key to Optimal Performance

• opt(…) and opr(…) applied to entire sets of subvectors (i = a…b) independently:z 1

a:b … z qa:b , op( a, b , v 1

a:b … v pa:b , z 1

a:b … z qa:b , )

• Communication between sets of subvectors only for NULL, oprr( 1 , 2 ) 2

RTOpT

apply_op(in sv[1..p], inout sz[1…q], inout beta )

ROpDotProd

apply_op(… )

TOpAssignVectors

apply_op(… )

…

Vector

applyOp( in : RTOpT, apply_op(in v[1..p] , inout z[1…q], inout beta)

SerialVectorStd

applyOp( … )

MPIVectorStd

applyOp( … )

Software Implementation (see RTOp paper on Trilinos website “Publications”)

• “Visitor” design pattern (a.k.a. function forwarding)

…

Technicalities about TSFCore Software

All interfaces are templated on Scalar type Currently supported types: real (float, double), complex (std::complex<float>,

std::complex<double>), extended precision (GNU MP mpf_class) Teuchos::ScalarTraits<> used for all math functions Teuchos::RawMPITraits<> and Teuchos::PrimativeTypeTraits<> used for MPI

Smart reference-counted pointer class Teuchos::RefCountPtr<> used for all dynamic memory management

C++ exception handling mechanism used to throw exceptions derived from std::exception TSFCore uses const everywhere religiously often resulting in const and non-const versions of

many operations Many operations have default implementations based on very few pure virtual methods RTOpT operators (and wrapper functions) are provided for many common level-1 vector and multi-

vector operations Supported platforms (Includes all SIERRA and ICC platforms): Yey Teuchos!

Linux/g++: g++ 3.1 … 3.3.x, software.sandia.gov, CS LAN DEC: Compaq C++ V6.5-014, stratus.sandia.gov, CS LAN Linux/Intel: Intel C++ version 7.1, liberty.sandia.gov, ICC Sun: Forte Developer 7 C++ 5.4 Patch 111715-14, sass3276a.sandia.gov, EngSci LAN SGI: MIPSpro Compilers: Version 7.4.2m, sasg1099.sandia.gov, EngSci LAN IBM: VisualAge C++ Professional for AIX, Version 7, saia028.sandia.gov, EngSci LAN (not yet) Janus (ASC Red): ciCC = pgCC Rel 3.1-4i, sasn100.sandia.gov, EngSci LAN

TSFCore Interface C++ Specification (pure virtual functions)

template<class Scalar> class VectorSpace {public: virtual Index dim() const = 0; virtual bool isCompatible( const VectorSpace<Scalar>& vecSpc ) const = 0; virtual Teuchos::RefCountPtr< Vector<Scalar> > createMember() const = 0; …};

template<class Scalar> class OpBase {public: virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > domain() const = 0; virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > range() const = 0; virtual bool opSupported(ETransp M_trans) const;};

template<class Scalar> class LinearOp : virtual public OpBase<Scalar> {public: virtual void apply( const ETransp M_trans, const Vector<Scalar> &x ,Vector<Scalar> *y ,const Scalar alpha=1.0 ,const Scalar beta=0.0 ) const = 0; …};

template<class Scalar> class MultiVector : virtual public LinearOp<Scalar> {public: virtual Teuchos::RefCountPtr<Vector<Scalar> > col(Index j) = 0; …};

template<class Scalar> class Vector : virtual public MultiVector<Scalar> {public: virtual Teuchos::RefCountPtr< const VectorSpace<Scalar> > space() const = 0; virtual void applyOp( const RTOpPack::RTOpT<Scalar> &op, const size_t num_vecs ,const Vector<Scalar>* vecs[], const size_t num_targ_vecs ,Vector<Scalar>* targ_vecs[] ,RTOpPack::ReductTarget *reduct_obj ,const Index first_ele ,const Index sub_dim ,const Index global_offset ) const = 0; …};

Shows only pure virtual functions on interfaces that must be implemented

Default MultiVector implementation based on a Vector implementation

Default Serial Vector, VectorSpace and VectorSpaceFactory implementation for domain space of MultiVectors for default MultiVectors

Only two nontrivial functions to override if MultiVectors not needed!

– LinearOp::apply(…)

– Vector::applyOp(…)

Providing an optimized MultiVector implementation requires overriding more functions

However, most concrete subclasses should not derive from these base cases directly!

TSFCore Interface C++ Specification (explicit element access)

template<class Scalar>class MultiVector : virtual public LinearOp<Scalar> {public: … virtual void getSubMultiVector( const Range1D &rowRng, const Range1D &colRng ,RTOpPack::SubMultiVectorT<Scalar> *sub_mv ) const; virtual void freeSubMultiVector( RTOpPack::SubMultiVectorT<Scalar>* sub_mv ) const; virtual void getSubMultiVector( const Range1D &rowRng, const Range1D &colRng ,RTOpPack::MutableSubMultiVectorT<Scalar> *sub_mv ); virtual void commitSubMultiVector( RTOpPack::MutableSubMultiVectorT<Scalar>* sub_mv ); …};

template<class Scalar>class Vector : virtual public MultiVector<Scalar> {public: … virtual void getSubVector( const Range1D& rng, RTOpPack::SubVectorT<Scalar>* sub_vec ) const; virtual void freeSubVector( RTOpPack::SubVectorT<Scalar>* sub_vec ) const; virtual void getSubVector( const Range1D& rng, RTOpPack::MutableSubVectorT<Scalar>* sub_vec ); virtual void commitSubVector( RTOpPack::MutableSubVectorT<Scalar>* sub_vec ); virtual void setSubVector( const RTOpPack::SparseSubVectorT<Scalar>& sub_vec ); …};

• Explicit sub-views are requested and released in separate function calls (allows creation of temporary storage data with copies)

• All have default implementations in terms of advanced RTOp subclasses

• In general, explicit access to Vector and MultiVector elements is an extremely bad thing to do (i.e. client/server), however:

• Use cases where explicit element access is needed or is convenient– Debugging small problems (i.e. printing Vector or MultiVector elements)

– Vectors and MultiVectors created from the domain space of a MultiVector (i.e. block GMRES)

– Automatic interoperability of serial and MPI SPMD Vector and MultiVector objects.

Three Use Cases for Fundamental TSFCore Interfaces

1. TSFCore as interoperability layer for linear ANA objects (i.e. linear operators, vectors, multi-vectors, reduction/transformation operators)

2. Development of concrete TSFCore implementation subclasses

1. Support for decoupling application-defined scalar products form concrete vector spaces

2. Support for concrete implementations

1. Support and concrete subclasses for serial shared-memory platforms

2. Support and concrete subclasses for MPI SPMD distributed memory platforms

3. Concrete “adatper” subclasses for Epetra objects

3. Using TSFCore objects for the development of ANAs

1. C++ wrapper functions for prewritten RTOpT subclasses

e.g. norm(x), assign(&y,x) …

2. Composite objects:

1. Product vectors, multi-vectors and vector spaces

e.g. x = [ v1; v2; v3; … ; vn ];

2. Composite linear operators

1. Additive linear operators: e.g. M = A + B + C + D;

2. Multiplicative linear operators: e.g. M = A * B * C * D;

3. Block linear linear operators: e.g. M = [ A, B ; C, D ];

3. Handle classes for operator overloading (currently in TSFExtended)

e.g. y = alpha*trans(A)*x + gamma*B*z + beta*y

Trilinos TSFCore website

TSFCore Subclasses for Serial and MPI Platforms

Support base classes for serial vector space, vector and multi-vector subclasses SerialVectorSpaceBase, SerialVectorBase, SerialMultiVectorBase Provide complete implementation of all functions (utilizes level-3 BLAS from Teuchos) in terms of

explicit element access provided by subclasses Universal interoperability of Serial Vectors and MultiVectors:

All Vector and MultiVector objects that return VectorSpace objects with isInCore()==true and dim()==dim() are automatically compatible through explicit element sub-views!

Standard concrete implementations of serial vector space, vector and multi-vector subclasses SerialVectorSpaceStd, SerialVectorStd, SerialMultiVectorStd Provide general implementations that are very efficient default implementations Note: Most serial applications only need to implement linear operators (e.g. derived from

SerialLinearOpBase) Support base classes for MPI SPMD vector space, vector and multi-vector subclasses

MPIVectorSpaceBase, MPIVectorBase, MPIMultiVectorBase Provide complete implementation of all functions (utilizes level-3 BLAS from Teuchos) in terms of

explicit element access provided by subclasses Universal interoperability of MPI SPMD Vectors and MultiVectors:

All Vector and MultiVector objects that return VectorSpace objects that support the MPIVectorSpaceBase interface with same numbers of local elements are automatically compatible through explicit element sub-views!

Standard concrete implementations of MPI SPMD vector space, vector and multi-vector subclasses MPIVectorSpaceStd, MPIVectorStd, MPIMultiVectorStd Provide general implementations that are very efficient default implementations Note: Most MPI applications only need to implement linear operators (e.g. derived from

MPILinearOpBase)

TSF Composite ANA Linear Algebra Subclasses

VectorSpaceblocks

1…m

ProductVectorSpace

Vectorblocks

1…m

ProductVector

<<create>>

“Composite” subclasses allow a collection of objects to be manipulated as one object Product vector spaces and product vectors:

Product vector spaces: Product vectors:

Block linear operator:

Multiplicative (composed) linear operator:

Additive (summation) linear operator:

blockRange

blockDomain LinearOp

BlockLinearOp

blocks

1…*

MultiplicativeLinearOp

1…*

AdditiveLinearOp

1…*

Extending Functionality Beyond TSFCore

Question: How to extend functionality beyond TSFCore interfaces?

Examples of extended functionality Explicit matrix transposition: i.e. A = AT

Preconditioner creation: i.e. A P, where P A-1

LinearOp-LinearOp multiplication: i.e. C = A * B, where C is formed explicitly

Possible approaches to extend functionality in an ANA

Extended mix-in interfaces inherited by concrete linear algebra subclasses Explicit matrix transposition : e.g. TSFExtended::ExplicitlyTransposableOp Preconditioner creation: e.g. TSFExtended::ILUFactorizableOp LinearOp-LinearOp multiplication: e.g. through TSFExtended::RowAccessableOp (Not an ANA!)

“Strategy” interfaces and subclasses that are external from linear algebra subclasses Example: Some ANA needs LinearOp-LinearOp multiplication C = A * B with A and B being

Epetra_VbrMatrix objects with large element blocks (level-3 BLAS) and with output C being a Epetra_CrsMatrix (EpetraExt?)!

TSFCore::LinearOp

TSFCore::EpetraLinearOp

Epetra_Operator

Epetra_CrsMatrix Epetra_VbrMatrix

LinearOpMultiplierStrategy

EpetraVbrToCrsMultiplier

SomeANA

multiply( in A, in B, out C)

Strategy interfaces offer the greater flexibility, better efficiency and preserve ANAs!

Dependencies between TSF-related Software

“ANA to linear algebra” handles for operator

overloading(e.g. y = A*x)(currently in

TSFExtended)

“ANA to linear algebra” composite subclasses

(e.g. ProductVector)(currently in TSFCore and

TSFExtended)

“ANA to linear algebra” concrete implementation

support subclasses(e.g. MPIVectorBase)

(currently in TSFCore)

Interfaces to steady-state nonlinear applications

(currently in TSFCore::Nonlin,similar to NOX Group)

Interfaces to iterative linear solvers

(currently in Belos?)

Interfaces to transient nonlinear applications

(currently do not exist?)

Various extended “strategy” interfaces

(???)

???

“ANA to linear algebra” Interfaces

(e.g. Vector, LinearOp)(currently in TSFCore)

Extended APP-LAL and LAL-LAL interfaces to linear algebra objects(e.g. LoadableVector)

(currently in TSFExtended)

Extended APP-LAL and LAL-LAL handles

(currently in TSFExtended)

???

Outline






Interfacing Packages to Implementations : Everyone for themselves!

Package 1

Package Interface 1

Package Interface 1 / Implementation 1

adapter

…

Implementation 1 Implementation 2


adapter

Package Interface 1 / Implementation N

adapter

Implementation N…

Package 2

Package Interface 2


adapter


adapter

Package Interface 2 / Implementation N

adapter

Package M

Package Interface M

Package Interface M / Implementation 1

adpater

Package Interface M / Implementation 2

adapter

Package Interface M / Implementation N

adapter

… … …

- Oh no, M * N adapter subclasses needed!- This is not a “scalable” approach!Package == ANA

Interfacing Packages to Implementations : Standard Interfaces (I)

Package 1

Package Interface 1

…

Implementation 1 Implementation 2 Implementation N…

Package 2

Package Interface 2

Package M

Package Interface M

+ Only M + N adapter subclasses needed!+ This is a “scalable” approach!

Std LANA Interface

Std LANA Interface / Implementation 1

adapter


adapter

Std LANA Interface / Implementation N

adapter

Package Interface 1 / Std LANA Interface

adapter


adapter

Package Interface M / Std LANA Interface

adapter

…

Interfacing Packages to Implementations : Standard Interfaces (II)

Package 1 …

Implementation 1 Implementation 2 Implementation N…

Package 2 Package M

+ Only N adapter subclasses needed!+ Less interface documentation, shared testing code

Std LANA Interface


adapter


adapter

Std LANA Interface / Implementation N

adapter

…

Interfacing Packages to Packages : Everyone for themselves!

Package 1

Package Interface 1

…

Package 2

Package Interface 2

Package Interface 2 / Package Interface 1

adapter

Package 3

Package Interface 3


adapter


adapter

- Major assumption: “Package Interface 1” satisfies the requirements of “Package Interface 2”?

Package M

- Oh no, as many as 1 + 2 + 3 + … + M-1 = O(M2) adapter subclasses needed if only one-way interactions!

Concrete Examples:

1) NOX/LOCA interfaces had to be extended to support Belos interfaces

2) NOX/LOCA vector interface does not support MOOCHO vector interface

Package Interface M

…

- This is not a “scalable” approach!

Interfacing Packages to Packages : Standard Interfaces (I)

Package Interface 1

…Package 2

Package Interface 2

Implementation 1

Std LANA Interface


adapter


adapter

Implementation 2


adapter

Extended LANA Interface 1


adapter

Package Interface 1 / Std LANA

Interface adapter

…

+ Only M adapters need+ Each Package knows nothing of other packages

- Result: Destroys ability to dynamic cast to “Extended LANA Inteface1” !

Package M

- Major assumption!!!“Package Interface 1” supports functionality of “Std LANA Interface”?

Concrete Example:

Belos::BlockGmres using Epetra adapters defines a TSFCore::LinearOp subclass to be used as a preconditioner in another package that also uses Epetra adapters.

Package 1

Concrete Example Problem : Belos, Epetra and TSFCore

TSFCore::LinearOp



Belos::Operator


TSFCore::MPIVectorSpaceBase

Belos::TSFCoreOperator

Belos::TSFCoreMultiVec

TSFCore::MPIMultiVectorBase

Belos::BlockGmres

<<create>>

The problem:• TSFCore::BelosLinearOp::range() returns a TSFCore::BelosVectorSpace object• The TSFCore::BelosVectorSpace object creates a TSFCore::BelosMultiVector object• MPIMultiVectorBase::applyOp( … ) is called on EpetraMultiVector object passing in the TSFCore::BelosMultiVector object• TSFCore::BelosMultiVector::range() is called which returns a TSFCore::BelosVectorSpace object• The dynamic cast from TSFCore::BelosVectorSpace to TSFCore::MPIVectorSpaceBase fails

TSFCore::BelosMultiVectorTSFCore::BelosVectorSpace

range, domainrange, domain

The scenario: Create a MultiVector object use the range space of aTSFCore::BelosLinearOp object and assign it to a MultiVector object created from a TSFCore::EpetraVectorSpace.

range, domainTSFCore::EpetraVectorSpace

TSFCore::EpetraMultiVector

<<create>>

Belos::MultiVec

TSFCore::BelosLinearOp

The Delema: What to use for range and domain spaces?

Define TSFCore::VectorSpace only in terms of Belos interfaces!

Concrete Example Solution : Belos, Epetra and TSFCore

TSFCore::LinearOp



Belos::Operator


TSFCore::MPIVectorSpaceBase

Belos::TSFCoreOperator

Belos::TSFCoreMultiVec

TSFCore::MPIMultiVectorBase

Belos::BlockGmres

The solution:• TSFCore::BelosLinearOp::range() returns a TSFCore::EpetraVectorSpace object• The TSFCore::EpetraVectorSpace object creates a TSFCore::EpetraMultiVector object• MPIMultiVectorBase::applyOp( … ) is called on EpetraMultiVector object passing in the TSFCore::EpetraMultiVector object• TSFCore::EpetraMultiVector::range() is called which returns a TSFCore::EpetraVectorSpace object• The dynamic cast from TSFCore::EpetraVectorSpace to TSFCore::MPIVectorSpaceBase succeeds

The scenario: Create a MultiVector object use the range space of aTSFCore::BelosLinearOp object and assign it to a MultiVector object created from a TSFCore::EpetraVectorSpace.

range, domainTSFCore::EpetraVectorSpace

TSFCore::EpetraMultiVector

<<create>>

Belos::MultiVec

The Delema: What to use for range and domain spaces?

Define TSFCore::VectorSpace in terms of underlying TSFCore::VectorSpace!

TSFCore::BelosLinearOp

Interfacing Packages to Packages : Standard Interfaces (II)

Package Interface 1

…Package 2

Package Interface 2

Implementation 1

Std LANA Interface


adapter


adapter

Implementation 2


adapter

Extended LANA Interface 1


adapter

…

+ Fewer new adapters needed

+ Dynamic casting of most objects maintained

Package Interface 1 / Std LANA

Interface adapter

The Recommended Practice!

Package 1

Pros and Cons of Standard Interfaces

Reasons to adopt “standard” interfaces

• “Automatic” interoperability only comes through “standard” interfaces

• Only way to guarantee interoperability is even possible (without revisions)

• One set of documentation

• Common set of unit tests for common interface objects

Reasons not to adopt “standard” interfaces

• Greater package autonomy / fewer dependencies

• Fewer layers of indirection within a single package

• Less sophisticated object-oriented concepts within a single package

Outline






Recommendations for TSF and Trilinos

Goal: Adopt “standard interfaces” but minimize dependencies

• Minimize “standard” interface code needed for interoperability

• Encourage use of “standard” interfaces but allow packages to keep their own interfaces as local APIs

• All ASC/internal uses of Trilinos packages should adhere to “standard interfaces”

• Encourage academic and external collaborators to use “standard interfaces” but offer local package interfaces as an alternative

• Partition “standard” interface code as much as possible to minimize dependencies:

• Linear ANA interfaces (currently in TSFCore)

• SPMD Matrix/Vector loading and accessing (currently in TSFExtended)

• Interfaces to linear solvers (being developed in Belos)

• Interfaces to steady-state nonlinear problems (currently in TSFCore::Nonlin)

• Interfaces to transient non-linear problems (not yet developed)

• Separate “convenience” code from “standard interface” code

• Composite LANA objects (i.e. product vectors) (currently in TSFCore)

• Composite APP/LAL objects (currently in TSFCore and TSFExtended)

• Handle classes and operator overloading (currently in TSFExtended)

Open Questions related to TSF

• How do we make it “easiest” to develop concrete TSF interface implementations of vector spaces, vectors, multi-vectors and linear operators?

• Is it reasonable to expect beginner C++ programmers to debug though all TSF code?

Analogy: Should beginning Microsoft Windows users be able to debug though Windows code when something goes wrong?

Example: STL is good but difficult to impossible to debug through

• With that said, is the code in TSFCore and TSFExtended too complicated for semi-experienced C++ developers to debug through?

• What type of documentation for TSF is required beyond what is already available?

• How should the code in TSF be organized in to packages and managed in packages?

• Should all linear algebra interfaces be put into a single package?

• Should handles for operator overloading automatically include matrix/vector loading and extraction interfaces?

• Should certain types of code be allowed to be disabled at configure time?

The End

Let the Discussion Begin!

Extra Slides

TSFCore Support for APP-specific Scalar Products

Linear operators must “obey” application-specific scalar product <x,y>

Specifically, for the linear operator (i.e. LinearOp, MultiVector etc.):

the following adjoint relation must hold:

Goal of TSFCore support subclasses

=> Separate the definition of application-specific scalar product from data-structure and computing platform concrete implementations of vector spaces, vectors, multi-vectors and linear operators as much as possible.

VectorSpace LinearOp

ScalarProd

EuclideanScalarProd LinearOpScalarProd

rangeScalarProdVecSpc, domainScalarProdVecSpc

ScalarProdVectorSpaceBaseEuclideanLinearOpBase

op

AppSpecificScalarProd…

ConcreteMultiVector …

ConcreteVectorSpace …

Different Approaches for Developing Concrete Subclasses

Approach 1: Define “simpler” interfaces and include standard “adapter” subclasses• Example: NOX::Abstract::Group, NOX::Epetra::Group and NOX::Epetra::InterfaceApproach 2: Define templated adapter subclasses with “simpler” traits classes• Example: fei::Vector, snl_fei::Vector<> and snl_fei::VectorTraits<>Approach 3: Define support base classes that override “general” functions in terms of “simpler” pure

virtual functions which adapter subclasses define• Currently used by TSFCore => Example: TSFCore::SerialVectorBase, TSFCore::MPIVectorBase• Advantages

• Provides tailored “simple” set of functions to override• Subclasses have full control to override every virtual function, not just “simpler” functions

• Disadvantages• Subclass developers will see the “general” functions even though they should, in most cases,

ignore themGeneral Vector

…

Serial Vector Implementation 1

General Serial Vector Base


MPI Vector Implementation 1


General MPI Vector Base


Adapter


Adapter


Adapter


Adapter

…

…

……


Approach 1: Define “simpler” interfaces and include standard “adapter” subclasses• Example: NOX::Abstract::Group, NOX::Epetra::Group and NOX::Epetra::Interface• Advantages

• Provides tailored “simple” interfaces for subclass development for specific categories of implementations

• Subclass developers never even see “general” interface• Disadvantages

• Only operations in “simple” interface can be overridden, not in “general” interface.

General Vector

…


General Serial Vector Adapter




General MPI Vector Adapter

Simple Serial Vector Base

Simple MPI Vector Base


Adapter


Adapter


Adapter


Adapter

…

…

……


Approach 2: Define templated adapter subclasses with “simpler” traits classes• Example: fei::Vector, snl_fei::Vector<> and snl_fei::VectorTraits<>• Advantages

• Provides tailored “simple” traits interfaces and does not require writing subclasses• Less subclasses than Approach 1• Subclass developers never even see “general” interface

• Disadvantages• Only operations in “simple” traits interface can be overridden, not in “general” interface • Templated traits may or may not be more complicated than using general inheritance• Traits can not maintain state for a particular object while general subclasses can

General Vector


General Serial Vector Adapter




General MPI Vector Adapter

…

……

ConcreteType ConcreteType

• Create traits class specializations for all concrete implementations• Instantiate templated adapters on the concrete implementation types

Uses SerialVectorTraits<> Uses MPIVectorTraits<>

TSF and Abstract Interfaces in Trilinos TSFCore, TSFExtended … Roscoe A. Bartlett Department 9211:...

Documents

Transcript of TSF and Abstract Interfaces in Trilinos TSFCore, TSFExtended … Roscoe A. Bartlett Department 9211:...