Investigating Scaling Behaviour ofMonte Carlo Codes

for Dense Matrix Inversion

V. Alexandrov (BSC, Spain)J. Straßburg (BSC,Spain & UoR, UK)

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Outline

MotivationBackground on Linear Algebra Problems and Monte CarloScaling experimentsConclusions

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Motivation

Predict behaviour on different systemFind bottlenecks, sweet spot, scaling problemsEasier then running on several machinesReproducible

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Scientific and Engineering Problems

Many scientific problems revolve around:inverting a real n by n matrix (MI)

• Given A• Find A−1

solving a system of linear algebraic equations (SLAE)• Given A and b• Solve, for x , Ax = b• Or find A−1 and calculate x = A−1b

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Traditional Methods• Gaussian elimination• Gauss-Jordan• Both take O(n3) steps

Time prohibitive if:• a large problem• or a real time solution is required

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Idea behind Monte Carlo Methods

Wish to estimate the quantity α

Define a random variable ξ

Where ξ has the mathematical expectation α

Take N independent realisations ξi of ξ

• Then ξ̄ = 1N

N∑i=1

ξi

• And ξ̄ ≈ α

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Reason for Using Monte Carlo

O(NT ) steps to find an element of the:• matrix inverse A−1

• solution vector xwhere:

• N number of Markov Chains• T length of Markov Chains

Independent of n - size of matrix or problemAlgorithms can be efficiently parallelised

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Parallel Algorithm

Matrix Setup

Use parallel Monte Carloto find B−1

Use parallel iterativerefinement to improve

accuracy of B−1

MASTER SETUP

SLAVECALCS

SLAVECALCS

SLAVECALCS

MASTER COLLECTS

INVERSE

MASTER DISTRIBUTES

SLAVECALCS

SLAVECALCS

SLAVECALCS

MASTER COLLECTS

DECISION

INPUT

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Native vs. simulated scaling

1

10

100

1000

10000

5 10 15 20

tim

e

cores

500x500

1000x1000

2000x2000

3000x3000

0.1

1

10

100

1000

10000

5 10 15 20

tim

e

cores

500x500

1000x1000

2000x2000

3000x3000

24 core SMP development machinesimilar on ORNL cluster

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Core scaling

0

500

1000

1500

2000

2500

3000

3500

4000

4500

1 2 4 8 16 32 64 128 256 512 1024 2048

Exe

cuti

on

Tim

e in

Se

con

ds

Simulated MPI Processes

500 1000 2000 3000 4000

960 core system240 cores for simulation due to memory bandwidth restrictions

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Core scaling

1

10

100

1000

1 2 4 8 16 32 64 128 256 512 1024 2048

Exe

cuti

on

Tim

e in

Se

con

ds

Simulated MPI Processes

500 1000 2000 3000 4000

Recurring behaviour for increasing MPI process sizesScales well, then plateaus

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Problem scaling

1

10

100

1000

500 1000 2000 3000 4000

Exe

cuti

on

Tim

e in

Se

con

ds

Problem Size

1 2 4 8 16 32 64 128 256 512 1024 2048

Linear behaviour up to 2000x2000 matrix sizeSlight degradation for larger problem sizes

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

MPI message count scaling

0.E+00

5.E+03

1.E+04

2.E+04

2.E+04

3.E+04

3.E+04

1 2 4 8 16 32 64 128 256 512 1024 2048

MP

I Me

ssag

e C

ou

nt

Simulated MPI Processes

500 1000 2000 3000 4000

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1 2 4 8 16 32 64 128 256 512 1024 2048

MP

I Me

ssag

e C

ou

nt

Simulated MPI Processes

500 1000 2000 3000 4000

Simulator also gathers MPI statisticsLinear increase of exchanged messages, as expected

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Outcome & Conclusions

Behaviour of code is predictableSimulation provides valuable informationForecast behaviour on varying systems possibleTime and resource concerns

Outline Motivation Linear Algebra Monte Carlo Scaling experiments

Future work

Improve the code and retestProfiling + scaling simulationOptimize program to be able to handle simulation of largerproblem sizesRuns on larger parallel systems, different networks