Parallel Computing with MATLAB -...

© 2

008 T

he M

ath

Work

s, In

c.

® ®

Parallel Computing with MATLAB

Jamie Winter, The MathWorks Account Manager

Sarah Wait Zaranek Ph.D, Senior Application Engineer

2

® ®

Solving Big Technical Problems

Large data set

Challenges

Long running

Computationally

intensive

Wait

Load data onto

multiple machines

that work together

in parallel

Solutions

Run similar tasks

on independent

processors in

parallel

Reduce size

of problem

You could…

3

® ®


TOOLBOXES

BLOCKSETS

CPUWorker

CPUWorker

CPUWorkerCPU

Worker

CPUWorker

4

® ®

Agenda

Develop parallel code interactively

Task parallel applications for faster processing

Data parallel applications for handling large data sets

Schedule your programs to run

Tips on scheduling parallel code

5

® ®

Implicit Multithreaded MATLAB

Toolbox Support:Optimization Toolbox ™

Genetic Algorithm and Direct Search ™

Bioinformatics Toolbox ™

Model Calibration Toolbox ™

SystemTest™

Simulink Response Optimization™

parfor

job and tasks


No code changes

Trivial changes

Extensive changes

Task Parallel Data Parallel

spmd

codistributed

MATLAB and MPI

9

® ®

Agenda






10

® ®

Distributing Tasks (Task Parallel)

Time Time

Pro

cesses

11

® ®

Demo: Monte Carlo Simulation of Coin

Tossing

11

Number of Heads Out of 20

7 12 15 7 9 9 7 8 1211

10 Simulations of Flipping 20 Coins at a Time

12

® ®

Parallel for-Loops

parfor i = 1 : n

% do something with i

end

Mix task-parallel and serial code in the same function

Run loops on a pool of MATLAB resources

Iterations must be order-independent

M-Lint analysis helps in converting existing for-loops into

to parfor-loops

13

® ®

Agenda






14

® ®

Large Data Sets (Data Parallel)

11 26 41

12 27 42

13 28 43

14 29 44

15 30 45

16 31 46

17 32 47

17 33 48

19 34 49

20 35 50

21 36 51

22 37 52

11 26 41

12 27 42

13 28 43

14 29 44

15 30 45

16 31 46

17 32 47

17 33 48

19 34 49

20 35 50

21 36 51

22 37 52

15

® ®

spmd

spmd : single program, multiple data

Primarily used for data parallel applications

distributed arrays

mpi functionality

Runs using a matlabpool

Data stays on workers between blocks

Allows users to interleave serial and data parallel MATLAB code

x = 1

spmd

y = x+1

end

y

16

® ®

Distributed Arrays and Parallel Algorithms

Distributed arrays

Store segments of data across participating workers

Create from any built-in class in MATLAB

Examples: doubles, sparse, logicals, cell arrays, and arrays of

structs

Parallel algorithms for distributed arrays

Matrix manipulation operations

Examples: indexing, data type conversion, and transpose

Parallel linear algebra functions, such as svd and lu

Data distribution

Automatic, specify your own, or change at any time

17

® ®

Enhanced MATLAB Functions That

Operate on Distributed Arrays

18

® ®

MPI-Based Functions in

Parallel Computing Toolbox™

Use when a high degree of control over parallel algorithm is required

High-level abstractions of MPI functions

labSendReceive, labBroadcast, and others

Send, receive, and broadcast any data type in MATLAB

Automatic bookkeeping

Setup: communication, ranks, etc.

Error detection: deadlocks and miscommunications

Pluggable

Use any MPI implementation that is binary-compatible with MPICH2

19

® ®

Using the matlabpool

SPMD … END

A MATLAB block

Worker communication

Mostly Data Parallel

PARFOR … END

A MATLAB block

Independent iterations (Task Parallel)

MATLAB turns user code into parallel

code.

TOOLBOXES

BLOCKSETS

20

® ®

Agenda






21

® ®

Scheduler

CPU

CPU

CPU

CPU

Worker

Worker

Worker

Worker

Computer Cluster

MATLAB Distributed Computing Server

Scheduler

Client Machine

Third-Party Scheduler (Condor)

Parallel

Computing

ToolboxTOOLBOXES

BLOCKSETS

Third-Party

Scheduler

22

® ®

Demo: Scheduled Monte Carlo Coin

>> batch

23

® ®

Compute Cluster

CPU

CPU

CPU

CPU

MATLAB® Distributed Computing Server™

Scheduler

Result

Result

Result

Result

Client Machine

Task

Task

Task

Task

Worker

Worker

Worker

Worker

Distributed Applications

Parallel

Computing

ToolboxTOOLBOXES

BLOCKSETS

Result

Job

24

® ®

25

® ®

26

® ®

Demo: Scheduled Monte Carlo Coin

>> createJob

>> createTask

27

® ®

Dependencies

job – FileDependencies

Files are copied from client to each worker machine

Zip compressed

Uncompressed and added to the MATLAB path

Convenient for .m files, but can be slow for large data files

job – PathDependencies

Shared directories are added to the MATLAB path

Mixing of Windows® and UNIX® paths allowed

Reduces the amount of data transfer from client to cluster

28

® ®

Configurations

Save environment-specific parameters for your cluster

Benefits

Enter cluster information only once

Modify configurations without changing MATLAB code

Apply multiple configurations when running within same session

29

® ®

Agenda






30

® ®

Factors to Consider for Scheduling

Share code and data with workers efficiently using FileDependencies or PathDependencies

There is always an overhead to distribution Don’t make a task too small

Combine small repetitive function calls into one larger one

Vectorize task creation

Minimize IO Use JobData if jobs share data

Use Workspace for batch

Use Diary or CaptureCommandWindowOutput

31

® ®

Development and Debugging Process

Run serial code normally

on local machine

Task or

Data Parallel?

parfor / Jobs and Tasksspmd, pmode

distributed Arrays

Run on N cluster nodes

Task Parallel Data Parallel

Run on local workers

32

® ®

Parallel Computing with MATLAB -...

Documents

Transcript of Parallel Computing with MATLAB -...