CS 515: Parallel Algorithms

Chandrima Sarkar

Atanu Roy

2010 - 02 -17

Agenda

• Architecture

• Parallel Programming Languages

• Precedence Graph

• Elementary Parallel Algorithms

• Sorting

• Matrix Multiplication

Download :- http://www.cs.montana.edu/~atanu.roy/Classes/CS515.html

Architecture

Flynn’s Classification S = single , M = multiple , I = instruction (stream), D = data (stream)

SISD SIMD

Architecture

Flynn’s Classification S = single , M = multiple , I = instruction (stream), D = data (stream)

MISD MIMD

Static Inter-connection Network

Linear Array

Ring

Ring arranged to use short wires

Fully Connected Topology

Chordal ring

Multidimensional Meshes and Torus

Tree

Tree Cont.

FAT TREE

STAR

Hypercube

1-D 2-D 3-D 4-D

001 011

000 010

100 110

111 101

0-D

5-D

Parallel Programming Languages

Control Mechanism Communication Mechanism

Shared Memory Message-passing

Control driven Fortran 90/HPF , C++ , HEP PL/I , Ada , Concurrent Pascal Modula-2 , MultiLisp (MIMD), Lisp Connection Machine (SIMD)

CSP , Ada , OCCAM (Von Neumann Language Extension )

Data driven VAL , ID LAU , SISAL ( data-flow languages )

Pattern driven Concurrent Prolog ( Shapiro )

Actors

Demand driven ( reduction language )

FP

Dijkstra’s High Level language construct

• Degree of Parallelism is static Algol-68,CSP

A parbegin C begin B parbegin D E parend G end parend H

Precendence Graph

Elementary Parallel Algorithms Finding sum using a 2D mesh architecture

Finding sum of 16 values in a Shuffle Exchange SIMD Model

Parallel summation in a Hypercube SIMD Model

Broadcast in a Hypercube Algorithm 1

Algorithm 2

Odd Even Transposition Sort

• (1) p = n • 14 – 5 – 15 – 8 – 4 – 11 – 13 – 12

• odd-even 14 5 – 15 8 – 4 11 – 13 12 • even-odd 14 – 5 15 – 4 8 – 11 13 – 12 • odd-even 5 14 – 4 15 – 8 11 – 12 13 • even-odd 5 – 4 14 – 8 15 – 11 12 – 13 • odd-even 4 5 – 8 14 – 11 15 – 12 13 • even-odd 4 – 5 8 – 11 14 – 12 15 – 13 • odd-even 4 5 – 8 11 – 12 14 – 13 15 • even-odd 4 – 5 8 – 11 12 – 13 14 – 15

Odd Even Transposition Sort (contd…)

• (2) p << n • S= {12, 7, 2, 4, 1, 11, 9, 5, 6, 3, 10, 8}, p = 4

P1 P2 P3 P4

{12, 7, 2} {4, 1, 11} {9, 5, 6} {3, 10, 8}

{2, 7, 12} {1, 4, 11} {5, 6, 9} {3, 8, 10}

{1, 2, 4} {7, 11, 12} {3, 5, 6} {8, 9, 10}

{1, 2, 4} {3, 5, 6} {7, 11, 12} {8, 9, 10}

{1, 2, 3} {4, 5, 6} {7, 8, 9} {10, 11, 12}

{1, 2, 3} {4, 5, 6} {7, 8, 9} {10, 11, 12}

Pseudocode

• Proc MERGE-SPLIT(S) for i:= 1 to p do in parallel

QUICKSORT(Si)

end for for (i := 1 to ceil(p/2)) for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for end for

2 – D mesh with Snake Order

Input : {23, 6, 1, 5, 11, 13, 55, 19, -3, 12, -5, -7, 9, 55, 28, -2}

Thompson and Kung (1977)

Snake Order (contd.)

Bitonic Merge Sort

• Bitonic Sequence :- 1, 3, 7, 8 6, 5, 4, 2

• Comparator

• Note :- Batcher’s Bitonic Merge Sort compares elements whose indices differ by a single bit.

Bitonic Merge Sort

Shuffle-Exchange Network

Bitonic Mergesort on Shuffle-Exchange Network

• A list of n = 2k unsorted elements can be sorted in time θ(lg2 n) with a network 2k-1[k (k-1) + 1] comparators using the shuffle-exchange network.

Sorting Network

Odd Even Merging Network

Systolic Matrix Multiplication

1. Multiply ai,k by ak,j

2. Add the result to ri,j

3. Send ai,k to cell ci+1,j

4. Send bk,j to cell ci,j+1

Home Work

• Show how the following 16 values would be sorted by Batcher’s Bitonic sort.

16, 7, 4, 12, 2, 10, 13, 9, 1, 8, 11, 3, 15, 6, 5, 14