Patterns for Parallel Programming

Markos Chandras

Salil Ponde

Wenjiang Xu

COMP60611: Fundamentals of Parallel and Distributed Systems (2010)School of Computer Science

University of Manchester

Distributed data moving mechanisms

Reduced synchronization overhead

Fine-grain parallelism

Difficulties in construction and managment of memories for data matching

Load balancing problems in large systems

Input/output data define the task interface

Hidden implementation details

No relationship among tasks is explicitly defined

The scheduler manages the tasks via the interface

Optimized scheduling using data packing,, data reuse

Fault tolerated, task migration

References

The 'data-driven' model

1) Dan I. Moldovan (1993) - ”'Parallel Processing – From Applications to Systems” (page 399)2) V. D. Tran, L. Hluchý, G. T. Nguyen - “Parallel Programming with Data Driven Model” ( Parallel and Distributed Processing, 2000. Proceedings. 8th Euromicro Workshop, Rhodos, Greece, 2000 )

Dealing effectively with large arrays

Array elements should be close to the Processing Element (PE) which processes it

Ways of distribution

(a) One Dimensional

(b) Two Dimesional

(c) Block Cyclic

Problem is defined in terms of global array indices

Program is written in terms of local array indices

Choosing the way of distribution

References

The Distributed Array Pattern

1) Timothy G. Mattson; Beverly A. Sanders; Berna L. Massingill (2004) - ”'Patterns for Parallel Programming”

Semi-independent tasks interact in an irregular fashion.

(a) interaction is determined by the flow of data

(b) the data implies the ordering constrains between the task

(c) direct graph with cycles

Solution: events & tasks

(a) Task: receive event-->process event -->(possibly) send events

(b) Event Flow: need a form of asychronous communication of events

message-passing: from task that generate the event to task that process it

(c) Avoiding deadlock (1) more informational (2) timeouts

(d) One task per PE or Multiple task (need good load balance)

(e) Communication if events should be efficient

References

The Event-Based Coordination Pattern

1) Timothy G. Mattson; Beverly A. Sanders; Berna L. Massingill (2004) - ”'Patterns for Parallel Programming”

List of ReferencesDan I. Moldovan (1993). Parallel Processing – From Applications to Systems. Morgan Kaufmann Publishers Inc.  San Francisco, CA, USA

V. D. Tran, L. Hluchý, G. T. Nguyen Parallel Programming with Data Driven Model. Parallel and Distributed Processing, 2000. Proceedings. 8th Euromicro Workshop, Rhodos, Greece, 2000

Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill (2004). Patterns for Parallel Programming. Addison-Wesley