High performance computing

Sri ManakulaVinayagar Engineering College, Puducherry

Department of Computer Science and Engineering

Subject Name: HIGH PERFORMANCE COMPUTING Subject Code: CS T82

Prepared By :Mr.B.Thiyagarajan,AP/CSEMr.D.Saravanan, AP/CSE

Verified by : Approved by :

UNIT – 1

Introduction: Need of high speed computing Increase the speed of computers History of parallel computers and recent parallel computers; Solving problems in parallel Temporal parallelism Data parallelism Comparison of temporal and data parallel processing Data parallel processing with specialized processors Inter-task dependency. The need for parallel computers Models of computation Analyzing algorithms Expressing algorithms.

TWO MARKS

1. What is high performance computing?

High Performance Computing most generally refers to the practice of aggregating computing power

in a way that delivers much higher performance than one could get out of a typical desktop computer

or workstation in order to solve large problems in science, engineering, or business.

2. Define Computing.

The process of utilizing computer technology to complete a task.

Computing may involve computer hardware and/or software, but must involve some form of a

computer system.

Most individuals use some form of computing every day whether they realize it or not.

3. What is Parallel Computing?

The simultaneous use of more than one processor or computer to solve a problem. Use of

multiple processors or computers working together on a common task.

Each processor works on its section of the problem

CS T82 High Performance Computing 1


Processors can exchange information

4. Why do we need parallel Computing?

Serial computing is too slow

Need for large amounts of memory not accessible by a single processor

To compute beyond the limits of single PU systems:

Achieve more performance;

Utilize more memory.

To be able to:

Solve that can't be solved in a reasonable time with single PU systems;

Solve problems that don’t fit on a single PU system or even a single system.

So we can:

Solve larger problems

Solve problems faster

Solve more problems

5. Why parallel Processing?

Single core performance growth has slowed.

More cost-effective to add multiple cores.

6. Write Limits of Parallel Computing.

Theoretical Upper Limits:

Amdahl's Law.

Gustafson’s Law

Practical Limits

Load Balancing.

Non-computational sections.

Other considerations:

Time to develop/rewrite code.

Time do debug and optimize code

7. How to Classify Shared and Distributed Memory?



All processors have access to a pool of

shared memory

Memory is local to each processor

Access times vary from CPU to CPU

in NUMA systems

Data exchange by message passing

over a network

Example: SGI Altix, IBM P5 nodes Example: Clusters with single-socket

blades

8. Define Hybrid system

A limited number, N, of processors have access to a common pool of shared memory

To use more than N processors requires data exchange over a network

Example: Cluster with multi-socket blades

9. Define Multi Core Systems.

Extension of hybrid model

Communication details increasingly complex

Cache access

Main memory access

Quick Path / Hyper Transport socket connections

Node to node connection via network

10. Define Accelerated Systems

Calculations made in both CPU and accelerator

Provide abundance of low-cost flops

Typically communicate over PCI-e bus

Load balancing critical for performance



11. What is data parallelism? (Apr2013)

Data parallelism is a form of parallelization of computing across multiple processors in parallel

computing environments. Data parallelism focuses on distributing the data across different parallel

computing nodes. It contrasts to task parallelism as another form of parallelism.

12. Define Stored Program Concept.

Memory is used to store both program and data instructions

Program instructions are coded data which tell the computer to do something

Data is simply information to be used by the program

A central processing unit (CPU) gets instructions and/or data from memory, decodes the

instructions and then sequentially performs them.

13. What is Parallel Computing?(April 2013)

Parallel computing is the simultaneous use of multiple compute resources to solve a computational

problem. The compute resources can include:

A single computer with multiple processors;

An arbitrary number of computers connected by a network;

A combination of both.

14. Why we use Parallel Computing?

There are two primary reasons for using parallel computing:

Save time - wall clock time


15. Why do we need high speed Parallel Computing?

The traditional scientific paradigm is first to do theory (say on paper), and then lab experiments to

confirm or deny the theory. The traditional engineering paradigm is first to do a design (say on



paper), and then build a laboratory prototype. Both paradigms are being replacing by numerical

experiments and numerical prototyping. There are several reasons for this.

1. Real phenomena are too complicated to model on paper (eg. climate prediction).

2. Real experiments are too hard, too expensive, too slow, or too dangerous for a laboratory (eg

oil reservoir simulation, large wind tunnels, overall aircraft design, galactic evolution, whole

factory or product life cycle design and optimization, etc.).

16. How do we increase the speed of Parallel Computers?

We can increase the speed of computers in several ways.

By increasing the speed of the processing element using faster semiconductor technology (by

using advanced technology)

By architecture methods. It in turn we can increase the speed of computer by applying

parallelism.

Use parallelism in Single processor

Overlap the execution of number of instructions by pipelining or by using multiple functional

units

Overlap the operation of different units.

Use parallelism in the problem

Use number of interconnected processors to work cooperatively to solve the problem.

17. Define Temporal Parallelism?

Use parallelism in Single processor. Temporal means Pertaining to time.

Overlap the execution of number of instructions by pipelining or by using multiple functional

units


18. State the advantages & Disadvantages of temporal Parallelism?

Synchronization: Identical time

Bubbles in pipeline Bubbles are formed

Fault tolerance: does not tolerate.

Inter task communication small

Scalability: Can’t be increased.

19. State the advantages & Disadvantages of Data Parallelism?

Advantages:

No Synchronization:

No Bubbles in pipeline

More Fault tolerance

No communication

Disadvantages:



Static assignment

Partitionable

Time to divide jobs is small.

20. Define Data Parallelism?

Use parallelism in the problem: use number of interconnected processors to work

cooperatively to solve the problem. Here input is divided into some set of jobs and each job is given

to one Processor and works simultaneously.

21. Define Inter task Dependency?

The following assumptions are made in evolving tasks to teacher:

The answer to a question is independent of answers to other questions

Teachers do not have to interact

The same instructions are used to grad all answer books

Tasks are inter related. Some tasks are done independently and simultaneously while others

have to wait for completion of previous tasks.

22. What is parallel Computer? (Apr 2013)

Parallel computing is a form of computation in which many calculations are carried out

simultaneously, operating on the principle that large problems can often be divided into smaller ones,

which are then solved concurrently.

23. Comparison Between Temporal and Data Parallelism. (November 2013)

TEMPORAL PARALLELISM DATA PARALLELISM

Independent task

Tasks take equal time

Bubbles leads to idling

Task assignment is static

Not tolerant to processor

Efficient with fine grained

Full jobs are assigned

Tasks take different time

No bubbles

Task assignment is static,

dynamic or quasi dynamic

Tolerates to processor

Efficient with coarse grained

11 Marks

1. What is Parallel Computing and explain?

Additionally, software has been written for serial computation:

To be executed by a single computer having a single Central Processing Unit (CPU);

Problems are solved by a series of instructions, executed one after the other by the CPU. Only

one instruction may be executed at any moment in time.



In the simplest sense, parallel computing is the simultaneous use of multiple compute

resources to solve a computational problem.

The compute resources can include:

A single computer with multiple processors;

An arbitrary number of computers connected by a network;

A combination of both.

The computational problem usually demonstrates characteristics such as the ability to be:

Broken apart into discrete pieces of work that can be solved simultaneously;

Execute multiple program instructions at any moment in time;

Solved in less time with multiple compute resources than with a single compute resource.

Parallel computing is an evolution of serial computing that attempts to emulate what has

always been the state of affairs in the natural world: many complex, interrelated events

happening at the same time, yet within a sequence. Some examples:

Planetary and galactic orbits

Weather and ocean patterns

Tectonic plate drift

Rush hour traffic in LA

Automobile assembly line

Daily operations within a business

Building a shopping mall

Ordering a hamburger at the drive through.

Traditionally, parallel computing has been considered to be "the high end of computing" and

has been motivated by numerical simulations of complex systems and "Grand Challenge

Problems" such as:

weather and climate

chemical and nuclear reactions

biological, human genome

geological, seismic activity

mechanical devices - from prosthetics to spacecraft

electronic circuits

manufacturing processes



Today, commercial applications are providing an equal or greater driving force in the

development of faster computers. These applications require the processing of large amounts

of data in sophisticated ways. Example applications include:

parallel databases, data mining

oil exploration

web search engines, web based business services

computer-aided diagnosis in medicine

management of national and multi-national corporations

advanced graphics and virtual reality, particularly in the entertainment industry

networked video and multi-media technologies

collaborative work environments

Ultimately, parallel computing is an attempt to maximize the infinite but seemingly scarce

commodity called time.

2. Why do we need Use Parallel Computing? (April 2013)

There are two primary reasons for using parallel computing:

Save time - wall clock time


Other reasons might include:

Taking advantage of non-local resources - using available compute resources on a wide

area network, or even the Internet when local compute resources are scarce.

Cost savings - using multiple "cheap" computing resources instead of paying for time on a

supercomputer.

Overcoming memory constraints - single computers have very finite memory resources.

For large problems, using the memories of multiple computers may overcome this

obstacle.

Limits to serial computing - both physical and practical reasons pose significant

constraints to simply building ever faster serial computers:

Transmission speeds - the speed of a serial computer is directly dependent upon how fast

data can move through hardware. Absolute limits are the speed of light (30

cm/nanosecond) and the transmission limit of copper wire (9 cm/nanosecond). Increasing

speeds necessitate increasing proximity of processing elements.



Limits to miniaturization - processor technology is allowing an increasing number of

transistors to be placed on a chip. However, even with molecular or atomic-level

components, a limit will be reached on how small components can be.

Economic limitations - it is increasingly expensive to make a single processor faster.

Using a larger number of moderately fast commodity processors to achieve the same (or

better) performance is less expensive.

The future: during the past 10 years, the trends indicated by ever faster networks,

distributed systems, and multi-processor computer architectures (even at the desktop level)

suggest that parallelism is the future of computing

3. What is the Need of high speed computing?

The traditional scientific paradigm is first to do theory (say on paper), and then lab

experiments to confirm or deny the theory.

The traditional engineering paradigm is first to do a design (say on paper), and then build a

laboratory prototype.

Both paradigms are being replacing by numerical experiments and numerical prototyping.

There are several reasons for this.

Real phenomena are too complicated to model on paper (eg. climate prediction).

Real experiments are too hard, too expensive, too slow, or too dangerous for a laboratory (eg

oil reservoir simulation, large wind tunnels, overall aircraft design, galactic evolution, whole

factory or product life cycle design and optimization, etc.).

Scientific and engineering problems requiring the most computing power to simulate are

commonly called "Grand Challenges like predicting the climate 50 years hence, are estimated

to require computers computing at the rate of 1 Tflop = 1 Teraflop = 10^12 floating point

operations per second, and with a memory size of 1 TB = 1 Terabyte = 10^12 bytes. Here is

some commonly used notation we will use to describe problem sizes:

1 Mflop = 1 Megaflop = 10^6 floating point operations per second

1 Gflop = 1 Gigaflop = 10^9 floating point operations per second

1 Tflop = 1 Teraflop = 10^12 floating point operations per second

1 MB = 1 Megabyte = 10^6 bytes

1 GB = 1 Gigabyte = 10^9 bytes

1 TB = 1 Terabyte = 10^12 bytes

1 PB = 1 Petabyte = 10^15 bytes

4. How do we Increase the speed of Computers.

We can increase the speed of computers in several ways.



By increasing the speed of the processing element using faster semiconductor technology (by using advanced technology)

By architecture methods. It in turn we can increase the speed of computer by applying parallelism.

Use parallelism in Single processor

overlap the execution of number of instructions by pipelining or by using multiple

functional units


Use parallelism in the problem

Use number of interconnected processors to work cooperatively to solve the problem.

5. Describe about History of parallel computers: A brief history of parallel computers are

given below

VECTOR SUPERCOMPUTERS:

Glory days: 76-90 Famous examples: Cray machines

Characterized by:

The fastest clock rates, because vector pipelines can be very simple.

Vector processing.

Quite good vectorizing compilers.

High price tag; small market share.

Not always scalable because of shared-memory bottleneck (vector processors need more data

per cycles than conventional processors). Vector processing is back in various forms: SIMD

extensions of commodity microprocessors (e.g. Intel's SSE), vector processors for game

consoles (Cell), multithreaded vector processors (Cray), etc.

Vector processors went down temporarily because of:

Market issues, price/performance, microprocessor revolution, commodity microprocessors.

Not enough parallelism for biggest problems. Hard to vectorize/parallelize automatically

Didn't scale down.

MPPs

Glory days: 90-96

Famous examples: Intel hypercubes and Paragon, TMC Connection Machine, IBM SP,

Cray/SGI T3E.



Characterized by:

Scalable interconnection network, up to 1000's of processors. We'll discuss these networks

shortly

Commodity (or at least, modest) microprocessors.

Message passing programming paradigm.

Killed by:

Small market niche, especially as a modest number of processors can do more and more.

Programming paradigm too hard.

Relatively slow communication (especially latency) compared to ever-faster processors (this

is actually no more and no less than another example of the memory wall).

Today

A state of flux in hardware.

But more stability in software, e.g., MPI and OpenMP.

Machines are being sold, and important problems are being solved, on all of the following:

Vector SMPs, e.g., Cray X1, Hitachi, Fujitsu, NEC.

SMPs and ccNUMA, e.g., Sun, IBM, HP, SGI, Dell, hundreds of custom boxes.

Distributed memory multiprocessors, e.g., Cray XT3, IBM Blue Gene.

Clusters: Beowulf (Linux) and many manufacturers and assemblers.

A complete top-down view: At the highest level you have either a distributed memory

architecture with a scalable interconnection network, or an SMP architecture with a bus.

A distributed memory architecture may or may not provide support for a global memory

consistency model (such as cache coherence, software distributed shared memory, coherent

RDMA, etc.). On an SMP architecture you expect hardware support for cache coherence.

A distributed memory architecture can be built from SMP or even (rarely) ccNUMA boxes.

Each box is treated as a tightly coupled node (with local processors and uniformly accessed

shared memory). Boxes communicate via message passing, or (less frequently) with hardware

or software memory coherence schemes. Both on distributed and on shared memory

architectures, the processors themselves may support an internal form of task or data

parallelism. Processors may be vector processors, commodity microprocessors with multiple

cores, or multiple threads multiplexed over a single core, heterogeneous multicore processors,

etc.

Programming: Typically MPI is supported over both distributed and shared-memory

substrates for portability (large existing base of code written and optimized in MPI). OpenMP



and POSIX threads are almost always available on SMPs and ccNUMA machines. OpenMP

implementations over distributed memory machines with software support for cache

coherence also exist, but scaling these implementation is hard and is a subject of ongoing

research.

Future

The end of Moore's Law?

Nanoscale electronics

Exotic architectures? Quantum, DNA/molecular. 6. Explain various methods for solving problems in parallel.

Method 1: Utilizing temporal parallelism

Solving a simple job can be solved in parallel in many ways.

Consider there are 1000 who appeared for the exam. There are 4 questions in each answer book. If a

teacher is to correct these answer books, the following instructions are given to them.

1. Take an answer book from the pile of answer books.

2. Correct the answer to Q1,namely A1.

3. Repeat step 2 for answers to Q2,Q3,Q4 namely A2,A3,A4.

4. Add marks.

5. Put answer book in pile of corrected answer books.

6. Repeat steps 1 to 5 until no answer books are left.

Ask 4 teachers to correct each answer book by sitting in one line.

The first teacher corrects answer Q1,namely A1 of first paper and passes the paper to the

second teacher.

When the first three papers are corrected, some are idle.

Time taken to correct A1=Time to correct A2= Time to correct A3= Time to correct A4=5

minutes. Then first answer book takes 20 min.

Total time taken to correct 1000 papers will be 20+(999*5)=5015 min.This is about 1/4th of

the time taken.

Temporal means pertaining to time.

The method is correct if:

Jobs are identical.

Independent tasks are possible.

Time is same.

No. of tasks is small compared to total no of jobs.



Let no of jobs=n

Time to do a job=p

Each job is divided into k tasks

Time for each task=p/k

Time to complete n jobs with no pipeline processing =np

Time complete n jobs with pipeline processing of k teachers=p+(n-1)p/k=p*[(k+n-1)/k]

Speedup due to pipeline processing=[np/p(k+n-1)/k]=[k/1+(k-1)/n]

Problems encountered:

Synchronization:

Identical time

Bubbles in pipeline

Bubbles are formed

Fault tolerance

Does not tolerate.

Inter task communication

small

Scalability

Cant be increased.

Method 2: Utilizing Data Parallelism

Divide the answer books into four piles and give one pile to each teacher.

Each teacher takes 20 min to correct an answer book, the time taken for 1000 papers is

5000 min.

Each teacher corrects 250 papers but simultaneously.

Let no of jobs=n

Time to do a job=p

Let there be k teachers

Time to distribute=kq

Time to complete n jobs by single teacher=np

Time to complete n jobs by k teachers=kq+np/k

Speed up due to parallel processing=np/kq+np/k=knp/k8k*q+np=k/1+(kq/np)

Advantages:

No Synchronization:

No Bubbles in pipeline

More Fault tolerance

No communication

Disadvantages:



Static assignment

Partitionable

Time to divide jobs is small

METHOD 3: Combined Temporal and Data Parallelism:

Combining method 1 and 2 gives this method.

Two pipelines of teachers are formed and each pipeline is given half of total no of jobs.

Halves the time taken by single pipeline.

Reduces time to complete set of jobs.

Very efficient for numerical computing in which a no of long vectors and large matrices are

used as data and could be processed.

METHOD 4: Data Parallelism With Dynamic Assignment

A head examiner gives one answer book to each teacher.

All teachers simultaneously correct the paper.

A teacher who completes goes to head examiner for another paper.

If second completes at the same time, then he queues up in front of head examiner.

Advantages:

Balancing of the work assigned to each teacher.

Teacher is not forced to be idle.

No bubbles

Overall time is minimized

Disadvantages:

Teachers Have To Wait In The Queue.

Head examiner can become bottle neck

Head examiner is idle after handing the papers.

Difficult to increase the number of teachers

If speedup of a method is directly proportional to the number, then the method is said to scale well.

Let total no of papers=n

Let there be k teachers

Time waited to get paper=q

Time for each teacher to get, grade and return a paper=(q+p)

Total time to correct papers by k teachers=[n(q+p)/k]

Speed up due to parallel processing=np/[n(q+p)/k]=k/[1+(q/p)]

METHOD 5: Data Parallelism with Quasi-Dynamic Scheduling



Method 4 can be made better by giving each teacher unequal sets of papers to correct. Teacher 1,2,3,4

may be given with 7, 9, 11, 13 papers. When finish that further papers will be given. This randomizes

the job completion and reduces the probability of queue. Each job is much smaller compared to the

time to actually do the job. This method is in between purely static and purely dynamic schedule. The

jobs are coarser grain in the sense that a bunch of jobs are assigned and the completion time will be

more than if one job is assigned.

Comparison between temporal and data parallelism:

TEMPORAL PARALLELISM DATA PARALLELISM

Independent task

Tasks take equal time

Bubbles leads to idling

Task assignment is static

Not tolerant to processor

Efficient with fine grained

Full jobs are assigned

Tasks take different time

No bubbles

Task assignment is static, dynamic

or quasi dynamic

Tolerates to processor

Efficient with coarse grained

Data parallel processing with specialized processor:

Data Parallel Processing is more tolerant but requires each teacher to be capable of correcting

answers to all questions with equal case.

METHOD 6: Specialist Data Parallelism

There is a head examiner who dispatches answer papers to teachers. We assume that teacher 1(T1)

grades A1, teacher 2(T2) grades A2 and teacher i(Ti) grades Ai to question Qi.

Procedure:

Give one answer book to T1,T2,T3,T4

When a corrected answer paper is returned check if all questions are graded. If yes add marks

and put the paper in the output pile.

If no check which questions are not graded

For each I,if Ai is ungraded and teacher Ti is idle send it to teacher Ti or if any other teacher

Tp is idle.

Repeat steps 2,3 and 4 until no answer paper remains in input pile

METHOD 7: Coarse Grained Specialist Temporal Parallelism

All teachers are independently and simultaneously at their pace. That teacher will end up spending a

lot of time inefficiently waiting for other teachers to complete their work.

Procedure:

Answer papers are divided into 4 equal piles and put in the in-trays of each teacher. Each teacher

repeats 4 times simultaneously steps 1 to 5.



For teachers Ti(i=1 to 4) do in parallel

Take an answer paper from in-tray

Grade answer Ai to question Qi and put in out-tray

Repeat steps 1 and 2 till no papers are left

Check if teacher (i+1)mod4’s in-tray is empty.

As soon as it is empty, empty own out-tray into in-tray of that teacher.

METHOD 8: Agenda Parallelism

Answer book is thought as an agenda of questions to be graded. All teachers are asked to work on the

first item on agenda, namely grade the answer to first question in all papers. Head examiner gives one

paper to each teacher and asks him to grade the answer A1 to Q1.When a teacher finishes this, he is

given with another paper. This is data parallel method with dynamic schedule and fine grain tasks.

7. Briefly explain about Inter Task Dependency with example

The following assumptions are made in evolving tasks to teacher:

The answer to a question is independent of answers to other questions

Teachers do not have to interact

The same instructions are used to grad all answer books

Tasks are inter related. Some tasks are done independently and simultaneously while others have to

wait for completion of previous tasks. The inter relations of various tasks of a job may be represented

graphically as a task graph.

Procedure: Recipe for Chinese vegetable fried rice:

T1: Clean and wash rice

T2: Boil water in a vessel with 1 teaspoon salt

T3: Put rice in boiling water with some oil and cook till soft

T4: Drain rice and cool

T5: Wash and scrape carrots

T6: Wash and string French beans

T7: Boil water with ½ teaspoon salt in 2 vessels

T8: Drop carrots and French beans in boiling water

T9: Drain and cool carrots and French beans

T10: Dice carrots

T11: Dice French beans

T12: Peel onions and dice into small pieces

T13: Clean cauliflower .Cut into small pieces.

T14: Heat oil in iron pan and fry diced onion cauliflower for 1 min in heated oil

T15: Add diced carrots and French beans to above and fry for 2 min.

T16: Add cooled cooked rice, chopped onions and soya sauce to the above and stir and fry for 5 min.



There are 16 tasks in this, in that they have to be carried out in sequence. A graph showing the

relationship among the tasks is given

8. Explain the Various Computation Models? (November 2013)

RAM

PRAM (parallel RAM)

Interconnection Network

Combinatorial Circuits

Parallel and Distributed Computation

Many interconnected processors working concurrently

Connection machine

Internet

Types of multiprocessing frameworks

Parallel

Distributed

Technical aspects

Parallel computers (usually) work in tight syncrony, share memory to a large extent and have

a very fast and reliable communication mechanism between them.

Distributed computers are more independent, communication is less Frequent and less

synchronous, and the cooperation is limited.

Purposes

Parallel computers cooperate to solve more efficiently (possibly) Difficult problems

Distributed computers have individual goals and private activities. Sometime communications

with other ones are needed. (e. G. Distributed data base operations).

The RAM Sequential Model

RAM is an acronym for Random Access Machine

RAM consists of

A memory with M locations.

Size of M can be as large as needed.

A processor operating under the control of a sequential program which can

load data from memory

store date into memory

execute arithmetic & logical computations on data.

A memory access unit (MAU) that creates a path from the processor to an arbitrary

memory location.

RAM Sequential Algorithm Steps



A READ phase in which the processor reads datum from a memory location and copies it into

a register.

A COMPUTE phase in which a processor performs a basic operation on data from one or two

of its registers.

A WRITE phase in which the processor copies the contents of an internal register into a

memory location.

PRAM (Parallel Random Access Machine)

Let P1, P2 , ... , Pn be identical processors

Each processor is a RAM processor with a private local memory.

The processors communicate using m shared (or global) memory locations, U1, U2, ..., Um.

Allowing both local & global memory is typical in model study.

Each Pi can read or write to each of the m shared memory locations.

All processors operate synchronously (i.e. using same clock), but can execute a different

sequence of instructions.

Some authors inaccurately restrict PRAM to simultaneously executing the same

sequence of instructions (i.e., SIMD fashion)

Each processor has a unique index called, the processor ID, which can be referenced by the

processor’s program.

Often an unstated assumption for a parallel model

Each PRAM step consists of three phases, executed in the following order:

A read phase in which each processor may read a value from shared memory

A compute phase in which each processor may perform basic arithmetic/logical

operations on their local data.

A write phase where each processor may write a value to shared memory.

Note that this prevents reads and writes from being simultaneous.

Above requires a PRAM step to be sufficiently long to allow processors to do different

arithmetic/logic operations simultaneously.

PRAM Memory Access Methods

Exclusive Read (ER): Two or more processors can not simultaneously read the same memory

location.

Concurrent Read (CR): Any number of processors can read the same memory location

simultaneously.

Exclusive Write (EW): Two or more processors can not write to the same memory location

simultaneously.



Concurrent Write (CW): Any number of processors can write to the same memory location

simultaneously.

Variants for Concurrent Write

Priority CW: The processor with the highest priority writes its value into a memory location.

Common CW: Processors writing to a common memory location succeed only if they write

the same value.

Arbitrary CW: When more than one value is written to the same location, any one of these

values (e.g., one with lowest processor ID) is stored in memory.

Random CW: One of the processors is randomly selected write its value into memory.

UNIT – IQUESTION BANK

TWO MARKS1. What is high performance computing?

2. Define Computing.

3. What is Parallel Computing?

4. Why do we need parallel Computing?

5. Why parallel Processing?

6. Write Limits of Parallel Computing.

7. How to Classify Shared and Distributed Memory?

8. Define Hybrid system

9. Define Multi Core Systems.

10. Define Accelerated Systems

3. What is data parallelism? (Apr 2013)

4. Define Stored Program Concept.

5. What is Parallel Computing? (April 2013)

6. Why we use Parallel Computing?

7. Why do we need high speed Parallel Computing?

8. How do we increase the speed of Parallel Computers?

9. Define Temporal Parallelism?

10. State the advantages & Disadvantages of temporal Parallelism?

11. State the advantages & Disadvantages of Data Parallelism?

12. Define Data Parallelism?

13. Define Inter task Dependency?

14. What is parallel Computer? (Apr 2013)

15. Comparison Between Temporal and Data Parallelism. (November 2013)



11 Marks

1. What is Parallel Computing and explain?

2. Why do we need Use Parallel Computing? (April 2013) ( Ref.Pg.No.8)

3. What is the Need of high speed computing?

4. How do we Increase the speed of Computers.

5. Describe about History of parallel computers:

6. Explain various methods for solving problems in parallel. (November 2013)(Ref.Pg.No.:12)

7. Briefly explain about Inter Task Dependency with example

8. Explain the Various Computation Models?(November 2013)(Ref.Pg.No.18)



UNIT – II

Parallel Programming Platforms:

Trends in microprocessor architectures

Limitations of memory system performance

Parallel computing platforms

Communication costs in parallel machines

Routing mechanisms for interconnection networks.

Principles of Parallel Algorithm Design:

Preliminaries

Decomposition techniques

Characteristics of tasks and interactions

Mapping techniques for load balancing

Methods for containing interaction overheads

Parallel algorithm models.

Basic Communication Operations:

One-to-all broadcast and all-to-one reduction

All-to-all broadcast reduction

All-reduce and prefix-sum operations

Scatter and gather

all-to-all personalized communication

Circular shift

Improving the speed of some communication operations.

2 MARKS

1. Write the trends in microprocessor architecture?

A computer is a system which processes data according to a specified algorithm. It contains

One or more programmable (in the sense that the user can specify its operation) digital

processors, also called central processing units (CPUs), memory for storing data and

instructions, and input and output devices.

2. What is the limitation of memory system performance?

The effective performance of a program on a computer relies not just on the speed of the

processor but also on the ability of the memory system to feed data to the processor. At the

logical level, a memory system, possibly consisting of multiple levels of caches, takes in a

request for a memory word and returns a block of data of size b containing the requested word



after l nanoseconds. Here, l is referred to as the latency of the memory. The rate at which data

can be pumped from the memory to the processor determines the bandwidth of the memory

system.

3. What is the parallel computing platform?

Traditionally, software has been written for serial computation

To be run on a single computer having a single Central Processing Unit (CPU);

A problem is broken into a discrete series of instructions.

Instructions are executed one after another.

Only one instruction may execute at any moment in time.

4. What is the communication cost in parallel machine? (April 2013)

One of the major overheads in the execution of parallel programs arises from communication

of information between processing elements. The cost of communication is dependent on a

variety of features including the programming model semantics, the network topology, data

handling and routing, and associated software protocols

5. What is the Routing mechanism for interconnection networks? (November 2013)

A routing mechanism determines the path a message takes through the network to get from

the source to the destination node. It takes as input a message's source and destination nodes. It

may also use information about the state of the network. It returns one or more paths through

the network from the source to the destination node.

6. Write the characteristics of task and interaction?

Task generation

Task size

Knowledge of task size

Size and data associated with task.

7. What is the mapping technique for load balancing?

Once a computation has been decomposed into tasks, these tasks are mapped onto processes

with the objective that all tasks complete in the shortest amount of elapsed time.

In order to achieve a small execution time, the overheads of executing the tasks in

parallel must be minimized. For a given decomposition, there are two key sources of

overhead.

The time spent in inter-process interaction is one source of overhead. Another

important source of overhead is the time that some processes may spend being idle.

Some processes can be idle even before the overall computation is finished for a

variety of reasons. Uneven load distribution may cause some processes to finish earlier

than others.



At times, all the unfinished tasks mapped onto a process may be waiting for tasks

mapped onto other processes to finish in order satisfying the constraints imposed by

the task-dependency graph. Both interaction and idling are often a function of

mapping.

Therefore, a good mapping of tasks onto processes must strive to achieve the twin

objectives of (1) reducing the amount of time processes spend in interacting with each

other, and (2) reducing the total amount of time some processes are idle while the others

are engaged in performing some tasks.

8. Write the method for containing interaction overhead?

As noted earlier, reducing the interaction overhead among concurrent tasks is important for an

efficient parallel program. The overhead that a parallel program incurs due to interaction among

its processes depends on many factors, such as the volume of data exchanged during

interactions, the frequency of interaction, the spatial and temporal pattern of interactions, etc.

9. What is the Parallel algorithm model?

In computer science, a parallel algorithm or concurrent algorithm, as opposed to a

traditional sequential (or serial) algorithm, is an algorithm which can be executed a piece at a

time on many different processing devices, and then combined together again at the end to get

the correct result.

10. What is the One to all broadcast and all to one reduction?

Parallel algorithm often requires a single process to send identical data to all other processes

or to subset of them. This operation is known as one to all broadcast. Initially, only the source

process has the data size m that needs to be broadcast. At the termination of the procedure, there

are p copies of the initial data one belonging to each process. The dual of one to all broadcast is

all to one reduction.

11. What is the All to all broadcast and reduction?

All to all broadcast is generalization of one to all broadcast in which all p nodes

simultaneously initiate a broadcast. A process sends the same m-word message to every other

process, but different processes may broadcast different message.

12. Write the types of all to all broadcast reduction?

Ring or linear array

Mesh

Hyper cube

13. What is the Scatter and gather?

In scatter operation a single node sends a unique message of size m to every other node. This

operation is also known as one to all personalized communication. The dual of one to all

personalized communication or the scatter operation is the gather operation.


http://en.wikipedia.org/wiki/Algorithm

http://en.wikipedia.org/wiki/Sequential_algorithm

http://en.wikipedia.org/wiki/Computer_science


14. What is the All to all personalized communication?

In all too all personalized communication each node sends a distinct message of size m to

every other node. Each node sends different messages to different nodes, unlike all to all

broadcast, in which each node send message to all other nodes.

15. Define Circular shift.

Circular shift is a member of a broader class of global communication operation known as

permutation. A permutation is a simultaneous, one to one data redistribution operation in which

node sends a packet of m words to a unique node.

16. Write the types of improving the speed of some communication operation

Splitting and routing message in parts

All port communication

17. Define Decomposition?

The process of dividing a computation into smaller parts, some of all parts which potentially

be executed in parallel, is called as decomposition.

18. What is the Granularity, Concurrency and task generation?

The number and size of task into which a problem is decomposed determines the granularity

of decomposition. Decomposition into a large no of small task is called fine-grained and

decomposition into a small number of large tasks is called as coarse grained.

19. Write the Characteristic of inter task interaction?

Static versus dynamic

Regular versus irregular

Read only versus read writ

One way versus two way

20. Define Recursive decomposition.

Recursive decomposition is a method for inducing concurrency in problem that can be solved

using the divide and conquers strategy.

21. Define Data decomposition.

Data decomposition is a powerful and commonly used method for deriving concurrency in

algorithm that operates on large data structures. In this method the decomposition of

computation is done in two steps. In the first step, the data on which the computations are

performed is portioned and in the second step the data partitioning is used to induce a

partitioning of the computation into task.

22. Define exploratory decomposition.

Exploratory decomposition is used to decompose problems whose underlying computation

corresponds to search of a space for solution.



23. Define Speculative decomposition.

Speculative decomposition is used when a program may take one of many possible

computationally significant branches depending on the output of other computations that

precede it.

24. Write the types of parallel algorithm model?

Data parallel model

Task graph model

Work pool model

Master-slave model

Pipeline or Producer-Consumer model

Hybrid model

25. Write the Decomposition techniques?

Recursive Decomposition

Exploratory Decomposition

Hybrid Decomposition



11 MARKS

1. Explain the limitations of memory system performance ( April 2013)

The effective performance of a program on a computer relies not just on the speed of the processor but also on the ability of the memory system to feed data to the processor.

At the logical level, a memory system, possibly consisting of multiple levels of caches, takes in a request for a memory word and returns a block of data of size b containing the requested word after l nanoseconds.

Memory system, and not processor speed, is often the bottleneck for many applications. Memory system performance is largely captured by two parameters, latency and bandwidth. Latency is the time from the issue of a memory request to the time the data is available at the

processor. Bandwidth is the rate at which data can be pumped to the processor by the memory system. It is very important to understand the difference between latency and bandwidth. Consider the example of a fire-hose. If the water comes out of the hose two seconds after the

hydrant is turned on, the latency of the system is two seconds. Once the water starts flowing, if the hydrant delivers water at the rate of 5 litres/second, the

bandwidth of the system is 5 litres/second. If you want immediate response from the hydrant, it is important to reduce latency. If you want to fight big fires, you want high bandwidth.

1. Improving Effective Memory Latency Using Caches

Caches are small and fast memory elements between the processor and DRAM. This memory acts as a low-latency high-bandwidth storage. If a piece of data is repeatedly used, the effective latency of this memory system can be

reduced by the ache. The fraction of data references satisfied by the cache is called the cache hit ratio of the

computation on the system. Cache hit ratio achieved by a code on a memory system often determines its performance.

Example

Consider the architecture from the previous example. In this case, we introduce a cache of size 32 KB with a latency of 1 ns or one cycle. We use this setup to multiply two matrices A and B of dimensions 32 X 32. We have carefully chosen these numbers so that the cache is large enough to store matrices A and B, as well as the result matrix C.

2. Impact of Memory Bandwidth

Memory bandwidth is determined by the bandwidth of the memory bus as well as the memory units.

Memory bandwidth can be improved by increasing the size of memory blocks. The underlying system takes l time units (where l is the latency of the system) to deliver b

units of data where b is the block size)

Example

The vector column_sum is small and easily fits into the cache



The matrix b is accessed in a column order. The strided access results in very poor performance.

Multiplying a matrix with a vector: (a) multiplying column-by-column, keeping a running sum; (b) computing each element of the result as a dot product of a row of the matrix with the vector.

Thus by using spatial locality and temporal locality can increase it.

Exploiting spatial and temporal locality in applications is critical for amortizing memory latency and increasing effective memory bandwidth.

The ratio of the number of operations to number of memory accesses is a good indicator of anticipated tolerance to memory bandwidth.

Memory layouts and organizing computation appropriately can make a significant impact on the spatial and temporal locality.

3. Alternate Approaches for Hiding Memory Latency

Consider the problem of browsing the web on a very slow network connection. We deal with the problem in one of three possible ways:

We anticipate which pages we are going to browse ahead of time and issue requests for them in advance;

We open multiple browsers and access different pages in each browser, thus while we are waiting for one age to load, we could be reading others.

We access a whole bunch of pages in one go amortizing the latency across various accesses. The first approach is called prefetching, the second multithreading, and the third one

corresponds to spatial locality in accessing memory words.

4. Tradeoffs of Multithreading and Prefetching:

Bandwidth requirements of a multithreaded system may increase very significantly because of the smaller cache residency of each thread.

Multithreaded systems become bandwidth bound instead of latency bound. Multithreading and prefetching only address the latency problem and may often exacerbate

the bandwidth problem. Multithreading and prefetching also require significantly more hardware resources in the form

of storage.



2. Explain in detail about Parallel algorithm models. (April 2013)

An algorithm model is typically a way of structuring a parallel algorithm by selecting a de-

composition and mapping technique and applying the appropriate strategy to minimize

interactions.

The Data-Parallel Model

The data-parallel model is one of the simplest algorithm models.

In this model, the tasks are statically or semi-statically mapped onto processes and

each task performs similar operations on different data.

This type of parallelism that is a result of identical operations being applied

concurrently on different data items is called data parallelism.

Data-parallel computation phases are interspersed with interactions to synchronize the

tasks or to get fresh data to the tasks.

Since all tasks perform similar computations, the decomposition of the problem into

tasks is usually based on data partitioning because a uniform partitioning of data

followed by a static mapping is sufficient to guarantee load balance.

Data-parallel algorithms can be implemented in both shared-address-space and

message passing paradigms.

The Task Graph Model

The task-dependency graph may be either trivial, as in the case of matrix

multiplication, or nontrivial parallel algorithms, the task dependency graph is

explicitly used in mapping.

In the task graph model, the interrelationships among the tasks are utilized to promote

locality or to reduce interaction costs.

Sometimes a decentralized dynamic mapping may be used, but even then, the mapping

uses the information about the task-dependency graph structure and the interaction

pattern of tasks to minimize interaction overhead.

Example: Sparse matrix factorization and many parallel algorithms derived via divide-

and conquer decomposition. This type of parallelism that is naturally expressed by

independent tasks in a task-dependency graph is called task parallelism.

The Work Pool Model

The work pool or the task pool model is characterized by a dynamic mapping of tasks

onto processes for load balancing in which any task may potentially be performed by

any process.

There is no desired premapping of tasks onto processes.

The mapping may be centralized or decentralized.



Pointers to the tasks may be stored in a physically shared list, priority queue, hash

table, or tree, or they could be stored in a physically distributed data structure.

The work may be statically available in the beginning, or could be dynamically

generated; i.e., the processes may generate work and add it to the global work pool.

If the work is generated dynamically and a decentralized mapping is used, then a

termination detection algorithm

In the message-passing paradigm, the work pool model is typically used when the

amount of data associated with tasks is relatively small compared to the computation

associated with the tasks.

In,Parallel tree search the work is represented by a centralized or distributed data

structure is an example of the use of the work pool model where the tasks are

generated dynamically.

The Master-Slave Model

In the master-slave or the manager-worker model, one or more master processes

generate work and allocate it to worker processes.

The tasks may be allocated by the manager for load balancing workers are assigned

smaller pieces of work at different times.

The manager-worker model can be generalized to the hierarchical or multi-level

manager-worker model in which the top level manager feeds large chunks of tasks to

second-level managers, who further subdivide the tasks among their own workers and

may perform part of the work themselves.

This model is generally equally suitable to shared-address-space or message-passing

paradigms since the interaction is naturally two-way; i.e., the manager knows that it

needs to give out work and workers know that they need to get work from the

manager.

While using the master-slave model, care should be taken to ensure that the master

does not become a bottleneck, which may happen if the tasks are too small

It may also reduce waiting times if the nature of requests from workers is

nondeterministic.

The Pipeline or Producer-Consumer Model

In the pipeline model, a stream of data is passed on through a succession of processes,

each of which performs some task on it. This simultaneous execution of different

programs on a data stream is called stream parallelism.



The processes could form such pipelines in the shape of linear or multidimensional

arrays, trees, or general graphs with or without cycles. A pipeline is a chain of

producers and consumers.

Each process in the pipeline can be viewed as a consumer of a sequence of data items

for the process preceding it in the pipeline and as a producer of data for the process

following it in the pipeline.

The pipeline does not need to be a linear chain; it can be a directed graph.

The pipeline model usually involves a static mapping of tasks onto processes.

An example of a two-dimensional pipeline is the parallel LU factorization algorithm,

Hybrid Models

More than one model to be combined, resulting in a hybrid algorithm model.

A hybrid model may be composed either of multiple models applied hierarchically or

multiple models applied sequentially to different phases of a parallel algorithm.

3. Explain decomposition techniques used to achieve concurrency with necessary examples.

(November 2013)

Splitting or diving the computations to be performed into a set of tasks for concurrent execution defined by the task-dependency graph.

These techniques are broadly classified as Recursive decomposition Data-decomposition, Exploratory decomposition Speculative decomposition. Hybrid decomposition The recursive- and data decomposition techniques are relatively general purpose as

they can be used to decompose a wide variety of problems. On the other hand, speculative- and exploratory-decomposition techniques are more of

a special purpose nature because they apply to specific classes of problems. Recursive Decomposition

Recursive decomposition is a method for inducing concurrency in problems that can be solved using the divide-and-conquer strategy.

In this technique, a problem is solved by first dividing it into a set of independent sub problems. Each one of these sub problems is solved by recursively applying a similar division into smaller sub problems followed by a combination of their results.

The divide-and-conquer strategy results in natural concurrency, as different sub problems can be solved concurrently.

Example Quick sort: The quick sort task-dependency graph based on recursive decomposition for sorting a sequence of 12 numbers.



Algorithm: A recursive program for finding the minimum in an array of numbers A of length n.

o procedure RECURSIVE_MIN (A, n)o begino if (n = 1) theno min := A[0];o elseo lmin:= RECURSIVE_MIN (A, n/2);o rmin:= RECURSIVE_MIN (&(A[n/2]), n - n/2);o if (lmin<rmin) theno min := lmin;o elseo min := rmin;o endelse;o endelse;o return min;o end RECURSIVE_MIN

Data Decomposition Data decomposition is a powerful and commonly used method for deriving

concurrency in algorithm that operate on large data structures. In this method, the decomposition of computations is done in two steps. In the first step, the data on which the computations are performed is partitioned, and

in the second step, this data partitioning is used to induce a partitioning of the computations into tasks. Example matrix multiplication, LU Factorization.

Types: Partitioning Output Data Partitioning Input Data Partitioning both Input Data& Output Data Partitioning intermediate Data

Partitioning Output Data In many computations, each element of the output can be computed independently of

others as a function of the input. In such computations, a partitioning of the output data automatically induces a

decomposition of the problems into tasks, where each task is assigned the work of computing a portion of the output.



Partitioning Input Data Partitioning of output data can be performed only if each output can be naturally

computed as a function of the input.Figure (a) Partitioning of input and output matrices into 2 x 2 Sub matrices. (b) A decomposition of matrix multiplication into four tasks based on the partitioning of the matrices in (a).

Partitioning Intermediate Data Algorithms are often structured as multi-stage computations such that the output of

one stage is the input to the subsequent stage. A decomposition of such an algorithm can be derived by partitioning the input or the

output data of an intermediate stage of the algorithm. Partitioning intermediate data can sometimes lead to higher concurrency than

partitioning input or output data.



The Owner-Computes Rule : Decomposition based on partitioning output or input data is also widely referred to as

the owner-computes rule.

Exploratory Decomposition Exploratory decomposition is used to decompose problems whose underlying

computations correspond to a search of a space for solutions. In exploratory decomposition, we partition the search space into smaller parts, and

search each one of these parts concurrently, until the desired solutions are found. Example 15-puzzle problem.

A 15-puzzle problem instance showing the initial configuration (a), the final configuration (d), and a sequence of moves leading from the initial to the final configuration.



The 15-puzzle is typically solved using tree-search techniques,one of these newly generated configurations must be closer to the solution by one move progress towards finding the solution.

The configuration space generated by the tree search is often referred to as a state space graph.

Each node of the graph is a configuration and each edge of the graph connects configurations that can be reached from one another by a single move of a tile.

Speculative Decomposition Speculative decomposition is used when a program may take one of many possible

computationally significant branches depending on the output of other computations. One task is performing the computation whose output is used in deciding the next

computation, other tasks can concurrently start the computations of the next stage. This is similar to evaluating one or more of the branches of a switch statement in C in

parallel before the input for the switch is available. A simple network for discrete event simulation.

Hybrid Decompositions Combining two or more decomposition to form a new model or new decomposition is

known as hybrid decomposition.Hybrid decomposition for finding the minimum of an array of size 16 using four tasks.



4. Discuss the all-to-all personalized communication on parallel computers with linear array, mesh and hypercube interconnection networks. (November 2013).

In all-to-all personalized communication, each node sends a distinct message of size m to every other node. Each node sends different messages to different nodes, unlike all-to-all broadcast, in which each node sends the same message to all other nodes.

All-to-all personalized communication is also known as total exchange. This operation is used in a variety of parallel algorithms such as fast Fourier transform, matrix

transpose, sample sort, and some parallel database join operations.

Ring:

The steps in an all-to-all personalized communication on a six-node linear array. To perform this operation, every node sends p - 1 pieces of data, each of size m.This shown in figure below

Figure: All-to-all personalized communication on a six-node ring. The label of each message is of the form {x, y}, where x is the label of the node that originally owned the message, and y is the label of the node that is the final destination of the message. The label ({x1, y1}, {x2, y2}, ..., {xn, yn}) indicates a message that is formed by concatenating n individual messages.



Mesh:

In all-to-all personalized communication on a mesh, each node first groups its p messages according to the columns of their destination nodes

Figure: The distribution of messages at the beginning of each phase of all-to-all personalized communication on a 3 x 3 mesh. At the end of the second phase, node i has messages ({0, i}, ...,

{8, i}), where 0 8. The groups of nodes communicating together in each phase are enclosed in dotted boundaries.



Therefore, the total time for all-to-all personalized communication of messages of size m on a p-node two-dimensional square mesh is

Hypercube:

One way of performing all-to-all personalized communication on a p-node hypercube is to simply extend the two-dimensional mesh algorithm to log p dimensions.



Cost Analysis

In the above hypercube algorithm for all-to-all personalized communication, mp/2 words of data are exchanged along the bidirectional channels in each of the log p iterations.

The resulting total communication time is



5. Discuss the shared address space Platforms.

There are two primary forms of data exchange between parallel tasks -accessing a shared data space and exchanging messages.

Platforms that provide a shared data space are called sharedaddress- space machines or multiprocessors.

Platforms that support messaging are also called message passing platforms or multicomputers.

Shared-Address-Space Platforms:

Part (or all) of the memory is accessible to all processors. Processors interact by modifying data objects stored in this shared-address-space. If the time taken by a processor to access any memory word in the system global or local is

identical, the platform is classified as a uniform memory access (UMA), else, a nonuniform memory access (NUMA) machine.

NUMA and UMA Shared-Address-Space Platforms

The distinction between NUMA and UMA platforms is important from the point of view of algorithm design. NUMA machines require locality from underlying algorithms for performance.

Programming these platforms is easier since reads and writes are implicitly visible to other processors.

However, read-write data to shared data must be coordinated (this will be discussed in greater detail when we talk about threads programming).

Caches in such machines require coordinated access to multiple copies. This leads to the cache coherence problem.

A weaker model of these machines provides an address map, but not coordinated access. These models are called non cache coherent shared address space machines.

NUMA and UM A shared address space platform figure:

Typical shared-address-space architectures: (a)Uniform-memory-access shared-address-space computer; (b)Uniform-memory-access shared-address-space computer with

caches and memories; (c) Non-uniform-memory-access shared-address-space computer with local memory only.

Shared-Address-Space vs. Shared Memory Machines

It is important to note the difference between the terms shared address space and shared memory.



We refer to the former as a programming abstraction and to the latter as a physical machine attribute.

It is possible to provide a shared address space using a physically distributed memory.

Message-Passing Platforms

These platforms comprise of a set of processors and their own (exclusive) memory. Instances of such a view come naturally from clustered workstations and non-shared-address-

space multicomputers. These platforms are programmed using (variants of) send and receive primitives. Libraries such as MPI and PVM provide such primitives.

Message Passing vs. Shared Address Space Platforms:

Message passing requires little hardware support, other than a network. Shared address space platforms can easily emulate message passing. The reverse is more

diffcult to do (in an effcient manner).

6. Explain the methods for containing interaction overheads. Interaction is defined as communication between processes or data exchange between sender

and receiver or data sharing between nodes. Overhead is the problems occurred during communication. There are several factors included in interaction overhead.

o Maximizing Data Localityo Minimize Volume of Data-Exchangeo Minimize Frequency of Interactionso Minimizing Contention and Hot Spotso Overlapping Computations with Interactionso Replicating Data or Computationso Using Optimized Collective Interaction Operations

Maximizing Data Locality The interaction overheads can be reduced by using techniques that promote the use of

local data or data that have been recently fetched. Data locality enhancing techniques encompass a wide range of schemes that try to

minimize the volume of nonlocal data that are accessed, maximize the reuse of recently accessed data, and minimize the frequency of accesses.

For example, in sparse matrix-vector multiplication. Minimize Volume of Data-Exchange

A fundamental technique for reducing the interaction overhead is to minimize the overall volume of shared data that needs to be accessed by concurrent processes. To maximizing the temporal data locality, i.e., making as many of the consecutive references to the same data as possible.

Minimizing Contention and Hot Spots Reducing interaction overheads by directly or indirectly reducing the frequency and

volume of data transfers. Data-access and inter-task interaction patterns can often lead to contention that can

increase the overall interaction overhead. Contention occurs when multiple tasks try to access the same resources concurrently. Multiple simultaneous transmissions of data over the same interconnection link,

multiple simultaneous accesses to the same memory block, or multiple processes sending messages to the same process at the same time, can all lead to contention.

This is because only one of the multiple operations can proceed at a time and the others are queued and proceed sequentially.



One way of reducing contention is to redesign the parallel algorithm to access data in contention-free patterns.

For the matrix multiplication algorithm, this contention can be eliminated by modifying the order in which the block multiplications

Replication of data or computations Replication of data or computations is another technique that may be useful in

reducing interaction overheads. In some parallel algorithms, multiple processes may require frequent read-only access

to shared data structure, such as a hash-table, in an irregular pattern. Replicate a copy of the shared data structure on each process so that after the initial

interaction during replication, all subsequent accesses to this data structure are free of any interaction overhead.

Therefore, the message-passing programming paradigm benefits the most from data replication, which may reduce interaction overhead

Data replication does not come without its own cost. Data replication increases the memory requirements of a parallel program. The

aggregate amount of memory required to store the replicated data increases linearly with the number of concurrent processes.

This may limit the size of the problem that can be solved on a given parallel computer. So data replication must be used selectively to replicate relatively small amounts of

data. Using Optimized Collective Interaction Operations

The interaction patterns among concurrent activities are static and regular. A class of such static and regular interaction patterns is those that are performed by

groups of tasks, and they are used to achieve regular data accesses or to perform certain type of computations on distributed data.

A number of key such collective interaction operations have been identified that appear frequently in many parallel algorithms.

Broadcasting some data to all the processes or adding up numbers, each belonging to a different process, are examples of such collective operations.

The collective data-sharing operations can be classified into three categories. The first category contains operations that are used by the tasks to access data. The second category of operations is used to perform some communication-intensive

computations, and the third category is used for synchronization. Highly optimized implementations of these collective operations have been developed

that minimize the overheads due to data transfer as well as contention. Optimized implementations of these operations are available in library form from the

vendors of most parallel computers, e.g., MPI (message passing interface ) Overlapping Interactions with Other Interactions

If the data-transfer capacity of the underlying hardware permits, then overlapping interactions between multiple pairs of processes can reduce the effective volume of communication.

As an example of overlapping interactions, consider the commonly used collective communication operation of one-to-all broadcast in a message-passing paradigm with four processes P0, P1, P2, and P3.

A commonly used algorithm to broadcast some data from P0 to all other processes works as:



In the first step, P0 sends the data to P2. In the second step, P0 sends the data to P1, and concurrently, P2 sends the same data that it had received from P0 to P3.

The entire operation is thus complete in two steps because the two interactions of the second step require only one time step.

On the other hand, Naive broadcast algorithm would send the data from P0 to P1 to P2 to P3.

Illustration of overlapping interactions in broadcasting data from one to four processes.

The naive broadcast algorithm may be adapted to actually increase the amount of overlap.

A parallel algorithm needs to broadcast four data structures one after the other. The entire interaction would require eight steps using the first two-step broadcast algorithm.

The naive algorithm accomplishes the interaction in only six steps as shown in figure. In the first step, P0 sends the first message to P1. In the second step P0 sends the second message to P1 while P1 simultaneously sends

the first message to P2. In the third step, P0 sends the third message to P1, P1 sends the second message to P2,

and P2 sends the first message to P3, similarly in a pipelined fashion, the last of the four messages is sent out of P0 after four steps and reaches P3 in six.

Since this method is rather expensive for a single broadcast operation, it is unlikely to be included in a collective communication library.

7. Elaborate on all – reduce and prefix-sum operations.

One of these operations is a third variation of reduction, in which each node starts with a buffer of size m and the final results of the operation are identical buffers of sizem on each node that are formed by combining the original p buffers using an associative operator.

Semantically, this operation, often referred to as the all-reduce operation, is identical to performing an all-to-one reduction followed by a one-to-all broadcast of the result.

This operation is different from all-to-all reduction, in which p simultaneous all-to-one reductions take place, each with a different destination for the result.

A simple method to perform all-reduce is to perform an all-to-one reduction followed by a one-to-all broadcast.

Therefore, the total communication time for all log p steps is



Algorithm : Prefix sums on a d-dimensional hypercube.

1. procedure PREFIX_SUMS_HCUBE(my_id, my number, d, result)

2. begin

3. result := my_number;

4. msg := result;

5. for i := 0 to d - 1 do

6. partner := my_id XOR 2i;

7. send msg to partner;

8. receive number from partner;

9. msg := msg + number;

10. if (partner < my_id) then result := result + number;

11. endfor;

12. end PREFIX_SUMS_HCUBE

Scatter and Gather

In the scatter operation, a single node sends a unique message of sizem to every other node. This operation is also known aso ne-to-all personalized communication.

One-to-all personalized communication is different from one-to-all broadcast in that the source node starts with p unique messages, one destined for each node. Unlike one-to-all broadcast, one-to-all personalized communication does not involve any duplication of data.

The dual of one-to-all personalized communication or the scatter operation is the gather operation, or concatenation, in which a single node collects a unique message from each node.

A gather operation is different from an all-to-one reduce operation in that it does not involve any combination or reduction of data



The gather operation is simply the reverse of scatter. Each node starts with an m word message. In the first step, every odd numbered node sends its

buffer to an even numbered neighbor behind it, which concatenates the received message with its own buffer.

Only the even numbered nodes participate in the next communication step which results in nodes with multiples of four labels gathering more data and doubling the sizes of their data.

The process continues similarly, until node 0 has gathered the entire data. Just like one-to-all broadcast and all-to-one reduction, the hypercube algorithms for scatter

and gather can be applied unaltered to linear array and mesh interconnection topologies without any increase in the communication time.

The time in which all data are distributed to their respective destinations is



8. Explain the Flynn’s Model.

Parallelism can be expressed at various levels of granularity from instruction level to

processes.

Between these extremes exist a range of models, along with corresponding architectural

support.

Processing units in parallel computers either operate under the centralized control of a single

control unit or work independently.

Two types:

1. SIMD (Single Instruction Multiple Data)

2. MIMD (Multiple Instruction Multiple Data)

If there is a single control unit that dispatches the same instruction to various processors (that

work on different data), the model is referred to as single instruction stream, multiple data

stream (SIMD).

If each processor has its own control control unit, each processor can execute different

instructions on different data items. This model is called multiple instruction stream, multiple

data stream (MIMD).

SIMD and MIMD processor Figure:

SIMD:

Consists of a single CPU executing individual instructions on individual data values.

SIMD stands for Single Instruction Multiple Data.

Each processor follows the same set of instructions.



With different data elements being allocated to each processor.

SIMD computers have distributed memory with typically thousands of simple processors, and

the processors run in lock step.

SIMD computers, popular in the 1980s, are useful for fine grain data parallel applications,

such as neural networks.

Example: Conditional Execution in SIMD Processors

Executing a conditional statement on an SIMD computer with four processors: (a) the conditional

statement; (b) the execution of the statement in two steps.

MIMD:

Executes different instructions simultaneously

Each processor must include its own control unit

The processors can be assigned to parts of the same task or to completely separate tasks.

MIMD stands for Multiple Instruction Multiple Data.

There are multiple instruction streams with separate code segments distributed among the

processors.

MIMD is actually a superset of SIMD, so that the processors can run the same instruction

stream or different instruction streams.

In addition, there are multiple data streams; different data elements are allocated to each

processor.

MIMD computers can have either distributed memory or shared memory.

While the processors on SIMD computers run in lock step, the processors on MIMD

computers run independently of each other.



MIMD computers can be used for either data parallel or task parallel applications.

Some examples of MIMD computers are the SGI Origin2000 computer and the HP V-Class

computer.

Example: Multiprocessors, multicomputer.

SPMD (Single program multiple data)

SPMD stands for Single Program Multiple Data.

SPMD is a special case of MIMD.

SPMD execution happens when a MIMD computer is programmed to have the same set of

instructions per processor.

With SPMD computers, while the processors are running the same code segment, each processor

can run that code segment asynchronously.

Unlike SIMD, the synchronous execution of instructions is relaxed.

SIMD-MIMD COMPARISON

SIMD computers require less hardware than MIMD computers (single control unit).

However, since SIMD processors ae specially designed, they tend to be expensive and have long

design cycles.

Not all applications are naturally suited to SIMD processors.

In contrast, platforms supporting the SPMD paradigm can be built from inexpensive off-the-shelf

components with relatively little effort in a short amount of time.

UNIT- II

QUESTION BANK

Two Marks

1. Write the trends in microprocessor architecture?

2. What is the limitation of memory system performance?

3. What is the parallel computing platform?

4. What is the communication cost in parallel machine? (April 2013)

5. What is the Routing mechanism for interconnection networks? (November 2013)

6. Write the characteristics of task and interaction?

7. What is the mapping technique for load balancing?

8. Write the method for containing interaction overhead?

9. What is the Parallel algorithm model?

10. What is the One to all broadcast and all to one reduction?

11. What is the All to all broadcast and reduction?

12. Write the types of all to all broadcast reduction?


http://www.techservers.hp.com/products/technical_servers/v-class/technical_v_class_pb.html

http://www.sgi.com/products/servers/


13. What is the Scatter and gather?

14. What is the All to all personalized communication?

15. Define Circular shift.

16. Write the types of improving the speed of some communication operation

17. Define Decomposition?

18. What is the Granularity, Concurrency and task generation?

19. Write the Characteristic of inter task interaction?

20. Define Recursive decomposition.

21. Define Data decomposition.

22. Define exploratory decomposition.

23. Define Speculative decomposition.

24. Write the types of parallel algorithm model?

25. Write the Decomposition techniques?

11 Marks

1. Explain the limitations of memory system performance ( April 2013) (Ref.Pg.No.27)

2. Explain in detail about Parallel algorithm models. (April 2013) (Ref.Pg.No.29)

3. Explain decomposition techniques used to achieve concurrency with necessary examples

(November 2013) (Ref.Pg.No.31)

4. Discuss the all-to-all personalized communication on parallel computers with linear array, mesh

and hypercube interconnection networks. (November 2013) (Ref.Pg.No.35)

5. Discuss the shared address space Platforms.

6. Explain the methods for containing interaction overheads.

7. Elaborate on all – reduce and prefix-sum operations.

8. Explain the Flynn’s Model.



UNIT – III

Analytical Modeling of Parallel Programs: Sources of overhead in parallel programs Performance metrics for parallel systems Scalability of parallel systems Minimum execution time and minimum cost Optimal execution time.

Programming using the Message-Passing Paradigm: Principles of message-passing programming The building blocks MPI Topologies and embedding Overlapping communication with computation Collective communication and computation operations Groups and communicators.

Programming Shared Address Space Platforms: Thread basics Synchronization primitives in Pthreads Controlling thread and synchronization attributes Composite synchronization constructs Tips for designing asynchronous programs OpenMP

2 MARKS

1. Define Inter-process Interaction

Any nontrivial parallel system requires its processing elements to interact and communicate

data (e.g., intermediate results). The time spent communicating data between processing elements is

usually the most significant source of parallel processing overhead.

2. What is the Performance Metrics for Parallel Systems?

Execution Time

Total Parallel Overhead

Speedup

Efficiency

Cost

3. Define Execution time?

The serial runtime of a program is the time elapsed between the beginning and the end of its

execution on a sequential computer. The parallel runtime is the time that elapses from the moment a

parallel computation starts to the moment the last processing element finishes execution. We denote

the serial runtime by TS and the parallel runtime by TP.

4. Define Total Parallel Overhead.

We define overhead function or total overhead of a parallel system as the total time

collectively spent by all the processing elements over and above that required by the fastest known



sequential algorithm for solving the same problem on a single processing element. We denote the

overhead function of a parallel system by the symbol To.

5. Define speedup.

Speedup is a measure that captures the relative benefit of solving a problem in parallel. It is

defined as the ratio of the time taken to solve a problem on a single processing element to the time

required to solve the same problem on a parallel computer with p identical processing elements. We

denote speedup by the symbol S.

6. Define efficiency

Efficiency is a measure of the fraction of time for which a processing element is usefully

employed; it is defined as the ratio of speedup to the number of processing elements. In an ideal

parallel system, speedup is equal to p and efficiency is equal to one.

7. Define scalability of parallel system

The scalability of a parallel system is a measure of its capacity to increase speedup in

proportion to the number of processing elements. It reflects a parallel system's ability to utilize

increasing processing resources effectively.

8. The Isoefficiency Metric of Scalability (November 2013)

For a given problem size, as we increase the number of processing elements, the overall

efficiency of the parallel system goes down. This phenomenon is common to all parallel

systems.

In many cases, the efficiency of a parallel system increases if the problem size is increased

while keeping the number of processing elements constant.

9. What is degree of concurrency?

The maximum number of tasks that can be executed simultaneously at any time in a parallel

algorithm is called its degree of concurrency. The degree of concurrency is a measure of the number

of operations that an algorithm can perform in parallel for a problem of size W; it is independent of

the parallel architecture. If C(W) is the degree of concurrency of a parallel algorithm, then for a

problem of size W, no more than C(W) processing elements can be employed effectively.

10. What is the Key attributes that characterize the message-passing programming?

There are two key attributes that characterize the message-passing programming paradigm.

The first is that it assumes a partitioned address space and the second is that it supports only explicit

parallelization.

11. Write the basic operations in the message-passing programming.

The basic operations in the message-passing programming paradigm are send and receive. In their

simplest form, the prototypes of these operations are defined as follows:

send (void *sendbuf, intnelems, intdest)



receive (void *recvbuf, intnelems, int source)

12. Define Message-passing interface

The message-passing interface or MPI defines a standard library for message-passing that can

be used to develop portable message-passing programs using either C or FORTRAN. The MPI

standard defines both the syntax as well as the semantics of a core set of library routines that are very

useful in writing message-passing programs.

13. What is meant by Communication domain?

A key concept used throughout MPI is that of the communication domain. A communication

domain is a set of processes that are allowed to communicate with each other. Information about

communication domains is stored in variables of type MPI_Comm,that arecalled communicators.

14. Define communicator

The communicator is used to define a set of processes that can communicate with each other.

This set of processes form a communication domain. In general, all the processes may need to

communicate with each other. For this reason, MPI defines a default communicator called

MPI_COMM_WORLD which includes all the processes involved in the parallel execution.

15. Write the Basic functions for sending and receiving messages in MPI.

The basic functions for sending and receiving messages in MPI are the MPI_Send and

MPI_Recv, respectively. The calling sequences of these routines are as follows:

intMPI_Send(void *buf, int count, MPI_Datatypedatatype,

intdest, int tag, MPI_Commcomm)

intMPI_Recv(void *buf, int count, MPI_Datatypedatatype,

int source, int tag, MPI_Commcomm, MPI_Status *status)

16. Define Minimum Execution Time.(April 2013)

Minimum Execution Time is refers to increased execution time faster or minimum possible execution time of a parallel algorithm is, provided that the number of processing elements is not a constraint. As we increase the number of processing elements of a given problem size, either the parallel runtime continues to decease and asymptotically approaches a minimum value or it starts rising after attaining a minimum value.

17. What is synchronization primitive in Pthreads? (April 2013)

In communication is implicit in shared-address-space programming, much of the effort

associated with writing correct threaded programs is spent on synchronizing concurrent threads with

respect to their data accesses or scheduling.


= 0


Using pthread_createand pthread_join calls, we can create concurrent tasks. These tasks work together to manipulate data and accomplish a given task. When multiple threads attempt to manipulate the same data item, the results can often be incoherent if proper care is not taken to synchronize them. Consider the following code fragment being executed by multiple threads. The variable my_costis thread-local and best_costis a global variable shared by all threads.

1 /* each thread tries to update variable best_cost as follows */2 if (my_cost<best_cost)3 best_cost = my_cost;

11 MARKS

1. Elaborate on the various performance metrics used in studying the parallel systems.

(April 2013) (November 2013)

A number of metrics have been used based on the desired outcome of performance analysis.

Execution Time

The serial runtime of a program is the time elapsed between the beginning and the end of its

execution on a sequential computer.

The parallel runtime is the time that elapses from the moment a parallel computation starts to

the moment the last processing element finishes execution.

We denote the serial runtime by TS and the parallel runtime by TP.

Total Parallel Overhead

The overheads incurred by a parallel program are encapsulated into a single expression

referred to as the overhead function.

We define overhead function or total overhead of a parallel system as the total time

collectively spent by all the processing elements over and above that required by the fastest

known sequential algorithm for solving the same problem on a single processing element.

We denote the overhead function of a parallel system by the symbol To

The total time spent in solving a problem summed over all processing elements is pTP. TS

units of this time are spent performing useful work, and the remainder is overhead.

Therefore, the overhead function (To) is given by

Speedup

Speedup is a measure that captures the relative benefit of solving a problem in parallel.



It is defined as the ratio of the time taken to solve a problem on a single processing element to

the time required to solve the same problem on a parallel computer with p identical processing

elements.

We denote speedup by the symbol S.

Efficiency

Efficiency is a measure of the fraction of time for which a processing element is usefully

employed, it is defined as the ratio of speedup to the number of processing elements.

In an ideal parallel system, speedup is equal to p and efficiency is equal to one.

Otherwise, speedup is less than p and efficiency is between zero and one, depending on the

effectiveness with which the processing elements are utilized. We denote efficiency by the

symbol E.

Cost

We define the cost of solving a problem on a parallel system as the product of parallel runtime

and the number of processing elements used.

Cost reflects the sum of the time that each processing element spends solving the problem.

A parallel system is said to be cost-optimal if the cost of solving a problem on a parallel computer

has the same asymptotic growth (inQ terms) as a function of the input size as the fastest-known

sequential algorithm on a single processing element.

Cost is sometimes referred to as work or processor-time product, and a cost-optimal system is

also known as a pTP-optimal system.



Computing the globalsum of 16 partial sums using 16 processing elements . denotes the

sum of numbers with consecutive labels from i to j.

2. Explain any six MPI functions commonly used to perform collective communication

operations with an application (November 2013)

MPI defines a standard library for message-passing that can be used to develop portable message-passing programs using either C or FORTRAN.

The MPI standard defines both the syntax as well as the semantics of a core set of library routines.

Vendor implementations of MPI are available on almost all commercial parallel computers. It is possible to write fully-functional message-passing programs by using only the six

routines. These routines are used to initialize and terminate the MPI library, to get information about

the parallel computing environment, and to send and receive messages is shown in table.

Table 1: MPI Message Passing Interface

Starting and Terminating the MPI Library

MPI_Init is called prior to any calls to other MPI routines. Its purpose is to initialize the MPI environment.

MPI_Finalize is called at the end of the computation, and it performs various clean-up tasks to terminate the MPI environment.

The prototypes of these two functions are:intMPI_Init(int *argc, char ***argv)

intMPI_Finalize()

MPI_Init also strips off any MPI related command-line arguments. All MPI routines, data-types, and constants are prefixed by “MPI_”. The return code for

successful completion is MPI_SUCCESS. Communicators

A communicator defines a communication domain -a set of processes that are allowed to communicate with each other.

Information about communication domains is stored in variables of type MPI_Comm.



Communicators are used as arguments to all message transfer MPI routines. A process can belong to many different (possibly overlapping) communication domains. MPI defines a default communicator called MPI_COMM_WORLD which includes all the

processes Getting information The MPI_Comm_size and MPI_Comm_rank functions are used to determine the number of

processes and the label of the calling process, respectively. The calling sequences of these routines are as follows:

intMPI_Comm_size(MPI_Commcomm, int *size)

intMPI_Comm_rank(MPI_Commcomm, int *rank)

The function MPI_Comm_size returns in the variable size the number of processes that belong to the communicator comm.

So,MPI_Comm_size(MPI_COMM_WORLD, &size) will return in size the number of processors used by the program.

Every process that belongs to a communicator is uniquely identified by its rank. The rank of a process is an integer that ranges from zero up to the size of the communicator

minus one. A process can determine its rank in a communicator by using the MPI_Comm_rank function

that takes two arguments: the communicator and an integer variable rank. Up on return, the variable rank stores the rank of the process.

Example program:

#include <mpi.h>

main(intargc, char *argv[])

{

intnpes, myrank;

MPI_Init(&argc, &argv);

MPI_Comm_size(MPI_COMM_WORLD, &npes);

MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

printf("From process %d out of %d, Hello World!\n",myrank, npes);

MPI_Finalize();

}

Sending and Receiving Messages

The basic functions for sending and receiving messages in MPI are the MPI_Send and MPI_Recv, respectively. The calling sequences of these routines are as follows:

intMPI_Send(void *buf, int count, MPI_Datatypedatatype,

intdest, int tag, MPI_Commcomm)



intMPI_Recv(void *buf, int count, MPI_Datatypedatatype,

int source, int tag, MPI_Commcomm, MPI_Status *status)

MPI_Send sends the data stored in the buffer pointed by buf. This buffer consists of consecutive entries of the type specified by the parameter datatype. The number of entries in the buffer is given by the parameter count.

However, MPI allows two additional datatypes that are not part of the C language. These are MPI_BYTE and MPI_PACKED.

MPI_BYTE corresponds to a byte (8 bits) and MPI_PACKED corresponds to a collection of data items that has been created by packing non-contiguous data.

MPI_Recv receives a message sent by a process whose rank is given by thesource in the communication domain specified by the comm argument. The tag of the sent message must be that specified by the tag argument. If there are many messages with identical tag from the same process, then any one of these messages is received.

MPI allows specification of wildcard arguments for both source and tag. If source is set to MPI_ANY_SOURCE, then any process of the communication domain can

be the source of the message. If tag is set to MPI_ANY_TAG, then messages with any tag are accepted. On the receive side, the message must be of length equal to or less than the length field

specified.



On the receiving end, the status variable can be used to get information about the MPI_Recv operation.

The corresponding data structure contains:typedefstructMPI_Status

{

int MPI_SOURCE;

int MPI_TAG;

int MPI_ERROR;

};

The MPI_Get_count function returns the precise count of data items received.

intMPI_Get_count(MPI_Status *status, MPI_Datatypedatatype,int *count)

3. Explain collective communication and computation operations ( November 2013)

MPI provides an extensive set of functions for performing common collective communication operations.

Each of these operations is defined over a group corresponding to the communicator. All processors in a communicator must call these operations. The barrier synchronization operation is performed in MPI using:

intMPI_Barrier(MPI_Commcomm)

The one-to-all broadcast operation is:intMPI_Bcast(void *buf, int count, MPI_Datatypedatatype,int source,

MPI_Commcomm)

The all-to-one reduction operation is:

intMPI_Reduce(void *sendbuf, void *recvbuf, intcount,MPI_Datatypedatatype, MPI_Op op, inttarget,MPI_Commcomm)

Predefined Reduction Operation



The operation MPI_MAXLOC combines pairs of values (vi, li) and returns the pair (v, l) such that v is the maximum among all vi's and l is the corresponding li (if there are more than one, it is the smallest among all these li's).

MPI_MINLOC does the same, except for minimum value of vi.

An example use of the MPI _MINLOC and MPI _MAXLOC operators. MPI datatypes for data-pairs used with the MPI_ MAXLOC and MPI_MINLOC reduction

operations.

If the result of the reduction operation is needed by all processes, MPI provides:intMPI_Allreduce(void *sendbuf, void *recvbuf, intcount,MPI_Datatypedatatype, MPI_Op op, MPI_Commcomm)

To compute prefix-sums, MPI provides:intMPI_Scan(void *sendbuf, void *recvbuf, intcount,MPI_Datatypedatatype, MPI_Op op, MPI_Commcomm)

The gather operation is performed in MPI using:intMPI_Gather(void *sendbuf, intsendcount,MPI_Datatypesenddatatype, void *recvbuf, intrecvcount,MPI_Datatyperecvdatatype, int target, MPI_Commcomm)

MPI also provides the MPI_Allgather function in which the data are gathered at all the processes.

intMPI_Allgather(void *sendbuf, intsendcount,MPI_Datatypesenddatatype, void *recvbuf, intrecvcount,MPI_Datatyperecvdatatype, MPI_Commcomm)

The corresponding scatter operation is:intMPI_Scatter(void *sendbuf, intsendcount,MPI_Datatypesenddatatype, void *recvbuf, intrecvcount,MPI_Datatyperecvdatatype, int source, MPI_Commcomm)

The all-to-all personalized communication operation is performed by:intMPI_Alltoall(void *sendbuf, intsendcount,MPI_Datatypesenddatatype, void *recvbuf, intrecvcount,MPI_Datatyperecvdatatype, MPI_Commcomm)

Using this core set of collective operations, a number of programs can be greatly simplified.



4. Discuss minimum execution time and minimum cost optimal execution time (6 marks)

We are often interested in knowing how fast a problem can be solved, or what the minimum possible execution time of a parallel algorithm

We can determine the minimum parallel runtime for a given W by differentiating the expression for TP with respect to p and equating the derivative to zero (assuming that the function TP (W, p) is differentiable with respect to p).

Let be the minimum time in which a problem can be solved by a cost-optimal parallel system.

A problem of size W can be solved cost-optimally if and only if W = W (f(p)). In other words, given a problem of size W, a cost-optimal solution requires that p = O (f (W)).

Since the parallel runtime is Q(W/p) for a cost-optimal parallel system,the lower bound on the parallel runtime for solving a problem of size W cost-optimally is

From this equation:

We get,

5. Explain In Detail About Open Mp With Example?

Introduction:

Open Multi-Processing

The OpenMPAPIsupports multi-platform shared-memory parallel programming in C/C++ and Fortran.

The OpenMP API defines a portable, scalable model with a simple and flexible interface for developing parallel applications on platforms from the desktop to the supercomputer.


http://en.wikipedia.org/wiki/File:Fork_join.svg


An illustration of multithreading where the master thread forks off a number of threads which execute blocks of code in parallel.

ARB:

The OpenMP Architecture Review Board (ARB) published its first API specifications, OpenMP for Fortran 1.0, in October 1997.

The OpenMP ARB (or just “ARB”) is the non-profit corporation that owns the OpenMP brand, oversees the OpenMP specification and produces and approves new versions of the specification.

Permanent Members of the ARB:

AMD ,CAPSEntreprise ,ConveyComputer ,Cray ,Fujitsu ,HP ,IBM ,Intel ,NEC ,NVIDIA ,Oracle Corporation ,Red Hat ,ST Microelectronics ,Texas Instruments.

Auxiliary Members of the ARB:

ANL ,ASC/LLNL ,BSC ,cOMPunity ,EPCC ,LANL ,NASA ,ORNL ,RWTHAachen University ,SNL-Sandia National Lab ,Texas Advanced Computing Center ,University of Houston.

RED HAT:

Red Hat has joined the OpenMP Architecture Review Board (ARB), a group of leading hardware and software vendors and research organizations creating the standard for today’s most prevalent shared-memory parallel programming model.

LIBDOUBANM:

This is a private library used to support a website.

libdoubanm is a data mining library, contains optimization and matrix/vector algorithms .OpenMP is used to parallelize the code.

With OpenMP, the threads number is reduced to one at each machine and the time/speed is improved greatly.

OpenMP is used to parallelize the matrix/vector operations automatically. OpenMP’s task is used for graph visit.

NAN-TOOLBOX:

A time series analysis, statistics and machine learning toolbox for Octave and Matlab® for data with and without MISSING VALUES encoded as NaN’s. (Statistics, Machine Learning, Signal processing)

The NaN-toolbox is about 30000 lines in total of both projects. The core functions using OpenMP contain aboout 1800 lines of C/C++ and Matlab/Octave and is available on Linux + Matlab/Octave,Win32/Win64 + Matlab, Debian/Ubuntu + Octave.

OpenMP API 4.0:

The OpenMP Consortium has released OpenMP API 4.0, a major upgrade of the OpenMP API standard language specifications.


http://www.uh.edu/

http://www.tacc.utexas.edu/

http://www.sandia.gov/

http://www.sandia.gov/

http://www.rwth-aachen.de/

http://www.ornl.gov/

http://www.nasa.gov/centers/ames

http://lanl.gov/

http://www.epcc.ed.ac.uk/

http://www.compunity.org/

http://www.bsc.es/

http://www.llnl.gov/

http://www.anl.gov/

http://www.ti.com/

http://www.st.com/

http://www.redhat.com/

http://www.sun.com/

http://www.sun.com/

http://nvidia.com/

http://www.nec.com/

http://www.intel.com/

http://www.ibm.com/

http://www.hp.com/

http://www.fujitsu.com/

http://cray.com/

http://conveycomputer.com/

http://www2.caps-entreprise.com/

http://www.amd.com/

http://en.wikipedia.org/wiki/Thread_(computer_science)


New Features in the OPENMP 4.0 API Include:

Support for accelerators SIMD constructs to vectorize both serial as well as parallelized loops Error handling Thread affinity Tasking extensions Support for Fortran 2003 User-defined reductions Sequentially consistent atomics

Chart of OpenMP constructs:

Features of OpenMP Used:#pragma omp parallel#pragma omp for schedule(dynamic) nowait#pragma omp parallel for schedule(dynamic)

omp for or omp do:Used to split up loop iterations among the threads, also called loop constructs.The parallelization is done by the OpenMP directive #pragma omp.

Compilers:

OpenMP has been implemented in many commercial compilers. C/C++/FORTRAN,Visual C++ 2005, 2008 and 2010, gcc ,nagforas well as Intel Parallel Studio for various processors.

#include "stdafx.h"# include <cstdlib># include <iostream># include <iomanip># include <omp.h>

using namespace std;

int main ( int argc, char *argv[] );

//****************************************************************************80


http://en.wikipedia.org/wiki/Intel_Parallel_Studio

http://en.wikipedia.org/wiki/File:OpenMP_language_extensions.svg


int main ( int argc, char *argv[] )

//****************************************************************************80//// Purpose://// HELLO has each thread print out its ID.//// Discussion://// HELLO is a "Hello, World" program for OpenMP.//{ int id; double wtime;

cout << "\n"; cout << "HELLO_OPENMP\n"; cout << " C++/OpenMP version\n";

cout << "\n"; cout << " Number of processors available = " << omp_get_num_procs ( ) << "\n"; cout << " Number of threads = " << omp_get_max_threads ( ) << "\n";

wtime = omp_get_wtime ( );//// Have each thread say hello//# pragma omp parallel \ private ( id ) { id = omp_get_thread_num ( ); cout << " This is process " << id << "\n"; }//// Finish up by measuring the elapsed time.// wtime = omp_get_wtime ( ) - wtime;//// Terminate.// cout << "\n"; cout << "HELLO_OPENMP\n"; cout << " Normal end of execution.\n";

cout << "\n"; cout << " Elapsed wall clock time = " << wtime << "\n";

return 0;}



Output:

6. Explain Asymptotic Analysis of Parallel Programs and other scalability metrics.

Asymptotic Analysis of Parallel Programs:

Consider the problem of sorting a list of n numbers. The fastest serial programs for this problem run in timeO(n log n). Let us look at four different parallel algorithms A1, A2, A3, and A4, for sorting a given list. The parallel runtime of the four algorithms along with the number of processing elements they

can use is given in Table.

Table: Comparison of four different algorithms for sorting a given list of numbers. The table shows number of processing elements, parallel runtime, speedup, efficiency and the pTP product.



By this metric, algorithm A1 is the best, followed by A3, A4, and A2. This is also reflected in the fact that the speedups of the set of algorithms are also in this order.

Other Scalability Metrics

A number of other metrics of scalability of parallel systems have been proposed. These metrics are specifically suited to different system requirements

For example, in real time applications, the objective is to scale up a system to accomplish a task in a specified time bound.

One such application is multimedia decompression, where MPEG streams must be decompressed at the rate of 25 frames/second.A parallel system must decode a single frame in 40 ms (or with buffering, at an average of 1 frame in 40 ms over the buffered frames).Scaled Speedup

This metric is defined as the speedup obtained when the problem size is increased linearly with the number of processing elements.

If the scaled-speedup curve is close to linear with respect to the number of processing elements, then the parallel system is considered scalable.

In this case the scaled-speedup metric provides results very close to those of isoefficiency analysis, and the scaled-speedup is linear or near-linear with respect to the number of processing element

Two generalized notions of scaled speedup have been examinedo In one method, the size of the problem is increased to fill the available memory on the

parallel computer. The assumption here is that aggregate memory of the system increases with the number of processing elements. o In the other method, the size of the problem grows with p subject to an upper-bound on

execution time.

7. Discuss the building blocks of message passing.

The basic operations in the message-passing programming paradigm are send and receive. In their simplest form, the prototypes of these operations are defined as follows:

send(void *sendbuf, intnelems, intdest)

receive(void *recvbuf, intnelems, int source)

Consider the following code segments:



In this simple example, process P0 sends a message to process P1 which receives and prints the message. The process P0 changes the value of a to 0 immediately following the send.

The semantics of the send operation require that the value received by process P1 must be 100 as opposed to 0. That is, the value of a at the time of the send operation must be the value that is received by process P1.

This motivates the design of the send and receive protocols.

Blocking Message Passing OperationsThere are two mechanisms by which this can be achieved.

Blocking Non-Buffered Send/Receive The sending process sends a request to communicate to the receiving process. When the

receiving process encounters the target receives, it responds to the request. The sending process upon receiving this response initiates a transfer operation.Since there are no buffers used at either sending or receiving ends, this is also referred to as a non-buffered blocking operation

Idling Overheads in Blocking Non-Buffered Operations, we will discuss three scenarios in which the send is reached before the receive is posted, the send and receive are posted around the same time, and the receive is posted before the send is reached.

In cases (a) and (c), we notice that there is considerable idling at the sending and receiving process.

Figure: Handshake for a blocking non-buffered send/receive operation. It is easy to see that

in cases where sender and receiver do not reach communication point at similar times, there can be considerable idling overheads.

Blocking Buffered Send/Receive A simple solution to the idling and deadlocking problem outlined above is to rely on buffers at

the sending and receiving ends. The sender simply copies the data into the designated buffer and returns after the copy

operation has been completed. The data must be buffered at the receiving end as well. Buffering trades off idling overhead for buffer copying overhead. Figure: Blocking buffered transfer protocols: (a) in the presence of communication

hardware with buffers at send and receive ends; and (b) in the absence of



communication hardware, sender interrupts receiver and deposits data in buffer at receiver end.

Deadlocks in Buffered Send and Receive Operations While buffering alleviates many of the deadlock situations, it is still possible to write code

that deadlocks. This is due to the fact that as in the non-buffered case, receive calls are always blocking (to

ensure semantic consistency). Thus, a simple code fragment such as the following deadlocks since both processes wait to

receive data but nobody sends it.

Once again, such circular waits have to be broken. However, deadlocks are caused only by waits on receive operations in this case.

Non-Blocking Message Passing Operations The programmer must ensure semantics of the send and receive. This class of non-blocking protocols returns from the send or receive operation before it is

semantically safe to do so. Non-blocking operations are generally accompanied by a check-status operation. When used correctly, these primitives are capable of overlapping communication overheads

with useful computations. Message passing libraries typically provide both blocking and non-blocking primitives.



Figure 1: Non-blocking non-buffered send and receive operations (a) in absence of communication hardware; (b) in presence of communication hardware.

Non-blocking operations can themselves be buffered or non-buffered. In the non-buffered case, a process wishing to send data to another simply posts a pending message and returns to the user program.This is shown in figure 2.

Figure 2: Space of possible protocols for send and receive operations.

8. Explain in detail about shared address space platforms.

Here, we discuss shared-address-space programming paradigms along with their performance issues and related extensions to directive-based paradigms.

Shared address space programming paradigms can vary on mechanisms for data sharing, concurrency models, and support for synchronization.

Thread Basics

A thread is a single stream of control in the flow of a program.



for (row = 0; row < n; row++)

for (column = 0; column < n; column++)

c[row][column] =dot_product(get_row(a, row),get_col(b, col));

can be transformed to:

for (row = 0; row < n; row++)

for (column = 0; column < n; column++)

c[row][column] =create_thread(dot_product(get_row(a, row),get_col(b, col)));

In this case, one may think of the thread as an instance of a function that returns before the function has finished executing.

All memory in the logicalmachine model of a thread is globally accessible to every thread. The stack corresponding to the function call is generally treated as being local to the thread for

liveness reasons. This implies a logical machine model with both global memory (default) and local memory

(stacks). It is important to note that such a flat model may result in very poor performance since

memory is physically distributed in typical machines.

The logical machine model of a thread-based programming paradigm.

Why Threads?

Threaded programming models offer significant advantages over message-passing programming models along with some disadvantages as well, let us briefly look at some of these.

Threads provide software portability. Inherent support for latency hiding. Scheduling and load balancing. Ease of programming and widespread use.



Software Portability Threaded applications can be developed on serial machines and run on parallel machines without any changes.This ability to migrate programs between diverse architectural platforms is a very significant advantage of threaded APIs.

Latency Hiding One of the major overheads in programs (both serial and parallel) is the access latency for memory access, I/O, and communication. By allowing multiple threads to execute on the same processor, threaded APIs enable this latency to be hidden

Scheduling and load balancing: Threaded APIs allow the programmer to specify a large number of concurrent tasks and support system-level dynamic mapping of tasks to processors with a view to minimizing idling overheads. By providing this support at the system level, threaded APIs rid the programmer of the burden of explicit scheduling and load balancing.

Easeof programming and widespread use: With widespread acceptance of the POSIX thread API, development tools for POSIX threads are more widely available and stable. These issues are important from the program development and software engineering aspects.

The POSIX Thread API Commonly referred to as Pthreads, POSIX has emerged as the standard threads API,

supported by most vendors. The concepts discussed here are largely independent of the API and can be used for

programming with other thread APIs (NT threads, Solaris threads, Java threads, etc.) as well.Thread Basics: Creation and Termination

Let us start with a simple threaded program for computing the value of p.

Pthreads provides two basic functions for specifying concurrency in a program:#include <pthread.h>

intpthread_create (pthread_t *thread_handle,constpthread_attr_t *attribute,void * (*thread_function)(void *),void *arg);

intpthread_join (pthread_tthread,void **ptr);

The function pthread_create invokes function thread_function as a thread. The pthread_create function creates a single thread that corresponds to the invocation of the

function thread_function (and any other functions called by thread_function). On successful creation of a thread, a unique identifier is associated with the thread and

assigned to the location pointed to by thread_handle. The thread has the attributes described by the attribute argument. The arg field specifies a pointer to the argument to function thread_function. This argument is

typically used to pass the workspace and other thread-specific data to a thread. The main program must wait for the threads to run to completion.This is done using the

function pthread_join which suspends execution of the calling thread until the specified thread terminates. The prototype of the pthread_join function is as follows:1) int

2) pthread_join (

3) pthread_t thread,

4) void **ptr);



A call to this function waits for the termination of the thread whose id is given by thread. On a successful call to pthread_join, the value passed to pthread_exit is returned in the

location pointed to by ptr. On successful completion, pthread_join returns 0, else it returns an error-code.

Synchronization Primitives in Pthreads

When multiple threads attempt to manipulate the same data item, the results can often be incoherent if proper care is not taken to synchronize them.

Consider:

/* each thread tries to update variable best_cost as follows */

if (my_cost<best_cost)

best_cost = my_cost;

Assume that there are two threads, the initial value of best cost is 100, and the values of my cost are 50 and 75 at threads t1 and t2.

Depending on the schedule of the threads, the value of best cost could be 50 or 75. The value 75 does not correspond to any serialization of the threads.

Mutual Exclusion: The code in the previous example corresponds to a critical segment; i.e., a segment that must

be executed by only one thread at any time. Critical segments in Pthreads are implemented using mutex locks. Mutex-locks have two states: locked and unlocked. At any point of time, only one thread can

lock a mutex lock. A lock is an atomic operation. A thread entering a critical segment first tries to get a lock. It goes ahead when the lock is

granted. The Pthreads API provides the following functions for handling mutex-locks:

intpthread_mutex_lock (pthread_mutex_t *mutex_lock);

intpthread_mutex_unlock (pthread_mutex_t *mutex_lock);

Example:Producer-Consumer Using Locks

The producer-consumer scenario imposes the following constraints:

The producer threadmust not overwrite the shared buffer when the previous task has not been picked up by a consumer thread.

The consumer threads must not pick up tasks until there is something present in the shared data structure.

Individual consumer threads should pick up tasks one at a time.Types of Mutexes

Pthreads supports three types of mutexes: normal, recursive, and error-check.

A normal mutex deadlocks if a thread that already has a lock tries a second lock on it. A recursive mutex allows a single thread to lock a mutex as many times as it wants. It simply

increments a count on the number of locks. A lock is relinquished by a thread when the count becomes zero.



An error check mutex reports an error when a thread with a lock tries to lock it again (as opposed to deadlocking in the first case, or granting the lock, as in the second case).

The type of the mutex can be set in the attributes object before it is passed at time of initialization.

Condition Variables for Synchronization

A condition variable is a data object used for synchronizing threads. This variable allows a thread to block itself until specified data reaches a predefined state.

A condition variable is associated with this predicate. When the predicate becomes true, the condition variable is used to signal one or more threads waiting on the condition.

A single condition variable may be associated with more than one predicate. A condition variable always has a mutex associated with it. A thread locks this mutex and

tests the predicate defined on the shared variable. If the predicate is not true, the thread waits on the condition variable associated with the

predicate using the function pthreadcond wait.

Pthreads provides the following functions for condition variables:intpthread_cond_wait(pthread_cond_t *cond,pthread_mutex_t *mutex);

intpthread_cond_signal(pthread_cond_t *cond);

intpthread_cond_broadcast(pthread_cond_t *cond);

Controlling Thread and Synchronization Attributes The Pthreads API allows a programmer to change the default attributes of entities using

attributes objects. An attributes object is a data-structure that describes entity(thread, mutex, condition variable)

properties. Once these properties are set, the attributes object can be passed to the method initializing the

entity. Enhances modularity, readability, and ease of modification.

Attributes Objects for Threads

Use pthread _attr _init to create an attributes object. Individual properties associated with the attributes object can be changed using the following

functions:pthread _attr _setdetachstate,

pthread _attr _setguardsize_ np,

pthread _attr _setstacksize,

pthread _attr _setinheritsched,

pthread _attr _setschedpolicy, and pthreadattrsetschedparam.

Attributes Objects for Mutexes

Initialize the attrributes object using function: pthread _mutexattr_init.



The function pthreadmutexattr_settype _np can be used for setting the type of mutex specified by the mutex attributes object.

pthread_mutexattr_settype_np (pthread_mutexattr_t *attr,int type);

Here, type specifies the type of the mutex and can take one of: PTHREAD_MUTEX _NORMAL _NP PTHREAD _MUTEX_ RECURSIVE _NP PTHREAD _MUTEX _ERRORCHECK_ NP

Thread Cancellation

A thread may cancel itself or cancel other threads. When a call to this function is made, a cancellation is sent to the specified thread.

Threads can protect themselves against cancellation. When a cancellation is actually performed, cleanup functions are invoked for reclaiming the thread data structures.

This process is similar to termination of a thread using the pthread_exit call. Posix threads provide this cancellation feature in the function pthread_cancel. The prototype

of this function is:1 int

2 pthread_cancel (

3 pthread_t thread);

Composite Synchronization Constructs

By design, Pthreads provide support for a basic set of operations. Higher level constructs can be built using basic synchronization constructs. We discuss two such constructs: read-write locks and barriers

Read-Write Locks

In many applications, a data structure is read frequently but written infrequently. For such applications, we should use read-write locks.

A read lock is granted when there are other threads that may already have read locks. If there is a write lock on the data (or if there are queued write locks), the thread performs a

condition wait. If there are multiple threads requesting a write lock, they must perform a condition wait. With this description, we can design functions for read locks mylib _rwlock_ rlock, write

locks mylib_ rwlock _wlock, and unlocking mylib _rwlock _unlock. The function mylib _rwlock_ rlock attempts a read lock on the data structure. It checks to see

if there is a write lock or pending writers. If so, it performs a condition wait on the condition variable readers proceed.

The function mylib_rwlock_wlock attempts a write lock on the data structure. It checks to see if there are readers or writers; if so, it increments the count of pending writers and performs a condition wait on the condition variable writer_proceed.

The function mylib_rwlock_unlock unlocks a read or write lock.Barriers

A barrier holds a thread until all threads participating in the barrier have reached it. Barriers can be implemented using a counter, a mutex and a condition variable.



A single integer is used to keep track of the number of threads that have reached the barrier. If the count is less than the total number of threads, the threads execute a condition wait. The last thread entering (and setting the count to the number of threads) wakes up all the

threads using a condition broadcast.

Barrier program:

typedefstruct {

pthread_mutex_tcount_lock;

pthread_cond_tok_to_proceed;

int count;

} mylib_barrier_t;

voidmylib_init_barrier(mylib_barrier_t *b) {

b -> count = 0;

pthread_mutex_init(&(b ->count_lock), NULL);

pthread_cond_init(&(b ->ok_to_proceed), NULL);

}

voidmylib_barrier (mylib_barrier_t *b, intnum_threads) {

pthread_mutex_lock(&(b ->count_lock));

b -> count ++;

if (b -> count == num_threads) {

b -> count = 0;

pthread_cond_broadcast(&(b ->ok_to_proceed));

}

Else

while (pthread_cond_wait(&(b ->ok_to_proceed),

&(b ->count_lock)) != 0);

pthread_mutex_unlock(&(b ->count_lock));

}

The barrier described above is called a linear barrier. The trivial lower bound on execution time of this function is therefore O(n) for n threads. This implementation of a barrier can be speeded up using multiple barrier variables organized

in a tree.



This process repeats up the tree.This is also called a log barrier and its runtime grows as O(log p).

UNIT – III

QUESTION BANK

2 MARKS

1. Define Inter-process Interaction 2. What is the Performance Metrics for Parallel Systems?3. Define Execution time?4. Define Total Parallel Overhead.5. Define speedup.6. Define efficiency7. Define scalability of parallel system8. The Isoefficiency Metric of Scalability (November 2013)9. What is degree of concurrency?10. What is the Key attributes that characterize the message-passing programming?11. Write the basic operations in the message-passing programming.12. Define Message-passing interface13. What is meant by Communication domain?14. 14.Define communicator15. Write the Basic functions for sending and receiving messages in MPI.16. Define Minimum Execution Time. (April 2013)17. What is synchronization primitive in Pthreads? (April 2013)

11 MARKS

1. Elaborate on the various performance metrics used in studying the parallel systems. 2. (April 2013) (November 2013)(Ref.Pg.No.51)3. Explain any six MPI functions commonly used to perform collective communication

operations with an application (November 2013) (Ref.Pg.No.53)4. Explain collective communication and computation operations ( November 2013)

(Ref.Pg.No.57)5. Discuss minimum execution time and minimum cost optimal execution time (6 marks)6. Explain in detail about open mp with example?7. Explain Asymptotic Analysis of Parallel Programs and other scalability metrics.8. Discuss the building blocks of message passing.9. Explain in detail about shared address space platforms.



UNIT – IV

Dense Matrix Algorithms:

Matrix-vector multiplication

Matrix-matrix multiplication

Solving a system of linear equations

FFT.

Sorting:

Issues in sorting on parallel computers

Sorting networks

Bubble sort and its variants

Quicksort

Bucket and sample sort

Other sorting algorithms.

Graph Algorithms:

Definitions and representation

Minimum spanning tree

Single-source shortest paths

All-pairs shortest paths.

TWO MARKS

1. What is a dense matrix algorithm?

Algorithms involving matrices and vectors are applied in several numerical and non numerical

contexts. Some key algorithm for dense algorithm.

1. Define Bitonic Sequence. (November 2013)

One of the key aspects of the bitonic algorithm is that it is communication intensive, and a

proper mapping of the algorithm must take into account the topology of the underlying

interconnection network.

The bitonic sorting network for sorting n elements contains log n stages, and stage i consists

of i columns of n/2 comparators.

3. Write Matrix vector multiplication algorithm.

procedure MAT-VECT(A,X,Y)

begin

for i=0to n-1 do

begin



y[i]:=0;

for j:=0 to n-1 do

y[i]:=y[i]+A[i,j]*x[j];

end for;

end MAT-VECT

3. What is block matrices operations?

It express a matrix computation involving scalar algebraic operations on all its elements in

terms of identical matrix algebraic operations on blocks or sub matrices of the original matrix.uch

algebraic operations on the sub matrices are called block matrix operations.

4. What is DNS algorithm?

It based on partitioning intermediate data that can use up to n3 processes and that performs

matrix multiplication in time (n3/log n) processes. This algorithm is known as the DNS algorithm.

5. What is LU Factorization?

In the context of upper-triangularization, a similar procedure can be used to factorize matrix A

as the product of a lower-triangular matrix L and a unit upper- triangular matrix U so that

A=L*U .This factorization is commonly referred to as LU factorization.

6. What is pipelined Gaussian elimination?

The performance of the algorithm can be improved substantially if the processes work

asynchronously; that is, no process waits for the others to finish an iteration before the starting the

next one. We call this the asynchronous or pipelined version of Gaussian elimination

7. What is partial pivoting?

The Gaussian elimination algorithm in algorithm fails if any diagonal entry A [K, K] of the

matrix of coefficients is close o equal to zero. To avoid this problem and to ensure the numerical

stability of the algorithm, a technique called partial pivoting.

8. What is scaled speedup?

It is defined as the problem size is increased linearly with the number of processes; that is if

W is chosen as a base problem size for a single process. Then

Scaled speedup=wp/Tp(Wp-p)

9. What is sorting?

It is defined as the task of arranging n unordered collection of elements into monotonically

increasing (or decreasing) order.

10. Types of sorting?

They are 2 types of sorting are

1. Internal sorting

2. External sorting



11. What is internal sorting?

The number of elements to be sorted is small enough to fit into the process main memory. It is

also called as internal sorting.

12. What is external sorting?

It uses auxiliary storage (such as tapes and hard disks) for sorting because the number of

elements to be sorted is too large to fit into memory. It is also called as external sorting.

13. What is comparison based algorithm?

It is unordered sequence of elements by repeatedly comparing pairs of elements and, if they

are out of order, exchanging them. This fundamental operation of comparison based sorting is called

compare exchange.

14. What is comparator?

It is a device with two inputs x and y and two output x’ and y’. It is also called as comparator.

15. Types of comparator?

2types are

1. Increasing comparator

2. Decreasing comparator

16. What is Bitonic sort? (November 2013)

A bitonic sorting network sorts n elements in (log2 n) time. The key operation of the bitonic

sorting network is the rearrangement of a bitonic sequence into a sorted sequence.

17. What is shell sort?

It is one such serial sorting algorithm. Let n be the number of elements to be sorted and p be

the number of processes. Each process is assigned a block of n/p elements.

18. What is quick sort?

Quick sort is a divide conquer algorithm that’s sorts a sequence by recursively dividing it into

smaller sub sequences.

19. What is bucket sort?

It sorting an array of n elements whose values are uniformly distributed over an interval [a, b]

is the bucket sort. It is also called bucket sort

20. What is sample sort?

The bucket sort may result in buckets that have a significantly different number of elements,

degrading performance. In such situation an algorithm called sample sort.

21. What is spanning tree?

It is undirected graph G is a sub graph of G that is a tree containing all the vertices of G. It is

also known as spanning tree.



22. What is weighted graph?

The weight of a sub graph is the sum of the weight of the edges in the sub graph .It is also

known weighted graph.

23. What is minimum spanning tree? (April 2013)

It denote as MST. The minimum spanning tree for a weighted undirected graph is a spanning

tree with minimum weight.

24. What is shortest path?

It is minimum weight path. it depend on the application, edge, weights may represent time ,

cost, penalty, loss or any other quantity that accumulates additively along a path and is to be

minimized.

25. What is connected components?

The connected components of an undirected graph are the equivalence classes of vertices

under the “is reachable from “relation.

26. Define sorting networks (April 2013)

Sorting networks are based on a comparison network model, in which many comparison

operations are performed simultaneously.

The key component of these networks is a comparator. A comparator is a device with two

inputsx and y and two outputs x' and y'. For an increasing comparator, x' = min{x, y} and y' =

max{x, y}; for a decreasing comparator x' = max{x, y} and y' = min{x, y}.

A sorting network is usually made up of a series of columns, and each column contains a

number of comparators connected in parallel.

11 MARKS

1. Explain how the parallel implementation of Gaussian elimination algorithm is used in solving a system of linear equations. (November 2013)

The problem of solving a system of linear equations of the form is discussed here:

In matrix notation, this system is written as Ax = b. A system of equations Ax = b is usually solved in two stages. First, through a series of

algebraic manipulations, the original system of equations is reduced to an upper-triangular system of the form



We write this as Ux= y, where U is a unit upper-triangular matrix – one in which all subdiagonal entries are zero and all principal diagonal entries are equal to one.

A Simple Gaussian Elimination Algorithm

The serial Gaussian elimination algorithm has three nested loops. Several variations of the algorithm exist, depending on the order in which the loops are arranged.

Algorithm: A serial Gaussian elimination algorithm that converts the system of linear

equationsAx = b to a unit upper-triangular system Ux= y. The matrix U occupies the

upper-triangular locations of A. This algorithm assumes that A[k, k] 0 when it is used as a divisor on lines 6 and 7.

1. procedureGAUSSIAN_ELIMINATION (A, b, y)

2. begin

3. for k := 0 to n - 1 do /* Outer loop */

4. begin

5. for j :=k + 1 to n - 1 do

6. A[k, j] := A[k, j]/A[k, k]; /* Division step */

7. y[k] := b[k]/A[k, k];

8. A[k, k] := 1;

9. for i :=k + 1 to n - 1 do

10. begin

11. for j :=k + 1 to n - 1 do

12. A[i, j] := A[i, j] - A[i, k] x A[k, j]; /* Elimination step */

13. b[i] := b[i] - A[i, k] x y[k];

14. A[i, k] := 0;

15. endfor; /* Line 9 */

16. endfor; /* Line 3 */



17. endGAUSSIAN_ELIMINATION

Gaussian Elimination with Partial Pivoting

The Gaussian elimination algorithm in Algorithm 8.4 fails if any diagonal entry A[k, k] of the matrix of coefficients is close or equal to zero.

To avoid this problem and to ensure the numerical stability of the algorithm, a technique called partial pivoting is used.

At the beginning of the outer loop in the kth iteration, this method selects a column i (called the pivot column) such that A[k, i] is the largest in magnitude among all A[k, j] such that k

j <n. It then exchanges the kth and the i th columns before starting the iteration. These columns can either be exchanged explicitly by physically moving them into each other's

locations, or they can be exchanged implicitly by simply maintaining an n x 1 permutation vector to keep track of the new indices of the columns of A.

2. Explain in detail Matrix vector multiplication. (April 2013)

The problem of multiplying a dense n x n matrix A with an n x 1 vector x to yield the n x 1 result vector y.

The Algorithm shows a serial algorithm for this problem. The sequential algorithm requires n2 multiplications and additions.

Assuming that a multiplication and addition pair takes unit time, the sequential run time is

At least three distinct parallel formulations of matrix-vector multiplication are possible, depending on whether

o rowwise 1-Do columnwise 1-Do 2-D partitioning is used.

Algorithm :A serial algorithm for multiplying an n x n matrix A with an n x 1 vector x to yield an n x 1 product vector y.

1. procedureMAT_VECT ( A, x, y)

2. begin

3. for i := 0 to n - 1 do

4. begin

5. y[i]:=0;

6. for j := 0 to n - 1 do

7. y[i] := y[i] + A[i, j] x x[j];

8. endfor;

9. endMAT_VECT



Row-wise 1-D Partitioning

The parallel algorithm for matrix-vector multiplication using rowwise block 1-D partitioning is discussed.

Figure 1: describes the distribution and movement of data for matrix-vector multiplication with block 1-D partitioning.

Multiplication of an n x n matrix with an n x 1 vector using rowwise block 1-D partitioning. For the one-row-per-process case, p = n.

One Row per Process

First, consider the case in which the n x n matrix is partitioned among n processes so that each process stores one complete row of the matrix.

The n x 1 vector x is distributed such that each process owns one of its elements. The initial distribution of the matrix and the vector for rowwise block 1-D partitioning is

shown in Figure (a). Since each process starts with only one element of x, an all-to-all broadcast is required to

distribute all the elements to all the processes in figure(b) After the vector x is distributed among the processes in figure(c) The result vector y is stored exactly the way the starting vector x was stored is shown in

figure(d). Thus, the entire procedure is completed by n processes in time Q (n), resulting in a process-

time product of Q (n2). The parallel algorithm iscost-optimal because the complexity of the serial algorithm is Q (n2).

Using Fewer than n Processes

Consider the case in which p processes are used such that p <n, and the matrix is partitioned among the processes by using block 1-D partitioning.



Each process initially stores n/p complete rows of the matrix and a portion of the vector of size n/p. Since the vector x must be multiplied with each row of the matrix, every process needs the entire vector.

This again requires an all-to-all broadcast as shown in Figure (b) and (c).

2-D PartitioningThis section discusses parallel matrix-vector multiplication for the case in which the matrix is distributed among the processes using a block 2-D partitioning.

Matrix-vector multiplication with block 2-D partitioning. For the one-element-per-process case, p = n2 if the matrix size is n x n.

One Element Per Process

As Figure (a) shows, the first communication step for the 2-D partitioning aligns the vector x along the principal diagonal of the matrix.

The second step copies the vector elements from each diagonal process to all the processes in the corresponding column is shown in figure (b).

Figure (c) shows this step, which requires an all-to-one reduction in each row with the last process of the row as the destination.

The parallel matrix-vector multiplication is complete after the reduction step. The overall parallel run time of this algorithm is Q (n). The cost (process-time product) is Q (n2 log n). Hence, the algorithm is not cost-optimal.

Using Fewer than n2 Processes

A cost-optimal parallel implementation of matrix-vector multiplication with block 2-D partitioning of the matrix can be obtained, if the granularity of computation at each process is increased by using fewer than n2 processes.



The initial data-mapping and the various communication steps for this case is shown in figure (a).

The entire vector must be distributed on each row of processes before the multiplication can be performed. First, the vector is aligned along the main diagonal.

Then a columnwise one-to-all broadcast of these elements takes place. Each process

then performs n2/p multiplications and locally adds the sets of products. At the end of this step in figure(c), each process has partial sums that must be accumulated

along each row to obtain the result vector. Hence, the last step of the algorithm is an all-to-one reduction of the values in each row, with

the rightmost process of the row as the destination. Thus, the parallel run time for this procedure is as follows:

Comparison of 1-D and 2-D Partitionings

If the number of processes is greater than n, then the 1-D partitioning cannot be used. However, even if the number of processes is less than or equal to n, the analysis in this section suggests that 2-D partitioning is preferable.

Among the two partitioning schemes, 2-D partitioning has a better (smaller) asymptotic isoefficiency function.

Thus, matrix-vector multiplication is more scalable with 2-D partitioning, i.e it can deliver the same efficiency on more processes with 2-D partitioning than with 1-D partitioning.

3. Discuss in cannon’s and DNS algorithm in details.

Cannon's Algorithm

Cannon's algorithm is a memory-efficient version of the simple algorithm To study this algorithm, we again partition matrices A and B into p square blocks. The first communication step of the algorithm aligns the blocks of A and B in such a way that

each process multiplies its local submatrices, this alignment is achieved for matrix A by shifting all submatrices Ai, j to the left (with wraparound) by i steps in figure(a).

Similarly, all submatrices Bi, j are shifted up (with wraparound) by j steps. These are circular shift operations in Figure(b)

Figure 8.3(c) shows the blocks of A and B after the initial alignment, when each process is ready for the first submatrix multiplication. After a submatrix multiplication step, each block of A moves one step left and each block of B moves one step up (again with wraparound)as shown in Figure (d).

A sequence of such submatrix multiplications and single-step shifts pairs up each Ai, k and Bk, j for at Pi, j. This completes the multiplication of matrices A and B.Figure: The communication steps in Cannon's algorithm on 16 processes.



Thus, the approximate overall parallel run time of this algorithm is



The DNS Algorithm

The matrix multiplication algorithms presented so far use block 2-D partitioning of the input and the output matrices and use a maximum of n2processes for n x n matrices.

As a result, these algorithms have a parallel run time of W (n) because there are Q (n3) operations in the serial algorithm.

We now present a parallel algorithm based on partitioning intermediate data that can use upton3processes and that performs matrix multiplication in time Q(log n) by using W(n3/log n) processes.

This algorithm is known as the DNS algorithm because it is due to Dekel, Nassimi, and Sahni. Figure: The communication steps in the DNS algorithm while multiplying 4 x 4 matrices

A and B on 64 processes. The shaded processes in part (c) store elements of the first row of A and the shaded processes in part (d) store elements of the first column of B.

The DNS algorithm has three main communication steps: (1) Moving the columns of A and the rows of B to their respective planes

(2) Performing one-to-all broadcast along the j axis for A and along the i axis for B

3) All-to-one reduction along the k axis. All these operations are performed within groups of n processes and take time Q (log n).



Thus, the parallel run time for multiplying two n x n matrices using the DNS algorithm on n3 processes is Q (log n).

Assume that the number of processes p is equal to q3for some q <n. To implement the DNS algorithm, the two matrices are partitioned into blocks of size (n/q) x

(n/q). Each matrix can thus be regarded as a q x q two-dimensional square array of blocks. The implementation of this algorithm on q3processes is very similar to that on n3processes. We get the following approximate expression for the parallel run time of the DNS algorithm:

4. Explain in detail about quick sort in parallel ( April 2013)

Quick sort is one of the most common sorting algorithms for sequential computers because of its simplicity, low overhead and optimal average complexity.

Quick sort is a divide-and-conquer algorithm that sorts a sequence by recursively dividing it into smaller subsequences. Assume that the n-element sequence to be sorted is stored in the array A[1...n]. Quick sort consists of two steps: divide and conquer.

During the divide step, a sequence A[q...r] is partitioned (rearranged) into two nonempty subsequences A[q...s] and A[s + 1...r] such that each element of the first subsequence is smaller than or equal to each element of the second subsequence.

During the conquer step, the subsequences are sorted by recursively applying quick sort. Since the subsequences A[q...s] and A[s + 1...r] are sorted and the first subsequence has smaller elements than the second, the entire sequence is sorted.

The sequence A[q...r] partitioned into two parts – one with all elements smaller than the other. This is usually accomplished by selecting one element x from A[q...r] and using this element

to partition the sequence A[q...r] into two parts – one with elements less than or equal to x and the other with elements greater than x. Element x is called the pivot.

Figure: Example of the quick sort algorithm sorting a sequence of size n= 8.

Algorithm: The sequential quicksort algorithm.

1. procedureQUICKSORT (A, q, r)

2. begin

3. ifq <r then

4. begin



5. x:= A[q];

6. s:= q;

7. for i :=q + 1 to r do

8. ifA[i] x then

9. begin

10. s:= s + 1;

11. swap(A[s], A[i]);

12. end if

13. swap(A[q], A[s]);

14. QUICKSORT (A, q, s);

15. QUICKSORT (A, s + 1, r);

16. end if

17. endQUICKSORT

Although quicksort can have O (n2) worst-case complexity, its average complexity is significantly better.

Parallelizing Quicksort

Quicksort can be parallelized in a variety of ways We present three distinct parallel formulations:

o One for a CRCW PRAM, o One for a shared-address-space architectureo One for a message-passing platform.

Each of these formulations parallelizes quicksort by performing the partitioning step in parallel.

1. Parallel Formulation for a CRCW PRAM

An arbitrary CRCW PRAM is a concurrent-read, concurrent-write parallel random-access machine in which write conflicts are resolved arbitrarily. In other words, when more than one process tries to write to the same memory location, only one arbitrarily chosen process is allowed to write, and the remaining writes are ignored.

Executing quicksort can be visualized as constructing a binary tree. In this tree, the pivot is the root; elements smaller than or equal to the pivot go to the left subtree, and elements larger than the pivot go to the right subtree.Figure: A binary tree generated by the execution of the quicksort algorithm. Each level of the tree represents adifferent array-partitioning iteration. If pivot selection is optimal, then the height of the tree is Q (log n), which is alsothe number of iterations.



2. Parallel Formulation for Practical Architectures Our discussion will focus on developing an algorithm for a shared-address-space system

and then we will show how this algorithm can be adapted to message-passing systems.

Shared-Address-Space Parallel Formulation

The algorithm starts by selecting a pivot element, which is broadcast to all processes. Each

process Pi, upon receiving the pivot, rearranges its assigned block of elements into two

sub-blocks, one with elements smaller than the pivot Si and one with elements larger than

the pivot Li.

This local rearrangement is done in place using the collapsing the loops approach of

quicksort.

The next step of the algorithm is to rearrange the elements of the original array A so that all

the elements that are smaller than the pivot are stored at the beginning of the array, and all

the elements that are larger than the pivot are stored at the end of the array.

Once this global rearrangement is done then the algorithm proceeds to partition the

processes into two groups, and assign to the first group the task of sorting the smaller

elements S, and to the second group the task of sorting the larger elements L.

Each of these steps is performed by recursively calling the parallel quicksort algorithm.

Note that by simultaneously partitioning both the processes and the original array each

group of processes can proceeded independently.

The recursion ends when a particular sub-block of elements is assigned to only a single

process, in which case the process sorts the elements using a serial quicksort algorithm.

Message-Passing Parallel Formulation

The message-passing version of the parallel quicksort, each process stores n/p elements of

array A. This array is also partitioned around a particular pivot element using a two-phase

approach.



In the first phase (which is similar to the shared-address-space formulation), the locally stored

array Ai at process Pi is partitioned into the smaller-than- and larger-than-the-pivot sub-arrays

Si and Li locally.

In the next phase, the algorithm first determines which processes will be responsible for

recursively sorting the smaller-than-the-pivot sub-arrays and which process will be

responsible for recursively sorting the larger-than-the-pivot sub-arrays

Once this is done, then the processes send theirS i and Li arrays to the corresponding

processes. After that, the processes are partitioned into the two groups, one for S and one for

L, and the algorithm proceeds recursively.

The recursion terminates when each sub-array is assigned to a single process, at which point it

is sorted locally.

Thus, the overall complexity for the split is Q(n/p) + Q(log p).

Pivot Selection

Pivot selection is particularly difficult, and it significantly affects the algorithm's performance.

In the parallel formulation, however, one poor pivot may lead to a partitioning in which a

process becomes idle, and that will persist throughout the execution of the algorithm.

If the initial distribution of elements in each process is uniform, then a better pivot selection

method can be derived.

Since the distribution of elements on each process is the same as the overall distribution of the

n elements, the median selected to be the pivot during the first step is a good approximation of

the overall median. Since the selected pivot is very close to the overall median, roughly half

of the elements in each process are smaller and the other half larger than the pivot.

Therefore, the first split leads to two partitions, such that each of them has roughly n/2

elements. Similarly, the elements assigned to each process of the group that is responsible for

sorting the smaller-than-the-pivot elements (and the group responsible for sorting the larger-

than-the-pivot elements) have the same distribution as the n/2 smaller (or larger) elements of

the original list.

Thus, the split not only maintains load balance but also preserves the assumption of uniform

element distribution in the process group.

Therefore, the application of the same pivot selection scheme to the sub-groups of processes

continues to yield good pivot selection.



5. By defining what a minimum spanning tree is explain the Prim’s sequential and parallel

algorithms. Also mention their complexity. (November 2013)

A spanning tree of an undirected graph G is a subgraph of G that is a tree containing all the

vertices of G. In a weighted graph, the weight of a subgraph is the sum of the weights of the

edges in the subgraph.

A minimum spanning tree (MST) for a weighted undirected graph is a spanning tree with

minimum weight. Many problems require finding an MST of an undirected graph.

Figure: An undirected graph and its minimum spanning tree.

If G is not connected, it cannot have a spanning tree. Instead, it has a spanning forest.

Prim's algorithm for finding an MST is a greedy algorithm. The algorithm begins by selecting

an arbitrary starting vertex. It then grows the minimum spanning tree by choosing a new

vertex and edge that are guaranteed to be in a spanning tree of minimum cost.

The algorithm continues until all the vertices have been selected.

Let G = (V, E, w) be the weighted undirected graph for which the minimum spanning tree is to

be found, and leAt = (ai, j) be its weighted adjacency matrix.

Figure: Prim's minimum spanning tree algorithm. The MST is rooted at vertex b. For

each iteration, vertices in VT as well as the edges selected so far are shown in bold. The

array d[v] shows the values of the vertices in V - VT after they have been updated.



Algorithm: Prim's sequential minimum spanning tree algorithm.

1. procedurePRIM_MST(V, E, w, r)

2. begin



3. VT := {r};

4. d[r] := 0;

5. for all v (V - VT )do

6. ifedge (r, v) exists set d[v] := w(r, v);

7. elseset d[v] := ;

8. whileVT V do

9. begin

10. find a vertex u such that d[u] :=min{d[v]|v (V - VT )};

11. VT :=VT {u};


13. d[v] := min{d[v], w(u, v)};

14. endwhile

15. endPRIM_MST

Thus, the total communication cost of each iteration is Q(log p). The parallel run time of this formulation is given by

Since the sequential run time is W = Q(n2), the speedup and efficiency are as follows:

6. Write and explain the single source shortest path. (6 marks)

For a weighted graph G = (V, E, w), the single-source shortest paths problem is to find the

shortest paths from a vertex v V to all other vertices in V. A shortest path from u to v is a minimum-weight path.

Depending on the application, edge weights may represent time, cost, penalty, loss, or any other quantity that accumulates additively along a path and is to be minimized.

Here, we present Dijkstra's algorithm, which solves the single-source shortest-paths problem on both directed and undirected graphs with non-negative weights.

Dijkstra's algorithm, which finds the shortest paths from a single vertex s, is similar to Prim's minimum spanning tree algorithm.



The main difference is that, for each vertex u (V - VT ), Dijkstra's algorithm stores l[u], the minimum cost to reach vertex u from vertex s by means of vertices in VT; Prim's algorithm stores d [u], the cost of the minimum-cost edge connecting a vertex in VT to u. The run time of Dijkstra's algorithm is Q(n2).

Algorithm: Dijkstra's sequential single-source shortest paths algorithm.

1. procedureDIJKSTRA_SINGLE_SOURCE_SP(V, E, w, s)

2. begin

3. VT := {s};


5. if(s, v) exists set l[v] := w(s, v);

6. elseset l[v] := ;

7. whileVT V do

8. begin

9. find a vertex u such that l[u] := min{l[v]|v (V - VT )};

10. VT :=VT {u};


12. l[v] := min{l[v], l[u] + w(u, v)};

13. endwhile

14. endDIJKSTRA_SINGLE_SOURCE_SP

Dijkstra's Algorithm in parallel formulations:

This algorithm can also be used to solve the all-pairs shortest paths problem by executing the single-source algorithm on each process, for each vertex v.

We refer to this algorithm as Dijkstra's all-pairs shortest paths algorithm. Since the complexity of Dijkstra's single-source algorithm is Q (n2), the complexity of the all-pairs algorithm is Q (n3).Parallel Formulations

Dijkstra's all-pairs shortest paths problem can be parallelized in two distinct ways. One approach partitions the vertices among different processes and has each process

compute the single-source shortest paths for all vertices assigned to it. We refer to this approach as the source-partitioned formulation.

Another approach assigns each vertex to a set of processes and uses the parallel formulation of the single-source algorithm to solve the problem on each set of processes. We refer to this approach as the source-parallel formulation.

Thus, the parallel run time of source-partitioned formulation is given by



Thus, the parallel run time of source-parallel formulation is given by

7. Explain and write the parallel algorithms of Floyd’s algorithm.

Let G = (V, E, w) be the weighted graph, and let V = {v1,v2,..., vn} be the vertices of G.

Consider a subset {v1, v2,...,vk} of vertices for some k where k n.

For any pair of vertices vi ,vj V, consider all paths from vi to vjwhose intermediate vertices

belong to the set {v1, v2,..., vk}. Let be the minimum-weight path among them, and let

be the weight of .

If vertex vkis not in the shortest path from vito v j, then is the same as .

However, if v k is in , then we can break into two paths – one from vi to vkand one from vkto vj. Each of these paths uses vertices from {v1, v2,...,vk-1}.

Thus, . These observations are expressed in the following recurrence equation:

Algorithm:Floyd's all-pairs shortest paths algorithm. This program computes the all-pairs shortest paths of the graph G = (V, E ) with adjacency matrix A.

1. procedure FLOYD_ALL_PAIRS_SP(A)

2. begin

3. D(0) = A;

4. for k := 1 to ndo

5. for i := 1 to ndo

6. for j := 1 to ndo



7.

8. endFLOYD_ALL_PAIRS_SP

Parallel Formulation

A generic parallel formulation of Floyd's algorithm assigns the task of computing matrix D (k) for each value of k to a set of processes.

Let p be the number of processes available. Matrix D (k) is partitioned into p parts, and each part is assigned to a process.

Each processcomputes the D (k) values of its partition. To accomplish this, a process must access the corresponding segments of the kthrow and

column of matrix D(k-1) A simple example of 2-D block mapping algorithm is shown below:

Algorithm: Floyd's parallel formulation using the 2-D block mapping. P*,jdenotes all the processes in the jth column, and Pi,* denotes all the processes in the ith row. The matrix D(0) is the adjacency matrix.

1. procedureFLOYD_2DBLOCK(D(0))

2. begin

3. for k := 1 to ndo

4. begin

5. each process Pi,jthat has a segment of the kth row of D(k-1);

broadcasts it to the P*,j processes;

6. each process Pi,jthat has a segment of the kth column of D(k-1);

broadcasts it to the Pi,* processes;

7. each process waits to receive the needed segments;

8. each process Pi,jcomputes its part of the D(k) matrix;

9. end

10. endFLOYD_2DBLOCK

Table 1:The performance and scalability of the all-pairs shortest paths algorithms on various

architectures with O (p) bisection bandwidth. Similar run times apply to all k - d cube

architectures, provided that processes are properly mapped to the underlying processors.



UNIT – IV

QUESTION BANK

2 MARKS

1. What is a dense matrix algorithm?

2. Define Bitonic Sequence. (November 2013)

3. Write Matrix vector multiplication algorithm.

4. What is block matrices operations?

5. What is DNS algorithm?

6. What is LU Factorization?

7. What is pipelined Gaussian elimination?

8. What is partial pivoting?

9. What is scaled speedup?

10. What is sorting?

11. Types of sorting?

12. What is internal sorting?

13. What is external sorting?

14. What is comparison based algorithm?

15. What is comparator?

16. Types of comparator?

17. What is Bitonic sort? (November 2013)

18. What is shell sort?

19. What is quick sort?

20. What is bucket sort?

21. What is sample sort?

22. What is spanning tree?

23. What is weighted graph?



24. What is minimum spanning tree? (April 2013)

25. What is shortest path?

26. What is connected components?

27. Define sorting networks (April 2013)

11 MARKS1. Explain how the parallel implementation of Gaussian elimination algorithm is used in solving

a system of linear equations. (November 2013) (Ref.Pg.No.78)2. Explain in detail Matrix vector multiplication. (April 2013) (Ref.Pg.No.80)3. Discuss in cannon’s and DNS algorithm in details.4. Explain in detail about quick sort in parallel ( April 2013) (Ref.Pg.No.86)5. By defining what a minimum spanning tree is explain the Prim’s sequential and parallel

algorithms. Also mention their complexity. (November 2013) (Ref.Pg.No.89)6. Write and explain the single source shortest path. (6 marks)7. Explain and write the parallel algorithms of Floyd’s algorithm.



UNIT - V

Search Algorithms for Discrete for Discrete Optimization Problems:

Definitions and examples Sequential search algorithms Search overhead factor Parallel depth-first search Parallel best-first search Speedup anomalies in parallel search algorithms.

Dynamic Programming: Overview.

TWO MARKS

1. Define discrete optimization problem ?( April 2013/December 2013)

A discrete optimization problem can be expressed as a tuple (S, f). The set S is a finite or

countably infinite set of all solutions that satisfy specified constraints. This set is called the set of

feasible solutions. The function f is the cost function that maps each element in set S onto the set of

real numbers R. f: S -> R

2. What is a 8-Puzzle problem?

The 8-puzzle problem consists of a 3 x 3 grid containing eight tiles, numbered one through

eight. One of the grid segments (called the "blank") is empty.

A tile can be moved into the blank position from a position adjacent to it, thus creating

a blank in the tile's original position.

Depending on the configuration of the grid, up to four moves are possible: up, down,

left, and right. The initial and final configurations of the tiles are specified.

The objective is to determine a shortest sequence of moves that transforms the initial

configuration to the final configuration.

3. Define the following terminal node, non-terminal node?

Terminal node is a node that has no successors and all other nodes apart from terminal nodes

are called as non terminal nodes

4. What is a state space?

The nodes of a graph are called states the cost of a path is the sum of the edge costs. Such a

graph is called a state space.

5. What is sequential search algorithm? (April 2013)

The sequential search examines the first element in the list and then examines each sequential

element in the list until a match is found. This match could be desired word that you are searching

for, or minimum number in the list



6. What are the various types of sequential search algorithms?

The various types of sequential search algorithms are

Depth First Search Algorithm

Best First Search Algorithm

7. Define DFS?

Depth-first search (DFS) is an algorithm for traversing or searching a tree, tree structure,

or graph. DFS begins by expanding the initial node and generating its successors. In each subsequent

step, DFS expands one of the most recently generated nodes. If this node has no successors then DFS

backtracks and expands a different node. In some DFS algorithms, successors of a node are expanded

in an order determined by their heuristic values.

8. What is backtracking?

Simple backtracking is a depth-first search method that terminates upon finding the first

solution. Thus, it is not guaranteed to find a minimum-cost solution. Simple backtracking uses no

heuristic information to order the successors of an expanded node. A variant, ordered backtracking,

does use heuristics to order the successors of an expanded node.

9. Define DFBB?

Depth-first branch-and-bound (DFBB) exhaustively searches the state space; that is, it

continues to search even after finding a solution path. Whenever it finds a new solution path, it

updates the current best solution path. DFBB discards inferior partial solution paths (that is, partial

solution paths whose extensions are guaranteed to be worse than the current best solution path). Upon

termination, the current best solution is a globally optimal solution.

10. Define iterative Deepening A*?

DFS algorithm may get stuck searching a deep part of the search space when a solution exists

higher up on another branch. For such trees, we impose a bound on the depth to which the DFS

algorithm searches. If the node to be expanded is beyond the depth bound, then the node is not

expanded and the algorithm backtracks. If a solution is not found, then the entire state space is

searched again using a larger depth bound. This technique is called iterative deepening depth-first

search

11. What is BFS?

Best-first search (BFS) algorithms can search both graphs and trees. These algorithms use

heuristics to direct the search to portions of the search space likely to yield solutions. Smaller

heuristic values are assigned to more promising nodes. BFS maintains two lists: open and closed.

12. What are the important parameters of Parallel Depth – First Search?

The important parameters of paralled DFS are the following

Work-Splitting Strategies

Load-Balancing Schemes



13. What is search overhead factor?

Let Wpbe the amount of work done by a single processor, and Wpbe the total amount of work

done by p processors. The search overhead factor of the parallel system is defined as the ratio of the

work done by the parallel formulation to that done by the sequential formulation, or Wp/W

14. What do you mean by lwork splitting strategies?

When work is transferred, the donor's stack is split into two stacks, one of which is sent to the

recipient. In other words, some of the nodes are removed from the donor's stack and added to the

recipient's stack. If too little work is sent, the recipient quickly becomes idle; if too much, the donor

becomes idle. Ideally, the stack is split into two equal pieces such that the size of the search space

represented by each stack is the same. Such a split is called a half-split.

15. Define parallel BFS?

Most important component of best-first search (BFS) algorithms is the open list. It maintains

the unexpanded nodes in the search graph, ordered according to their l-value. In the sequential

algorithm, the most promising node from the open list is removed and expanded, and newly generated

nodes are added to the open list. In most parallel formulations of BFS, different processors

concurrently expand different nodes from the open list.

16. What is the communication Strategies for Parallel Best-First Tree Search?

The communication strategies for parallel BFS are

Random communication strategy

Ring communication strategy

Blackboard communication strategy

17. Define random communication strategy?

In the random communication strategy, each processor periodically sends some of its

best nodes to the open list of randomly selected processor.

This strategy ensures that, if a processor stores a good part of the search space, the

others get part of it. If nodes are transferred frequently, the search overhead factor can

be made very small; otherwise it can become quite large.

The communication cost determines the best node transfer frequency. If the

communication cost is low, it is best to communicate after every node expansion.



18. Illustrate the Blackboard communication strategy?

19. What is Speedup Anomalies in Parallel Search Algorithms?

In parallel search algorithms, speedup can vary greatly from one execution to another because

the portions of the search space examined by various processors are determined dynamically and can

differ for each execution.’ The order of node expansions is indicated by node labels. The sequential

formulation generates 13 nodes before reaching the goal node G.

20. Define Dynamic programming?( April/May 2013)

Dynamic programming (DP) is a commonly used technique for solving a wide variety of

discrete optimization problems such as scheduling, string-editing, packaging, and inventory

management.. DP views a problem as a set of interdependent subproblems. It solves subproblems and

uses the results to solve larger subproblems until the entire problem is solved.



11 MARKS

1. Explain sequential search algorithm?

The most suitable sequential search algorithm to apply to a state space depends on whether the space forms a graph or a tree.

In a tree, each new successor leads to an unexplored part of the search space. An example of this is the 0/1 integer-programming problem.

In a graph, however, a state can be reached along multiple paths. An example of such a problem is the 8-puzzle.

For such problems, whenever a state is generated, it is necessary to check if the state has already been generated. If this check is not performed, then effectively the search graph is unfolded into a tree in which a state is repeated for every path that leads to it (Figure).

Figure: Two examples of unfolding a graph into a tree.

For many problems (for example, the 8-puzzle), unfolding increases the size of the search space by a small factor.

For some problems, however, unfolded graphs are much larger than the original graphs. Figure illustrates a graph whose corresponding tree has an exponentially higher number of

states. In this section, we present an overview of various sequential algorithms used to solve DOPs that are formulated as tree or graph search problems.

2. Explain In Detail About Depth-First Search With Example?

Depth-first search (DFS) algorithms solve DOPs that can be formulated as tree-search

problems. DFS begins by expanding the initial node and generating its successors. In each subsequent

step, DFS expands one of the most recently generated nodes. If this node has no successors (or cannot

lead to any solutions), then DFS backtracks and expands a different node.



In some DFS algorithms, successors of a node are expanded in an order determined by their

heuristic values. A major advantage of DFS is that its storage requirement is linear in the depth of the

state space being searched. The following sections discuss three algorithms based on depth-first

search.

Simple Backtracking

Simple backtracking is a depth-first search method that terminates upon finding the first

solution. Thus, it is not guaranteed to find a minimum-cost solution. Simple backtracking uses no

heuristic information to order the successors of an expanded node. A variant, ordered backtracking,

does use heuristics to order the successors of an expanded node.

Depth-First Branch-and-Bound

Depth-first branch-and-bound (DFBB) exhaustively searches the state space; that is, it

continues to search even after finding a solution path. Whenever it finds a new solution path, it

updates the current best solution path. DFBB discards inferior partial solution paths (that is, partial

solution paths whose extensions are guaranteed to be worse than the current best solution path). Upon

termination, the current best solution is a globally optimal solution.

Iterative Deepening A*

Trees corresponding to DOPs can be very deep. Thus, a DFS algorithm may get stuck

searching a deep part of the search space when a solution exists higher up on another branch. For

such trees, we impose a bound on the depth to which the DFS algorithm searches. If the node to be

expanded is beyond the depth bound, then the node is not expanded and the algorithm backtracks. If a

solution is not found, then the entire state space is searched again using a larger depth bound. This

technique is called iterative deepening depth-first search (IDDFS).

Note that this method is guaranteed to find a solution path with the fewest edges. However, it is not

guaranteed to find a least-cost path.

Iterative deepening A* (IDA*) is a variant of ID-DFS. IDA* uses the l-values of nodes to

bound depth (recall from Section 11.1 that for node x, l(x) = g(x) + h(x)). IDA* repeatedly performs

cost-bounded DFS over the search space. In each iteration, IDA* expands nodes depth-first. If the l-

value of the node to be expanded is greater than the cost bound, then IDA*backtracks. If a solution is

not found within the current cost bound, then IDA* repeats the entire depth-first search using a higher

cost bound. In the first iteration, the cost bound is set to the l value of the initial state s. Note that

since g(s) is zero, l(s) is equal to h(s). In each subsequent iteration, the cost bound is increased. The

new cost bound is equal to the minimum l-value of the nodes that were generated but could not be



expanded in the previous iteration. The algorithm terminates when a goal node is expanded. IDA* is

guaranteed to find an optimal solution if the heuristic function is admissible. It may appear that IDA*

performs a lot of redundant work across iterations. However, for many problems the redundant work

performed by IDA* is minimal, because most of the work is done deep in the search space.

Example: Depth-first search: the 8-puzzle

Figure shows the execution of depth-first search for solving the 8-puzzleproblem. The search

starts at the initial configuration. Successors of this state are generated by applying possible moves.

During each step of the search algorithm anew state is selected, and its successors are generated. The

DFS algorithm expands the deepest node in the tree. In step 1, the initial state A generates states B

and C. One of these is selected according to a predetermined criterion. In the example, we order

successors by applicable moves as follows: up, down, left, and right. In step 2,the DFS algorithm

selects state B and generates states D, E, and F. Note that the state D can be discarded, as it is a

duplicate of the parent of B. In step 3, state E is expanded to generate states G and H. Again G can be

discarded because it is a duplicate of B. The search proceeds in this way until the algorithm

backtracks or the final configuration is generated.

Figure: States resulting from the first three steps of

Depth-first search applied to an instance of the 8-puzzle.

In each step of the DFS algorithm, untried alternatives must be stored. For example, in the 8-puzzle

problem, up to three untried alternatives are stored at each step. In general, if m is the amount of

storage required to store a state, and d is the maximum depth, then the total space requirement of the

DFS algorithm is O (md). The state-space tree searched by parallel DFS can be efficiently represented



as a stack. Since the depth of the stack increases linearly with the depth of the tree, the memory

requirements of a stack representation are low.

There are two ways of storing untried alternatives using a stack.

In the first representation, untried alternates are pushed on the stack at each step. The

ancestors of a state are not represented on the stack. Figure (b) illustrates this representation

for the tree shown in Figure (a).

In the second representation, shown in Figure (c), untried alternatives are stored along with

their parent state. It is necessary to use the second representation if the sequence of

transformations from the initial state to the goal state is required as a part of the solution.

Furthermore, if the state space is a graph in which it is possible to generate an ancestor state by

applying a sequence of transformations to the current state, then it is desirable to use the second

representation, because it allows us to check for duplication of ancestor states and thus remove any

cycles from the state-space graph. The second representation is useful for problems such as the 8-

puzzle. In Example 11.5, using the second representation allows the algorithm to detect that nodes D

and G should be discarded.

Figure: Representing a DFS tree: (a) the DFS tree; successor nodes shown with dashed lines

have already been explored; (b) the stack storing untried alternatives only; and (c) the stack

storing untried alternatives along with their parent. The shaded blocks represent the parent

state and the block to the right represents successor states that have not been explored.



3. Explain About Best-First Search Algorithms With Example?

Best-first search (BFS) algorithms can search both graphs and trees. These algorithms use heuristics to direct the search to portions of the search space likely to yield solutions. Smaller heuristic values are assigned to more promising nodes.

BFS maintains two lists: open and closed.

At the beginning, the initial node is placed on the open list. This list is sorted according to a heuristic evaluation function that measures how likely each node is to yield a solution. In each step of the search, the most promising node from the open list is removed. If this node is a goal node, then the algorithm terminates. Otherwise, the node is expanded.

The expanded node is placed on the closed list. The successors of the newly expanded node are placed on the open list under one of the following circumstances: (1) the successor is not already on the open or closed lists, and (2) the successor is already on the open or closed list but has a lower heuristic value. In the second case, the node with the higher heuristic value is deleted.

A common BFS technique is the A* algorithm. The A* algorithm uses the lower bound function l as a heuristic evaluation function. Recall from Section 11.1 that for each node x ,l(x) is the sum of g(x) and h(x). Nodes in the open list are ordered according to the value of the l function.

At each step, the node with the smallest l-value (that is, the best node) is removed from the open list and expanded. Its successors are inserted into the open list at the proper positions and the node itself is inserted into the closed list. For an admissible heuristic function, A* finds an optimal solution.

The main drawback of any BFS algorithm is that its memory requirement is linear in the size of the search space explored. For many problems, the size of the search space is exponential in the depth of the tree expanded. For problems with large search spaces, memory becomes alimitation.

Example Best-first search: the 8-puzzle

Consider the 8-puzzle problem from Examples 11.2 and 11.4. Figure 11.6 illustrates four steps of best-first search on the 8-puzzle. At each step, a state x with the minimum l-value (l(x) = g(x) + h(x)) is selected for expansion. Ties are broken arbitrarily. BFS can check for a duplicate nodes, since all previously generated nodes are kept on either the open or closed list.

Figure: Applying best-first search to the 8-puzzle: (a)initial configuration; (b) final configuration; and (c) states resulting from the first four steps of best-first search. Each state is labeled with its h-value (that is, the Manhattan distance from the state to the final state).



4. Explain In Detail About Search Overhead Factor? (6 MARKS)

Parallel search algorithms incur overhead from several sources. These include communication overhead, idle time due to load imbalance, and contention for shared data structures. Thus, if both the sequential and parallel formulations of an algorithm do the same amount of work, the speedup of parallel search on p processors is less than p. However, the amount of work done by a parallel formulation is often different from that done by the corresponding sequential formulation because they may explore different parts of the search space.

Let W be the amount of work done by a single processor, and Wpbe the total amount of work done by p processors. The search overhead factor of the parallel system is defined as the ratio of the work done by the parallel formulation to that done by the sequential formulation, or Wp/W. Thus, the upper bound on speedup for the parallel system is given by p x(W/Wp). The actual speedup, however, may be less due to other parallel processing overhead. In most parallel search algorithms, the search overhead factor is greater than one. However, in some cases, it may be less than one, leading to superlinear speedup. If the search overhead factor is less than one on the average, then it indicates that the serial search algorithm is not the fastest algorithm for solving the problem.

To simplify our presentation and analysis, we assume that the time to expand each node is the same, and W and Wpare the number of nodes expanded by the serial and the parallel formulations, respectively. If the time for each expansion is tc, then the sequential run time is given by TS =



tcW. In the remainder of the chapter, we assume that tc= 1. Hence, the problem sizeW and the serial run time TS become the same.

5. Explain in detail about parallel depth-first search? ( April 2013) (November 2013)

The critical issue in parallel depth-first search algorithms is the distribution of the search space among the processors. Consider the tree shown in Figure 11.7. Note that the left subtree (rooted at node A) can be searched in parallel with the right subtree (rooted at node B). By statically assigning a node in the tree to a processor, it is possible to expand the whole subtree rooted at that node without communicating with another processor. Thus, it seems that such a static allocation yields a good parallel search algorithm.

Figure: The unstructured nature of tree search and the imbalance resulting from static partitioning.

Let us see what happens if we try to apply this approach to the tree in Figure 11.7. Assume that we have two processors. The root node is expanded to generate two nodes (A and B), and each of these nodes is assigned to one of the processors. Each processor now searches the subtrees rooted at its assigned node independently. At this point, the problem with static node assignment becomes apparent. The processor exploring the subtree rooted at node A expands considerably fewer nodes than does the other processor. Due to this imbalance in the workload, one processor is idle for a significant amount of time, reducing efficiency. Using a larger number of processors worsens the imbalance. Consider the partitioning of the tree for four processors.

Nodes A and B are expanded to generate nodes C, D, E, and F. Assume that each of these nodes is assigned to one of the four processors. Now the processor searching the subtree rooted at node E does most of the work, and those searching the subtrees rooted at nodes C and D spend most of their time



idle. The static partitioning of unstructured trees yields poor performance because of substantial variation in the size of partitions of the search space rooted at different nodes. Furthermore, since the search space is usually generated dynamically, it is difficult to get a good estimate of the size of the search space beforehand. Therefore, it is necessary to balance the search space among processors dynamically.

In dynamic load balancing, when a processor runs out of work, it gets more work from another processor that has work. Consider the two-processor partitioning of the tree in Figure (a). Assume that nodes A and B are assigned to the two processors as we just described. In this case when the processor searching the subtree rooted at node A runs out of work, it requests work from the other processor. Although the dynamic distribution of work results in communication overhead for work requests and work transfers, it reduces load imbalance among processors. This section explores several schemes for dynamically balancing the load between processors.

A parallel formulation of DFS based on dynamic load balancing is as follows. Each processor performs DFS on a disjoint part of the search space. When a processor finishes searching its part of the search space, it requests an unsearched part from other processors. This takes the form of work request and response messages in message passing architectures, and locking and extracting work in shared address space machines. Whenever a processor finds a goal node, all the processors terminate. If the search space is finite and the problem has no solutions, then all the processors eventually run out of work, and the algorithm terminates.

Since each processor searches the state space depth-first, unexplored states can be conveniently stored as a stack. Each processor maintains its own local stack on which it executes DFS. When a processor's local stack is empty, it requests (either via explicit messages or by locking) untried alternatives from another processor's stack. In the beginning, the entire search space is assigned to one processor, and other processors are assigned null search spaces (that is, empty stacks). The search space is distributed among the processors as they request work. We refer to the processor that sends work as the donor processor and to the processor that requests and receives work as the recipient processor.

As illustrated in Figure each processor can be in one of two states: active (that is, it has work) or idle (that is, it is trying to get work). In message passing architectures, an idle processor selects a donor processor and sends it a work request. If the idle processor receives work (part of the state space to be searched) from the donor processor, it becomes active. If it receives a reject message (because the donor has no work), it selects another donor and sends a work request to that donor. This process repeats until the processor gets work or all the processors become idle. When a processor is idle and it receives a work request, that processor returns a reject message. The same process can be implemented on shared address space machines by locking another processors' stack, examining it to see if it has work, extracting work, and unlocking the stack.

Figure: A generic scheme for dynamic load balancing.



On message passing architectures, in the active state, a processor does a fixed amount of work (expands a fixed number of nodes) and then checks for pending work requests. When a work request is received, the processor partitions its work into two parts and sends one part to the requesting processor. When a processor has exhausted its own search space, it becomes idle.

This process continues until a solution is found or until the entire space has been searched. If a solution is found, a message is broadcast to all processors to stop searching. A termination detection algorithm is used to detect whether all processors have become idle without finding a solution

Important Parameters of Parallel DFS

Two characteristics of parallel DFS are critical to determining its performance. First is the method for splitting work at a processor, and the second is the scheme to determine the donor processor when a processor becomes idle.

Work-Splitting Strategies

When work is transferred, the donor's stack is split into two stacks, one of which is sent to the recipient. In other words, some of the nodes (that is, alternatives) are removed from the donor's stack and added to the recipient's stack. If too little work is sent, the recipient quickly becomes idle; if too much, the donor becomes idle. Ideally, the stack is split into two equal pieces such that the size of the search space represented by each stack is the same. Such a split is called a half-split. It is difficult to get a good estimate of the size of the tree rooted at an unexpanded alternative in the stack. However, the alternatives near the bottom of the stack (that is, close to the initial node) tend to have bigger trees rooted at them, and alternatives near the top of the stack tend to have small trees rooted at them. To



avoid sending very small amounts of work, nodes beyond a specified stack depth are not given away. This depth is called the cutoff depth.

Some possible strategies for splitting the search space are (1) send nodes near the bottom of the stack, (2) send nodes near the cutoff depth, and (3) send half the nodes between the bottom of the stack and the cutoff depth. The suitability of a splitting strategy depends on the nature of the search space. If the search space is uniform, both strategies 1 and 3 work well. If the search space is highly irregular, strategy 3 usually works well. If a strong heuristic is available (to order successors so that goal nodes move to the left of the state-space tree), strategy 2 is likely to perform better, since it tries to distribute those parts of the search space likely to contain a solution. The cost of splitting also becomes important if the stacks are deep .For such stacks, strategy 1 has lower cost than strategies 2 and 3.

Figure: shows the partitioning of the DFS tree of Figure 11.5(a)into two subtrees using strategy 3. Note that the states beyond the cutoff depth are not partitioned. Figure 11.9alsoshows the representation of the stack corresponding to the two subtrees. The stack representation used in the figure stores only the unexplored alternatives.

Figure: Splitting the DFS tree in Figure 11.5. The two subtrees along with their stack representations are shown in (a) and (b).

This section discusses three dynamic load-balancing schemes: asynchronous round robin, global round robin, and random polling. Each of these schemes can be coded for message passing as well as shared address space machines. Asynchronous Round Robin In asynchronous round robin (ARR), each processor maintains an independent variable, target. Whenever a processor runs out of work, it uses target as the label of a donor processor and attempts to get work from it. The value of target is incremented (modulo p) each time a work request is sent. The initial value of target at each processor is set to ((label+ 1) modulo p) where label is the local processor label. Note that work requests are



generated independently by each processor. However, it is possible for two or more processors to request work from the same donor at nearly the same time.

Global Round Robin Global round robin (GRR) uses a single global variable called target. This variable can be stored in a globally accessible space in shared address space machines or at a designated processor in message passing machines. Whenever a processor needs work, it requests and receives the value of target, either by locking, reading, and unlocking on shared address space machines or by sending a message requesting the designated processor (say P0).

The value of target is incremented (modulo p) before responding to the next request. The recipient processor then attempts to get work from a donor processor whose label is the value of target. GRR ensures that successive work requests are distributed evenly over all processors. A drawback of this scheme is the contention for access to target.

Random PollingRandom polling (RP) is the simplest load-balancing scheme. When a processor becomes idle, it randomly selects a donor. Each processor is selected as a donor with equal probability, ensuring that work requests are evenly distributed.

6. Explain About Dynamic Programming? (April 2013)

Dynamic programming (DP) is a commonly used technique for solving a wide variety of

discrete optimization problems such as scheduling, string-editing, packaging, and inventory

management. More recently, it has found applications in bioinformatics in matching sequences of

amino-acids and nucleotides (the Smith-Waterman algorithm). DP views a problem as a set of

interdependent sub problems. It solves sub problems and uses the results to solve larger sub problems

until the entire problem is solved. In contrast to divide-and-conquer, where the solution to a problem

depends only on the solution to its sub problems, in DP there may be interrelationships across sub

problems. In DP, the solution to a sub problem is expressed as a function of solutions to one or more

sub problems at the preceding levels.

Overview of Dynamic programming

Example 12.1 the shortest-path problem

Consider a DP formulation for the problem of finding a shortest (least-cost) path between a pair of vertices in an acyclicgraph. (Refer to Section 10.1 for an introduction to graph terminology.) An edge connecting node i to node j has cost c(i,j). If two vertices i and j are not connected then c(i, j) = . The graph contains n nodes numbered 0, 1, ...,n - 1, and has an edge from node i to node j only if i <j. The shortest-path problem is to find a least-cost path between nodes 0 and n - 1. Let f (x) denote the cost of the least-cost path from node 0 to node x. Thus, f (0) is zero, and finding f (n - 1) solves the problem. The DP formulation for this problem yields the following recursive equations for f (x):



As an instance of this algorithm, consider the five-node acyclic graph shown in Figure. The problem is to find f (4). It can be computed given f (3) and f (2). More precisely,Figure

Figure: A graph for which the shortest path between nodes 0 and 4 is to beComputed

Therefore, f (2) and f (3) are elements of the set of subproblems on which f (4) depends. Similarly, f (3) depends on f (1)and f (2),andf (1) and f (2) depend on f (0). Since f (0) is known, it is used to solve f (1) and f (2), which are used to solve f(3).

In general, the solution to a DP problem is expressed as a minimum (or maximum) of possible alternate solutions. Each of these alternatesolutions is constructed by composing one or more subproblems. If r represents the cost of a solution composed of subproblemsx1,x2, ...,xl, then r can be written as

The function g is called the composition function, and its nature depends on the problem. If the optimal solution to each problem is determined by composing optimal solutions to the subproblems and selecting the minimum (or maximum), the formulation is said to be a DP formulation. Figure: illustrates an instance of composition and minimization of solutions. The solution to problem x8 is the minimum of the three possible solutions having costs r1, r2, and r3. The cost of the first solution is determined by composing solutions tosubproblemsx1 and x3, the second solution by composing solutions to subproblemsx4 and x5, and the third solution by composing solutions to subproblemsx2, x6, and x7.

Figure: The computation and composition of subproblem solutions to solve problem f (x8).

DP represents the solution to an optimization problem as a recursive equation whose left side is an unknown quantity and whose right side is a minimization (or maximization) expression. Such an equation is called a functional equation or an optimization equation.

In Equation, the composition function g is given by f (j) + c(j, x). This function is additive, since it is the sum of two terms. In a general DPformulation, the cost function need not be additive. A functional equation that contains a single recursive term (for example, f (j)) yields a monadic DP formulation. For an arbitrary DP formulation, the cost function may contain multiple recursive terms.



DP formulations whose cost function contains multiple recursive terms are called polyadic formulations.

The dependencies between sub problems in a DP formulation can be represented by a directed graph. Each node in the graph represents a subproblem. A directed edge from node i to node j indicates that the solution to the subproblem represented by node i is used to compute the solution to the subproblem represented by node j. If the graph is acyclic, then the nodes of the graph can be organized into levels suchthat subproblems at a particular level depend only on subproblems at previous levels. In this case, the DP formulation can be categorized as follows.

If subproblems at all levels depend only on the results at the immediately preceding levels, the formulation is called a serial DP formulation; otherwise, it is called a nonserial DP formulation.Based on the preceding classification criteria, we define four classes of DP formulations: serial monadic, serial polyadic, nonserialmonadic, and nonserialpolyadic. These classes, however, are not exhaustive; some DP formulations cannot be classified into any ofthese categories.

Due to the wide variety of problems solved using DP, it is difficult to develop generic parallel algorithms for them. However, parallel formulations of the problems in each of the four DP categories have certain similarities. In this chapter, we discuss parallel DP formulations for sample problems in each class. These samples suggest parallel algorithms for other problems in the same class. Note, however, that not all DP problems can be parallelized as illustrated in these examples.

Serial Monadic DP FormulationsWe can solve many problems by using serial monadic DP formulations. This section discusses

the shortest-path problem for a multistage graph and the 0/1 knapsack problem. We present parallel algorithms for both and point out the specific properties that influence these parallel formulations.

The Shortest-Path ProblemConsider a weighted multistage graph of r + 1 level, as shown in Figure. Each node at level i

is connected to every node at level i + 1.Levels zero and r contain only one node, and every other level contains n nodes. We refer to the node at level zero as the starting node S and the node at level r



as the terminating node R. The objective of this problem is to find the shortest path from S to R. The I th node at level l in the graph is labeled. The cost of an edge connecting to node is labeled. The cost of reaching the goal node R from any node is represented by. If there are n nodes at level l, the vector is referred to as Cl.

The shortest-path problem reduces to computing C0. Since the graph has only one starting node, . The structure of thegraph is such that any path from to R includes a node (0 j n - 1). The cost of any such path is the sum of the cost ofthe edge between and and the cost of the shortest path between and R (which is given by ). Thus, thecost of the shortest path between and R, is equal to the minimum cost over all paths through each node in levell + 1. Therefore,

An example of a serial monadic DP formulation for finding the shortest path in a graph whose nodes can be organized into levels.

The 0/1 Knapsack ProblemA one-dimensional 0/1 knapsack problem is defined as follows. We are given a knapsack of

capacity c and a set of n objects numbered 1,2, ..., n. Each object i has weight wi and profit pi. Object profits and weights are integers. Let v = [v1, v2, ...,vn] be a solution vector in which vi = 0 if object i is not in the knapsack, and vi = 1 if it is in the knapsack. The goal is to find a subset of objects to put into the knapsack so that

7. Explain Briefly About Parallel Best-First search? (November 2013)

An important component of best-first search (BFS) algorithms is the open list. It maintains the unexpanded nodes in the search graph, ordered according to their l-value. In the sequential algorithm, the most promising node from the open list is removed and expanded, and newly generated nodes are added to the open list.



In most parallel formulations of BFS, different processors concurrently expand different nodes from the open list. These formulations differ according to the data structures they use to implement the open list. Given p processors, the simplest strategy assigns each processor to work on one of the current best nodes on the open list. This is called the centralized strategy because each processor gets work from asingle global open list.

Since this formulation of parallel BFS expands more than one node at a time, it may expand nodes that would not be expanded by a sequential algorithm. Consider the case in which the first node on the open list is a solution. The parallel formulation still expands the first p nodes on the open list. However, since it always picks the best p nodes, the amount of extra work is limited. Figure: illustrates this strategy. There are two problems with this approach:

The termination criterion of sequential BFS fails for parallel BFS. Since at any moment, p nodes from the open list are being expanded, it is possible that one of the nodes may be a solution that does not correspond to the best goal node (or the path found is not the shortest path). This is because the remaining p - 1 nodes may lead to search spaces containing better goal nodes. Therefore, if the cost of a solution found by a processor is c, then this solution is not guaranteed to correspond to thebest goal node until the cost of nodes being searched at other processors is known to be at least c.

The termination criterion must be modified to ensure that termination occurs only after the best solution has been found.1.Since the open list is accessed for each node expansion; it must be easily accessible to all processors, which can severely limit performance. Even on shared-address-space architectures, contention for the open list limits speedup. Let texp be the average time to expand a single node and t access be the average time to access the open list for a single-node expansion. If there are n nodes to be expanded by both the sequential and parallel formulations (assuming that they do an equal amount of work), then the sequential run time is given by n(taccess+ texp). Assume that it is impossible to parallelize the expansion of individual nodes. Then the parallel run time will be at least nt access, because the open list must be accessed at least once for each node expanded. Hence, an upper bound on the speedup is (taccess+ texp)/taccess.

Figure: A general schematic for parallel best-first search using a centralized strategy. The locking operation is used here to serialize queue access by various processors.



One way to avoid the contention due to a centralized open list is to let each processor have a local open list. Initially, the search space is statically divided among the processors by expanding some nodes and distributing them to the local open lists of various processors.

All the processors then select and expand nodes simultaneously. Consider a scenario where processors do not communicate with each other. In this case, some processors might explore parts of the search space that would not be explored by the sequential algorithm. This leads to a high search overhead factor and poor speedup. Consequently, the processors must communicate among themselves to minimize unnecessary search. The use of a distributed open list trades-off communication and computation: decreasing communication between distributed open lists increases search overhead factor, and decreasing search overhead factor with increased communication increases communication overhead.

The best choice of communication strategy for parallel BFS depends on whether the search space is a tree or a graph. Searching a graph incurs the additional overhead of checking for duplicate nodes on the closed list. We discuss some communication strategies for tree andgraph search separately.

Communication Strategies for Parallel Best-First Tree Search

A communication strategy allows state-space nodes to be exchanged between open lists on different processors. The objective of a communication strategy is to ensure that nodes with good l-values are distributed evenly among processors. In this section we discuss three such strategies, as follows.

In the random communication strategy, each processor periodically sends some of its best nodes to the open list of a randomly selected processor. This strategy ensures that, if a processor stores a good part of the search space, the others get part of it. If nodes are transferred frequently, the search overhead factor can be made very small; otherwise it can become quite large. The communication cost determines the best node transfer frequency. If the communication cost is low, it is best to communicate after every node expansion.

In the ring communication strategy, the processors are mapped in a virtual ring. Each processor periodically exchanges some of its best nodes with the open lists of its neighbors in the ring. This strategy can be implemented on message passing as well as shared address space machines with the processors organized into a logical ring. As before, the cost ofcommunication determines the node transfer frequency. Figure: illustrates the ring communication strategy.



Figure: A message-passing implementation of parallel best-first search using the ring communication strategy.

.

8. Explain About Speedup Anomalies In Parallel Search Algorithms?

In parallel search algorithms, speedup can vary greatly from one execution to another because the portions of the search space examined by various processors are determined dynamically and can differ for each execution. Consider the case of sequential and parallel DFS performed on the tree illustrated in Figure (a) illustrates sequential DFS search. The order of node expansions is indicated by node labels. The sequential formulation generates 13 nodes before reaching the goal node G.

Figure: The difference in number of nodes searched by sequential and parallel formulations of DFS. For this example, parallel DFS reaches a goal node after searching fewer nodes than sequential DFS.

Now consider the parallel formulation of DFS illustrated for the same tree in Figure (b) for two processors. The nodes expanded by the processors are labeled R and L. The parallel formulation reaches the goal node after generating only nine nodes. That is, the parallel formulation arrives at the goal node after searching fewer nodes than its sequential counterpart. In this case, the search overhead factor is9/13 (less than one), and if communication overhead is not too large, the speedup will be superlinear. Finally, consider the situation in Figure 11.18. The sequential formulation (Figure



11.18(a)) generates seven nodes before reaching the goal node, but the parallel formulation generates 12 nodes. In this case, the search overhead factor is greater than one, resulting in sublinear speedup.Figure: A parallel DFS formulation that searches more nodes than its sequential counterpart.

In summary, for some executions, the parallel version finds a solution after generating fewer nodes than the sequential version, making itpossible to obtain superlinear speedup. For other executions, the parallel version finds a solution after generating more nodes, resulting insublinear speedup. Executions yielding speedups greater than p by using p processors are referred to as acceleration anomalies.

Speedups of less than p using p processors are called deceleration anomalies. Speedup anomalies also manifest themselves in best-first search algorithms. Here, anomalies

are caused by nodes on the open list that have identical heuristic values but require vastly different amounts of search to detect a solution.

Assume that two such nodes exist; node A leads rapidly to the goal node, and node B leads nowhere after extensive work. In parallel BFS, both nodes are chosen for expansion by different processors. Consider the relative performance of parallel and sequential BFS. If the sequential algorithm picks node A to expand, it arrives quickly at a goal. However, the parallel algorithm wastes time expanding node B, leading to a deceleration anomaly. In contrast, if the sequential algorithm expands node B, it wastes substantial time before abandoning it in favor of node A. However, the parallel algorithm does not waste as much time on node B, because node A yields a solution quickly, leading to an acceleration anomaly.

Analysis of Average Speed up in Parallel DFS

In isolated executions of parallel search algorithms, the search overhead factor may be equal to one, less than one, or greater than one. It is interesting to know the average value of the search overhead factor. If it is less than one, this implies that the sequential search algorithm is not optimal. In this case, the parallel search algorithm running on a sequential processor (by emulating a parallel processor by using time-slicing) would expand fewer nodes than the sequential algorithm on the average. In this section, we show that for a certain type of search space, the average value of the search overhead factor in parallel DFS is less than one. Hence, if the communication overhead is not too large, then on the average, parallel DFS will provide superlinear speedup for this type of search space.

Assumptions



We make the following assumptions for analyzing speedup anomalies:

The state-space tree has M leaf nodes. Solutions occur only at leaf nodes. The amount of computation needed to generate each leaf node is the same. The number of nodes generated in the tree is proportional to the number of leaf nodes generated. This is a reasonable assumption for search trees in which each node has more than one successor on the average.Both sequential and parallel DFS stop after finding one solution. In parallel DFS, the state-space tree is equally partitioned among p processors; thus, each processor gets a subtree with M/p leaf nodes.

There is at least one solution in the entire tree. (Otherwise, both parallel search and sequential search generate the entire tree without finding a solution, resulting in linear speedup.)

There is no information to order the search of the state-space tree; hence, the density of solutions across the unexplored nodes is independent of the order of the search.

The solution density r is defined as the probability of the leaf node being a solution. We assume a Bernoulli distribution of solutions; that is, the event of a leaf node being a solution is independent of any other leaf node being a solution. We also assume that r 1.

The total number of leaf nodes generated by p processors before one of the processors finds a solution is denoted by W p. The average number of leaf nodes generated by sequential DFS before a solution is found is given by W. Both W and Wp are less than or equal to M.

Analysis of the Search Overhead Factor

Consider the scenario in which the M leaf nodes are statically divided into p regions, each with K = M/p leaves. Let the density of solutions among the leaves in the ith region be ri. In the parallel algorithm, each processor Pi searches region i independently until a processor finds a solution. In the sequential algorithm, the regions are searched in random order

UNIT – V

QUESTTION BANK

TWO MARKS

1. Define discrete optimization problem ?( April 2013/December 2013)

2. What is a 8-Puzzle problem ?

3. Define the following terminal node , nonterminal node ?

4. What is a state space?

5. What is sequential search algorithm? (April 2013)

6. What are the various types of sequential search algorithms?

7. Define DFS?

8. What is backtracking?

9. Define DFBB ?

10. Define iterative Deepening A* ?

11. What is BFS?

12. What are the important parameters of Parallel Depth – First Search?

13. What is search overhead factor?

14. what do you mean by work splitting strategies ?



15. Define parallel BFS ?

16. What is the communication Strategies for Parallel Best-First Tree Search?

17. Define random communication strategy?

18. Illustrate the Blackboard communication strategy?

19. What is Speedup Anomalies in Parallel Search Algorithms ?

20. Define Dynamic programming? ( April/May 2013)

11 MARKS

1 Explain sequential search algorithm?2 Explain in detail about depth-first search with example?3 Explain about best-first search algorithms with example?4 Explain in detail about search overhead factor? (6 marks)5 Explain in detail about parallel depth-first search? (April 2013) (November 2013)

(Ref.Pg.No.108)6 Explain about dynamic programming? (April 2013) (Ref.Pg.No.114)

7 Explain briefly about parallel best-first search? (November 2013) (Ref.Pg.No.117)8 Explain about speedup anomalies in parallel search algorithms?

Pondicherry University Questions

UNIT- I

1. Why do we need Use Parallel Computing? (April 2013) (Ref.Pg.No.8)2. Explain various methods for solving problems in parallel. (November 2013) (Ref.Pg.No.12)

3. Explain the Various Computation Models? (November 2013) (Ref.Pg.No.18)

UNIT- II

1. Explain the limitations of memory system performance ( April 2013) (Ref.Pg.No.27)

2. Explain in detail about Parallel algorithm models. (April 2013) (Ref.Pg.No.29)

3. Explain decomposition techniques used to achieve concurrency with necessary examples

(November 2013) (Ref.Pg.No.31)

4. Discuss the all-to-all personalized communication on parallel computers with linear array,

mesh and hypercube interconnection networks. (November 2013) (Ref.Pg.No.35)

UNIT- III

1. Elaborate on the various performance metrics used in studying the parallel systems. (April 2013) (November 2013) (Ref.Pg.No.51)



2. Explain any six MPI functions commonly used to perform collective communication operations with an application (November 2013) (Ref.Pg.No.53)

3. Explain collective communication and computation operations (November 2013) (Ref.Pg.No.57)

UNIT- IV

1. Explain how the parallel implementation of Gaussian elimination algorithm is used in solving a system of linear equations. (November 2013) (Ref.Pg.No.78)

2. Explain in detail Matrix vector multiplication. (April 2013) (Ref.Pg.No.80)3. Explain in detail about quick sort in parallel ( April 2013) (Ref.Pg.No.86)4. By defining what a minimum spanning tree is explain the Prim’s sequential and parallel

algorithms. Also mention their complexity. (November 2013) (Ref.Pg.No.89)

UNIT- V

1. Explain in detail about parallel depth-first search? (April 2013) (November 2013) (Ref.Pg.No.108, Qn.No.5)

2. Explain about dynamic programming? (April 2013) (Ref.Pg.No.114,Qn.No.6)

3. Explain briefly about parallel best-first search? (November 2013) (Ref.Pg.No.117,Qn.No.7)


High performance computing

Documents

Transcript of High performance computing