Parallel Processing Steve Terpe CS 147. Overview What is Parallel Processing What is Parallel...

18
Parallel Parallel Processing Processing Steve Terpe Steve Terpe CS 147 CS 147

Transcript of Parallel Processing Steve Terpe CS 147. Overview What is Parallel Processing What is Parallel...

Parallel ProcessingParallel Processing

Steve TerpeSteve TerpeCS 147CS 147

OverviewOverview

What is Parallel ProcessingWhat is Parallel Processing Parallel Processing in NatureParallel Processing in Nature Parallel Processing vs. MultitaskingParallel Processing vs. Multitasking Amdahl’s LawAmdahl’s Law Challenges in Parallel ProcessingChallenges in Parallel Processing

What is Parallel Processing?What is Parallel Processing? How to make machines How to make machines

solve problems better solve problems better and faster?and faster?

Physical barriers limit the Physical barriers limit the extent to which single extent to which single processor performance processor performance can be improved—Clock can be improved—Clock Speed & Heat DissipationSpeed & Heat Dissipation

The next most obvious The next most obvious solution is to distribute solution is to distribute the computing load the computing load among several among several processorsprocessors

What is Parallel Processing?What is Parallel Processing?

Parallel Processing encompasses a wide Parallel Processing encompasses a wide variety of different things:variety of different things:

Intel Core Duo, Quad, Cell multiprocessoIntel Core Duo, Quad, Cell multiprocessors, networked and distributed computer rs, networked and distributed computer systems, SETI@Home, Folding@Home, systems, SETI@Home, Folding@Home, neural nets are all examplesneural nets are all examples

Parallel Processing in NatureParallel Processing in Nature

The world’s most The world’s most powerful parallel powerful parallel processor comes processor comes standardstandard

For everything else For everything else there’s American there’s American Express…Express…

Parallel Processing in NatureParallel Processing in Nature

For humans parallel For humans parallel processing comes eaprocessing comes easysy

Human VisionHuman VisionColor, Motion,Depth, Color, Motion,Depth, ShapeShape

Parallel Processing in NatureParallel Processing in Nature Machines that are more Machines that are more

like us, and hence more like us, and hence more useful, will need to be able useful, will need to be able to process information to process information more like us—in parallel.more like us—in parallel.

Parallel Processing is key Parallel Processing is key element in pattern element in pattern recognition which recognition which distinguishes human & distinguishes human & machine intelligencemachine intelligence

This is not an easy hurdleThis is not an easy hurdle

Parallel Processing vs. MultitaskingParallel Processing vs. Multitasking

Today, computers are great at Today, computers are great at multi-tasking, but…multi-tasking, but…

Multi-tasking only creates the Multi-tasking only creates the illusion of parallel processingillusion of parallel processing

The processor must switch The processor must switch between activities—it is only between activities—it is only the speed with which it does the speed with which it does so that creates the illusion of so that creates the illusion of simultaneous executionsimultaneous execution

The illusion is most easily The illusion is most easily shattered when running virus shattered when running virus scan while attempting to do scan while attempting to do anything elseanything else

Parallel Processing vs. MultitaskingParallel Processing vs. Multitasking

Think about it this way:Think about it this way:

Multi-taskingMulti-tasking

Parallel ProcessingParallel Processing

Amdahl’s LawAmdahl’s Law

Consider a single Consider a single processorprocessor

Or two…Or two… We tend to think that We tend to think that

2x as much work will 2x as much work will be done in the same be done in the same timetime

Or that the same Or that the same amount of work will be amount of work will be done in half the timedone in half the time

Amdahl’s LawAmdahl’s Law

Do n processors Do n processors imply that a imply that a computational job computational job should complete in should complete in 1/n time?1/n time?

Sadly, no…Sadly, no…

Amdahl’s LawAmdahl’s Law 1967 Gene Amdahl recognizes 1967 Gene Amdahl recognizes

the interrelationship of all the interrelationship of all componentscomponents

Overall speedup of a system Overall speedup of a system depends on the speedup of a depends on the speedup of a particular component & how particular component & how much that component is usedmuch that component is used

S = 1/([1- f] + f / k)S = 1/([1- f] + f / k)

S = overall system speedS = overall system speed f = fraction of the work f = fraction of the work

performed by the faster performed by the faster componentcomponent

k = speedup of new k = speedup of new componentcomponent

Amdahl’s LawAmdahl’s Law Additionally, no matter Additionally, no matter

how well (or much) how well (or much) you parallelize an you parallelize an application there will application there will always be a small always be a small portion of work that portion of work that must be done serially.must be done serially.

Other processors must Other processors must simply sit and wait in simply sit and wait in this intervalthis interval

Every algorithm has a Every algorithm has a sequential part that sequential part that limits potential limits potential speedupspeedup

Challenges in Parallel ProcessingChallenges in Parallel Processing

Not always obvious where to “split” Not always obvious where to “split” workload or even possible.workload or even possible.

If you don’t use it, you lose it…If you don’t use it, you lose it…programs not specifically written for programs not specifically written for parallel architecture run no more parallel architecture run no more efficiently on parallel systemsefficiently on parallel systems

Challenges in Parallel ProcessingChallenges in Parallel Processing

Connecting your CPUsConnecting your CPUs Dynamic vs Static—connections can change from one communicatDynamic vs Static—connections can change from one communicat

ion to nextion to next Blocking vs Nonblocking—can simultaneous connections be preseBlocking vs Nonblocking—can simultaneous connections be prese

nt?nt? Connections can be complete, linear, star, grid, tree, hypercube, etConnections can be complete, linear, star, grid, tree, hypercube, et

c.c. Bus-based routingBus-based routing Crossbar switching—impractical for all but the most expensive supCrossbar switching—impractical for all but the most expensive sup

er-computerser-computers 2X2 switch—can route inputs to different destinations2X2 switch—can route inputs to different destinations

Challenges in Parallel ProcessingChallenges in Parallel Processing

Dealing with memoryDealing with memory Various options:Various options: Global Shared MemoryGlobal Shared Memory Distributed Shared MemoryDistributed Shared Memory Global shared memory with Global shared memory with

separate cache for processorsseparate cache for processors

Potential Hazards:Potential Hazards: Individual CPU caches or Individual CPU caches or

memories can become out of memories can become out of synch with each other. synch with each other. “Cache Coherence”“Cache Coherence”

Solutions:Solutions: UMA/NUMA machinesUMA/NUMA machines Snoopy cache controllersSnoopy cache controllers Write-through protocolsWrite-through protocols

ConclusionConclusion

What is Parallel ProcessingWhat is Parallel Processing Parallel Processing in NatureParallel Processing in Nature Parallel Processing vs. MultitaskingParallel Processing vs. Multitasking Amdahl’s LawAmdahl’s Law Challenges in Parallel ProcessingChallenges in Parallel Processing

Questions?Questions?