Slide02 Parallel Computers
Transcript of Slide02 Parallel Computers
![Page 1: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/1.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 1/44
Parallel ComputerArchitecture
![Page 2: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/2.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 2/44
The End of the Road
![Page 3: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/3.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 3/44
Advantages of
Multi rocessors• Able to create powerful computers by
simply connecting multiple processors
• More cost-effective than building a high-performance single processor
• Obtain fault-tolerance to carry on thetasks, albeit with degraded performance
![Page 4: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/4.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 4/44
4 Decades of Computing
![Page 5: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/5.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 5/44
Batch Era (1960s)
• IBM System/360 mainframedominated the corporate
computer centers (10 MBdisk, 1 MB magnetic corememory)
•Typical batch processing
machine
• No connection beyond thecomputer room
![Page 6: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/6.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 6/44
Time-Sharing Era (1970s)
• Advancing in ss-memory &ICs spawned the
minicomputer era
• Small, fast, and inexpensiveenough to be spreadthroughout the company at
the divisional level
• Still too expensive anddifficult to use to hand overto end-users
• Time-sharing computing
•Existing 2 kinds:
• centralized data processingmainframes
• time-sharing minicomputers
![Page 7: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/7.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 7/44
Desktop Era (1980s)
• PCs were introduced in1977
• Many players (Altairs, Tandy,
Commondore, Apple, IBM,and etc)
• Became pervasive and
change the face of computing
• Along came networkedcomputers (LAN & WAN)
![Page 8: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/8.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 8/44
Network Era (1990s)
• Advance network technologiesled to network computingparadigm
• Transition from a processor-centric view of computing to anetwork-centric view
•A number of commercialparallel computers withmultiple processors:
• Shared memory systems
• Distributed memory systems
![Page 9: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/9.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 9/44
Four Decades of ComputingFeature Batch Time-Sharing Desktop Network
Decade 1960s 1970s 1980s 1990s
Location Computer Room Terminal Room Desktop Mobile
Users Experts Specialists Individuals Groups
Data Alphanumeric Text, numbers Fonts, graphs Multimedia
Objective Calculate Access Present Communicate
Interface Punched card Kbd & CRT See & point Ask & tell
Operation Process Edit Layout Orchestrate
Connectivity None Peripheral cable LAN Internet
OwnersCorporate
computer centersDivisional IS
shopsDepartmental
end-usersEveryone
![Page 10: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/10.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 10/44
Current Trends
• The substitution of expensive and specialized parallelmachines by the more cost-effective clusters of workstations
• A cluster is a collection of stand-alone computersconnected using some interconnection network
• A pervasiveness of the Internet created interest innetwork computing and more recently in grid
computing
• Grids are geographically distributed platforms of computation - dependable, consistent, pervasive, andless expensive access to HPC facilities
![Page 11: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/11.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 11/44
Flynn’s Taxonomy of
Com uter Architecture• Based on the notion of a stream of
information
• instruction
•data
CPU
Memory
fetch execute(manipulate data as
programmed)
![Page 12: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/12.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 12/44
SingleInstruction
MultipleInstruction
SingleData
MultipleData
![Page 13: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/13.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 13/44
SIMD Architecture
![Page 14: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/14.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 14/44
Single Instruction,
Multi le Data SIMD
t i m e
P1 P2 Pn
![Page 15: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/15.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 15/44
MIMD Architecture
![Page 16: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/16.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 16/44
Multiple Instruction,
Multi le Data MIMD
t i m e
P1 P2 Pn
![Page 17: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/17.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 17/44
SIMD Architecture Model
• Consists of two parts:
• a front-end
computer
• a processor array
• each element in the processor array is identical toone another and performs operation on different datain sync
•front-end can access PE’s memory via the bus
![Page 18: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/18.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 18/44
SIMD Architecture Model
• lock-step
synchronization
• Processors either donothing or exactly thesame ops simultaneously
• In SIMD, parallelism is exploited by applyingsimultaneous operations across large sets of data
![Page 19: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/19.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 19/44
SIMD Configurations
Each PE has its own localmemory
PEs and memory modulescommunicate via the IN
![Page 20: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/20.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 20/44
![Page 21: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/21.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 21/44
![Page 22: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/22.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 22/44
MIMD Architecture
Interconnection Network
P
MM M M
P P P
Shared Memory MIMD Architecture
Interconnection Network
P P P P
MM M M
Message Passing MIMD Architecture
information exchangethrough central shared
memory
information exchangethrough network in
message passing systems
![Page 23: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/23.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 23/44
MIMD Architecture
Interconnection Network
P
MM M M
P P P
Shared Memory MIMD Architecture
• using bus/cachearchitecture
• called SMP (symmetricmultiprocessor) since
• equal chance to read/
write memory
• equal access speed
![Page 24: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/24.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 24/44
MIMD Architecture
Interconnection Network
P P P P
MM M M
Message Passing MIMD Architecture
• also known asdistributed memory
• no global memory• using message passing to
move data from one toanother (Send/Recieve
pair of commands)
• this architecture giveway to Internet
connected systems
![Page 25: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/25.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 25/44
MIMD Architecture
Interconnection Network
P
MM M M
P P P
Shared Memory MIMD Architecture
Interconnection Network
P P P P
MM M M
Message Passing MIMD Architecture
programming is easier provided scalability
DSM (distributed-shared memory) is
the hybrid between the two
![Page 26: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/26.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 26/44
DSM
• memory is physically distributed [messagepassing]
• memory can be addressed as one (logicallyshared) address space [shared memory]
• programming-wise, the architecture looks
and behaves like a shared memorymachine, but a message passing architecturelives underneath the software
![Page 27: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/27.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 27/44
SGI Origin2000
![Page 28: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/28.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 28/44
SIMD
• access control - which process accesses arepossible to which resources
• synchronization - constraints limit the timeof accesses from sharing processes toshared resources
![Page 29: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/29.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 29/44
SIMD
• protection - a system feature that preventsprocesses from making arbitrary access toresources belonging to other processes
![Page 30: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/30.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 30/44
MIMD
• nodes are typically able to simultaneously
• store messages in buffers
• perform send/receive operations
• scalable - the number of processors can beincreased without significant decrease in
efficiency of operation
![Page 31: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/31.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 31/44
InterconnectionNetworks
![Page 32: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/32.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 32/44
Interconnection
Networks INs• Can be classified based on
• mode of operation
• control strategy
• switching techniques
• topology
![Page 33: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/33.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 33/44
Mode of Operation
• Accordingly, INs are classified as:
• Synchronous
•a single global clock used by all
• operating in a lock-step manner
• Asynchronous
•does not require a global clock
• handshaking signals are used
• Sync tends to be slower than async, sync is raceand hazard-free, however.
![Page 34: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/34.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 34/44
Control Strategy
• Accordingly, INs are classified as
• Centralized
• a single central CU is used to overseeand control the operation
•Decentralized
• the control function is distributedamong different components
![Page 35: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/35.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 35/44
Control Strategy
• The function and reliability of the central
control unit can become the bottleneck ina centralized control system
• While the crossbar is a centralized system,
the multistage interconnection networksare decentralized
![Page 36: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/36.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 36/44
Switching Techniques
• INs can be classified as:
• circuit switching
• a complete path has to be established and remain
existence during the whole communication
• packet switching
• communication takes place via messages that are dividedinto smaller entities (packets)
• packets travel in a store-and-forward manner
• While packet s/w tends to use resources more efficiently, itsuffers from variable packet delays
![Page 37: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/37.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 37/44
Topology
• Topology describes how to connectprocessors and memories to other
processors and memories
![Page 38: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/38.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 38/44
Shared Memory INs
bus-based switch-based
![Page 39: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/39.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 39/44
Message Passing INs
• Static interconnection network
• Dynamic interconnection network
![Page 40: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/40.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 40/44
Static INs
![Page 41: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/41.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 41/44
Dynamic INs
• Establish a connection between two ormore nodes on the fly as messages are
routed along the links
• The number of hops in a path from sourceto destination node is equal to the number
of point-to-point links a message musttraverse to reach its destination
![Page 42: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/42.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 42/44
Single-stage
![Page 43: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/43.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 43/44
![Page 44: Slide02 Parallel Computers](https://reader031.fdocuments.in/reader031/viewer/2022021213/577d27dc1a28ab4e1ea509c4/html5/thumbnails/44.jpg)
8/6/2019 Slide02 Parallel Computers
http://slidepdf.com/reader/full/slide02-parallel-computers 44/44
Crossbar switch