Evolution and Convergence of Parallel Architectures Todd C. Mowry CS 495 August 28, 2002.
CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 -...
Transcript of CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 -...
![Page 1: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/1.jpg)
0-1CS 755 - Fall 2014
CS 755 – System and Network Architectures and Implementation
Module 0 - Introduction
Martin Karsten
![Page 2: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/2.jpg)
0-2CS 755 - Fall 2014
Distributed System
● Leslie Lamport:
“A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.”
![Page 3: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/3.jpg)
0-3CS 755 - Fall 2014
Distributed System - Description
● A collection of computing components● with non-neglible communication latency
– possibly non-deterministic● with potential partial failures
– communication– components
● … that appears as coherent system
→ distribution transparency
![Page 4: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/4.jpg)
0-4CS 755 - Fall 2014
Scaling Dimensions
● time:cycle, memory, disk, network, human, daily,...
● space:core, chip, board, chassis, building, campus,...
● administrative:uniform vs. cooperation vs. competition
![Page 5: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/5.jpg)
0-5CS 755 - Fall 2014
Tentative Schedule
● today: organization, overview, background
● 5 lectures covering networking basics
● 5 classes with lectures/paper discussion
● final exam
![Page 6: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/6.jpg)
0-6CS 755 - Fall 2014
Basics
● Sep 17: Channels● communication between neighbours
● Sep 24: Network● communication in a network (graph)
● Oct 1: Transport● end-to-end challenges
![Page 7: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/7.jpg)
0-7CS 755 - Fall 2014
Basics
● Oct 8: Remote Services● service beyond basic communication
● Oct 15: Naming and Mobility● labelling and finding communication entities
![Page 8: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/8.jpg)
0-8CS 755 - Fall 2014
Network Model
![Page 9: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/9.jpg)
0-9CS 755 - Fall 2014
Web Services Model
![Page 10: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/10.jpg)
0-10CS 755 - Fall 2014
Advanced
● Oct 22: Ordering and Consistency● difficult: timing in distributed system
● Oct 29: Fault Tolerance● even worse: components might fail
![Page 11: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/11.jpg)
0-11CS 755 - Fall 2014
Advanced
● Nov 5: Storage & Replication● essential distributed service
● Nov 12: Cloud Services● novel paradigm?
● Nov 19: Big Data● important application area
![Page 12: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/12.jpg)
0-12CS 755 - Fall 2014
Important Dates
● before Oct 1:● send me priority list of papers for presentation
(student must be registered in Quest)● FCFS selection
● Oct 17, 5pm: 1st critical review due● Nov 21, 5pm: 2nd critical review due● Nov 26: last class & final exam
![Page 13: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/13.jpg)
0-13CS 755 - Fall 2014
Model of a Computer
● von Neumann model● CPU
● control & data path
● I/O● user, storage, network
● memory
program & data stored in memory
![Page 14: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/14.jpg)
0-14CS 755 - Fall 2014
Technology Trends
● electronics technology continues to evolve● increased capacity
and performance● reduced cost DRAM capacity
Year Technology Relative performance/cost
1951 Vacuum tube 1
1965 Transistor 35
1975 Integrated Circuit 900
1995 Very large scale IC (VLSI) 2,400,000
2005 Ultra large scale IC 6,200,000,000
![Page 15: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/15.jpg)
0-15CS 755 - Fall 2014
Performance:Latency vs. Throughput
● Tim Horton's● time to coffee vs. customers/hour● low latency => high throughput● but not vice versa● faster coffee makers vs. more (and more space)
● latency (response time)● completion time of specific task
● throughput● total work done over time period
![Page 16: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/16.jpg)
0-16CS 755 - Fall 2014
Performance
● reduce latency?● faster processor● better algorithm (software)● more processors (needs parallelization)● generally increases throughput
● increase throughput?● more processors● rearrange system components (scheduling):
often increases latency
![Page 17: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/17.jpg)
0-17CS 755 - Fall 2014
Efficiency Matters
● network-centric computing, Internet
→ large data centers● hardware cheap, but
● power consumption → heat● heat → cooling → more power consumption● money and environment costs
● often:software performance (throughput) ~ efficiency
![Page 18: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/18.jpg)
0-18CS 755 - Fall 2014
Moore's Law
● transistor density doubles every two years● every year 1959-1975
● in the past● transistor density translated into processing power● almost double speed every 2 years...● reduce latency, increase throughput
● recently: memory wall● more recently: power wall
![Page 19: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/19.jpg)
0-19CS 755 - Fall 2014
Memory Wall
![Page 20: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/20.jpg)
0-20CS 755 - Fall 2014
Power Wall
● power = capacitive load x voltage2 x frequency● cannot reduce voltage further (path length)● cannot remove more heat
![Page 21: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/21.jpg)
0-21CS 755 - Fall 2014
Uniprocessor Performance
Constrained by power, instruction-level parallelism, memory latency
![Page 22: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/22.jpg)
0-22CS 755 - Fall 2014
Multiprocessors
● multicore microprocessors● more than one processor per chip
● requires explicitly parallel programming● compare with instruction level parallelism (hidden)
● hard to do● programming for performance● load balancing● optimizing communication and synchronization
![Page 23: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/23.jpg)
0-23CS 755 - Fall 2014
Amdahl's Law
● improve some part of a computer program● or it's execution speed (e.g., through parallelization)
● limits overall performance improvement
unaffectedaffected
improved Tfactor timprovemen
TT
![Page 24: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/24.jpg)
0-24CS 755 - Fall 2014
Amdahl's Law
Source: Wikimedia Commons
![Page 25: CS 755 – System and Network Architectures and ...mkarsten/cs755-F14/introduction.pdf · CS 755 - Fall 2014 0-1 CS 755 – System and Network Architectures and Implementation Module](https://reader033.fdocuments.in/reader033/viewer/2022050102/5f41939d856ab026710a171f/html5/thumbnails/25.jpg)
0-25CS 755 - Fall 2014
Trade-Offs
● almost everything in CS is a trade-off● very few absolute truths
● “fast, good, or cheap – pick two”