History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945)...
Transcript of History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945)...
![Page 1: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/1.jpg)
History
Jurassic Period (-1945)
• Astronomical/Geometric tools
• Schickard, Leibniz, Pascal (17th Cen.)
• Babbage: Analytic engine (1842) – first “programmable” machine.
• Scheutz (1853)
• Hollerith (1911) – IBM (1924)
• Mechanic Tools (Slide Rule, Marchand Calculator)
• Parallel Processing in room
JAH 1
![Page 2: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/2.jpg)
Early Electronics (1937-53)
Often electromechanical
• Atanasoff (1937): PDE machine
• Zuse (1938): Z1 (binary system, high level language)
• Turing (1943): Colossus
• Eckert,Mauchly (1943): ENIAC
• Eckert,Mauchly,von Neumann : EDVAC (program memory)
JAH 2
![Page 3: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/3.jpg)
Second Generation (1954-62)
• First commercial machines
• Magnetic cores
• Register
• Languages: FORTRAN (1956), ALGOL (1958), COBOL (1959)
JAH 3
![Page 4: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/4.jpg)
Third Generation (1963-72)
Transition to semiconductor technology
• Cray (1964): CDC6600 – parallelism, 1MFlops
• IBM 360
• Illiac
• early UNIX
JAH 4
![Page 5: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/5.jpg)
Fourth Generation (1973-84)
• C language, Unix
• Vector Processors
• Functional Programming, Prolog
• Rise of PC
• Lax report (1982): National supercomputer centers
JAH 5
![Page 6: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/6.jpg)
Fifth Generation (1985-9x)
• Massive parallel machines
• Distributed memory architecture
• Workstations
• RISC technology
JAH 6
![Page 7: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/7.jpg)
Development
Source: Meuer/Strohmeier: Scientific Computing World
JAH 7
![Page 8: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/8.jpg)
Now
• Push for higher performance (ASCI)
• Web, Libraries, Linux
• Commodification
• Cluster (Network Of Workstations)
JAH 8
![Page 9: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/9.jpg)
Processor Type
Source: www.top500.org
JAH 9
![Page 10: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/10.jpg)
Chip Technology
Source: www.top500.org
JAH 10
![Page 11: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/11.jpg)
Architecture
Source: www.top500.org
JAH 11
![Page 12: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/12.jpg)
Parallelization
There are various (obvious) obstacles to have a single processor runfaster.
If single-processor performance does not suffice, the only waytherefore is to use several processors at once, that is to parallelize thecalculation.
• One person takes a workday to carry 5000 bricks from A to B .
• 8 persons take one hour.
• 5000 persons take a minute.
• 300000 persons take 1 second ?• A flight from Denver to London takes 10 hours. How long does it
take using 2 planes in parallel?
JAH 12
![Page 13: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/13.jpg)
Embarrasing parallelization
If I have to solve two different problems, I can solve them in parallel ontwo computers.
Some calculations can be parallelized in almost the same way: Thereare (comparatively) long calculations that do not interact, the resultsare collected and used afterwards.
Such a parallelization is easy and gives almost linear speedup.
Problems arise if different processes have to communicate or have toshare resources.
JAH 13
![Page 14: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/14.jpg)
Example
Sit in a square arrangement. Take a piece of paper and write on it a(random) positive integer between 1 and 100.
Now we perform the following process:
• Take the numbers of your (horizontal, vertical) neighbours (assumecyclic arrangement),
• average them (round to the nearest integer), and
• replace your number with this number.
• Iterate
What is the result?
JAH 14
![Page 15: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/15.jpg)
Problems
• Communication
• When is new data available?
• Idling.
• When is the process done/ What to do with result?
JAH 15
![Page 16: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/16.jpg)
Obstacle: Dependency
If task A depends on the result of task B, they cannot run in parallel.
(There are fancy methods in project managment (PERT, C/PM) thatcan be used to identify such situations.)
For such tasks, parallelization gives no speedup.
Similarly it could happen that one process is idle while waiting forother processes. Proper load balancing tries to resolve this.
JAH 16
![Page 17: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/17.jpg)
Obstacle: Deadlock
12 philosophers sit on a table, eating long spaghetti.
One needs 2 forks to eat, there is one fork between 2 plates. If 1 iseating, 2 cannot.
Suppose 1 and 3 are eating alternatively, 2 is permanently blocked.
JAH 17
![Page 18: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/18.jpg)
If everyone takes up his left fork, the system is deadlocked. (The samecan happen if different processes require the same resource.)
Remedy:
Coordinated Use signals (semaphores) to indicate the desire to eat.Then dispatch fork selection according to semaphore queue.
Uncoordinated If you are blocked, wait a random amount of time(based on maximal task time) and try again. (Used in ethernetprotocol.)
Typically this is resolved on a low level in hardware or thecommunication protocol.
JAH 18
![Page 19: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/19.jpg)
Obstacle: Communication
If two processes exchange data, this data has to be transferred fromthe processor executing process A to the processor executing processB:
A −→ B
If the processors have a direct communication link, this is easy.
If not, the data has to be transmitted via intermediate processors:
A −→ C1 −→ C2 −→ · · · −→ C3 −→ B
This takes processing power away from the processors Ci. In the worstcase, the system ends up spending almost all time in communication.
One way to remedy it is to give many different communication links(Hypercube architecture). However fast links are expensive.
JAH 19
![Page 20: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/20.jpg)
Fortunately many “real world” calculations (for example weatherforecast) require on the lowest level only communication between“adjacent cells”. This limits communication requirements.
Common Network Another way to give total communication is toconnect all processes to a common network. The problem then ispotential deadlock.
Again, one can spend almost unlimited amounts of money to get“better” network connection (for example: fast switches).
JAH 20
![Page 21: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/21.jpg)
Obstacle: Bootstrap/Crash Recovery
When running programs in parallel we have to start them all. Typicallythis has to be done indirectly.
The processes have to initialize properly and have to initiatecommunication.
Similarly, one has to deal with errors (or crashes) in single processes.Possible reqctions are:
• Terminate all calculations.
• Move load to other process.
• Restart dead process.
JAH 21
![Page 22: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/22.jpg)
Amdahl’s Law
Not all parts of a program can be parallelized, there is always a serial“residue”. Suppose this residue is a ratio of b of the total calculation.
Then the speedup of a program using n processors (over a singleprocessor) is
S =n
bn + (1− b)
For a constant nonzero b this yields a curve that grows substantiallyslower than n, in fact it has limit 1
bfor n →∞.
For example for b = 1/20 we get:
JAH 22
![Page 23: History Jurassic Period (-1945)hulpke/lectures/gs510/sli10.pdf · History Jurassic Period (-1945) •Astronomical/Geometric tools •Schickard, Leibniz, Pascal (17th Cen.) •Babbage:](https://reader035.fdocuments.in/reader035/viewer/2022071016/5fcea8b595b06264da37bc3c/html5/thumbnails/23.jpg)
For problems with nonparallelizable part this can substantially limit thegain due to parallelization.
Note that initialization and communication always staynonparallelizable.
JAH 23