Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as...
-
Upload
frank-anthony -
Category
Documents
-
view
226 -
download
0
Transcript of Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as...
![Page 1: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/1.jpg)
Lecture 1 – Introduction
CSE 490h – Introduction to Distributed Computing, Spring 2007
Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License.
![Page 2: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/2.jpg)
Outline Administrivia Scope of Problems A Brief History Parallel vs. Distributed Computing Parallelization and Synchronization Problem Discussion Prelude to MapReduce
![Page 3: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/3.jpg)
Staff Aaron Kimball – Instructor
[email protected] OH – Wed 2:30-3:30
Taj Campbell – TA [email protected] OH – Fri 10:30-11:30
Sierra Michels-Slettvet – TA [email protected] OH – Thurs 1:30-2:30
Adrian Caulfield – lab manager [email protected]
![Page 4: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/4.jpg)
Goals and Expectations
Weekly Lecture Weekly Reading
Short response questions + discussion
Weekly Lab (x4) Designed (mostly) to be done during the two hour lab.
Final Project Proposal (Optional) Final Project
![Page 5: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/5.jpg)
Computer Speedup
Moore’s Law: “The density of transistors on a chip doubles every 18 months, for the same cost” (1965)
Image: Tom’s Hardware
![Page 6: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/6.jpg)
Scope of problems
What can you do with 1 computer? What can you do with 100 computers? What can you do with an entire data
center?
![Page 7: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/7.jpg)
Distributed problems
Rendering multiple frames of high-quality animation
Image: DreamWorks Animation
![Page 8: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/8.jpg)
Distributed problems
Simulating several hundred or thousand characters
Happy Feet © Kingdom Feature Productions; Lord of the Rings © New Line Cinema
![Page 9: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/9.jpg)
Distributed problems
Indexing the web (Google) Simulating an Internet-sized network for
networking experiments (PlanetLab) Speeding up content delivery (Akamai)
What is the key attribute that all these examples have in common?
![Page 10: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/10.jpg)
Parallel vs. Distributed
Parallel computing can mean:Vector processing of data (SIMD)Multiple CPUs in a single computer (MIMD)
Distributed computing is multiple CPUs across many computers (MIMD)
![Page 11: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/11.jpg)
A Brief History… 1975-85
Parallel computing was favored in the early years
Primarily vector-based at first
Gradually more thread-based parallelism was introduced
Cray 2 supercomputer (Wikipedia)
![Page 12: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/12.jpg)
“Massively parallel architectures” start rising in prominence
Message Passing Interface (MPI) and other libraries developed
Bandwidth was a big problem
A Brief History… 1985-95
![Page 13: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/13.jpg)
A Brief History… 1995-Today
Cluster/grid architecture increasingly dominant
Special node machines eschewed in favor of COTS technologies
Web-wide cluster software Companies like Google take this to the
extreme (10,000 node clusters)
![Page 14: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/14.jpg)
Parallelization & Synchronization
![Page 15: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/15.jpg)
Parallelization Idea
Parallelization is “easy” if processing can be cleanly split into n units:
work
w1 w2 w3
Partition problem
![Page 16: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/16.jpg)
Parallelization Idea (2)
w1 w2 w3
thread thread thread
Spawn worker threads:
In a parallel computation, we would like to have as many threads as we have processors. e.g., a four-processor computer would be able to run four threads at the same time.
![Page 17: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/17.jpg)
Parallelization Idea (3)
thread thread thread
Workers process data:
w1 w2 w3
![Page 18: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/18.jpg)
Parallelization Idea (4)
results
Report results
thread thread threadw1 w2 w3
![Page 19: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/19.jpg)
Parallelization Pitfalls
But this model is too simple!
How do we assign work units to worker threads? What if we have more work units than threads? How do we aggregate the results at the end? How do we know all the workers have finished? What if the work cannot be divided into
completely separate tasks?
What is the common theme of all of these problems?
![Page 20: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/20.jpg)
Parallelization Pitfalls (2)
Each of these problems represents a point at which multiple threads must communicate with one another, or access a shared resource.
Golden rule: Any memory that can be used by multiple threads must have an associated synchronization system!
![Page 21: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/21.jpg)
What is Wrong With This?
Thread 1:
void foo() {
x++;
y = x;
}
Thread 2:
void bar() {
y++;
x++;
}
If the initial state is y = 0, x = 6, what happens after these threads finish running?
![Page 22: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/22.jpg)
Multithreaded = Unpredictability
When we run a multithreaded program, we don’t know what order threads run in, nor do we know when they will interrupt one another.
Thread 1:
void foo() {
eax = mem[x];
inc eax;
mem[x] = eax;
ebx = mem[x];
mem[y] = ebx;
}
Thread 2:
void bar() {
eax = mem[y];
inc eax;
mem[y] = eax;
eax = mem[x];
inc eax;
mem[x] = eax;
}
Many things that look like “one step” operations actually take several steps under the hood:
![Page 23: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/23.jpg)
Multithreaded = Unpredictability
This applies to more than just integers:
Pulling work units from a queue Reporting work back to master unit Telling another thread that it can begin the
“next phase” of processing
… All require synchronization!
![Page 24: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/24.jpg)
Synchronization Primitives
A synchronization primitive is a special shared variable that guarantees that it can only be accessed atomically.
Hardware support guarantees that operations on synchronization primitives only ever take one step
![Page 25: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/25.jpg)
Semaphores
A semaphore is a flag that can be raised or lowered in one step
Semaphores were flags that railroad engineers would use when entering a shared track
Set: Reset:
Only one side of the semaphore can ever be red! (Can both be green?)
![Page 26: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/26.jpg)
Semaphores
set() and reset() can be thought of as lock() and unlock()
Calls to lock() when the semaphore is already locked cause the thread to block.
Pitfalls: Must “bind” semaphores to particular objects; must remember to unlock correctly
![Page 27: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/27.jpg)
The “corrected” exampleThread 1:
void foo() {
sem.lock();
x++;
y = x;
sem.unlock();
}
Thread 2:
void bar() {
sem.lock();
y++;
x++;
sem.unlock();
}
Global var “Semaphore sem = new Semaphore();” guards access to x & y
![Page 28: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/28.jpg)
Condition Variables
A condition variable notifies threads that a particular condition has been met
Inform another thread that a queue now contains elements to pull from (or that it’s empty – request more elements!)
Pitfall: What if nobody’s listening?
![Page 29: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/29.jpg)
The final exampleThread 1:
void foo() {
sem.lock();
x++;
y = x;
fooDone = true;
sem.unlock();
fooFinishedCV.notify();
}
Thread 2:
void bar() {
sem.lock();
if(!fooDone) fooFinishedCV.wait(sem);
y++;
x++;
sem.unlock();
}
Global vars: Semaphore sem = new Semaphore(); ConditionVar fooFinishedCV = new ConditionVar(); boolean fooDone = false;
![Page 30: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/30.jpg)
Barriers
A barrier knows in advance how many threads it should wait for. Threads “register” with the barrier when they reach it, and fall asleep.
Barrier wakes up all registered threads when total count is correct
Pitfall: What happens if a thread takes a long time?
![Page 31: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/31.jpg)
Too Much Synchronization? Deadlock
Synchronization becomes even more complicated when multiple locks can be used
Can cause entire system to “get stuck”
Thread A:Thread A:semaphore1.lock();semaphore2.lock();/* use data guarded by semaphores */semaphore1.unlock(); semaphore2.unlock();
Thread B:semaphore2.lock();semaphore1.lock();/* use data guarded by semaphores */semaphore1.unlock(); semaphore2.unlock();
(Image: RPI CSCI.4210 Operating Systems notes)
![Page 32: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/32.jpg)
The Moral: Be Careful! Synchronization is hard
Need to consider all possible shared stateMust keep locks organized and use them
consistently and correctly Knowing there are bugs may be tricky; fixing
them can be even worse! Keeping shared state to a minimum reduces
total system complexity
![Page 33: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/33.jpg)
Activity: Parallelizing Raytracer
![Page 34: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/34.jpg)
Prelude to MapReduce
We saw earlier that explicit parallelism/synchronization is hard
Synchronization does not even answer questions specific to distributed computing, like how to move data from one machine to another
Fortunately, MapReduce handles this for us
![Page 35: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/35.jpg)
Prelude to MapReduce
MapReduce is a paradigm designed by Google for making a subset (albeit a large one) of distributed problems easier to code
Automates data distribution & result aggregation
Restricts the ways data can interact to eliminate locks (no shared state = no locks!)
![Page 36: Lecture 1 – Introduction CSE 490h – Introduction to Distributed Computing, Spring 2007 Except as otherwise noted, the content of this presentation is licensed.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649f515503460f94c75206/html5/thumbnails/36.jpg)
Next time…
We will go over MapReduce in detail Discuss the functional programming
paradigms of fold and map, and how they relate to distributed computation