Rajiv Gupta
Chen Tian, Min Feng, Vijay Nagarajan
Speculative Parallelization of Applications on Multicores
2
Goal: Exploit parallelism frequently observed --- not guaranteed to be present due the presence of dependences:
Dependences due to Cold Code
Dependences that are Harmless
Dependences that are Silent
Speculative Parallelization
3
Thread-based Execution ModelNon-Speculative thread and state
Speculative threads and state
Committing Results
Rollback-free Recovery
Software Only – Coarse-grained parallelismSpeculative parallelization of loops
Speedups on a real machine
Outline
4
Main Thread Performs Non-speculative Computation Non-Parallelizable Code Parts of Parallelized Code
Controls Parallel Threads Initialization & Memory Allocation Termination & Miss-speculation Checks Commit Results In-Order
Multiple Parallel ThreadsPerform Speculative Computations E.g., Speculative Loop Bodies
Execution Model
5
prologuespeculative bodyepilogue
Static code Sequential Execution
p1
e1
sp1
p2
e2
sp2
p3
e3
sp3
p4
sp4
e4
Execution Model
6
prologuespeculative bodyepilogue
Static code Sequential Execution
p1
e1
sp1
p2
e2
sp2
p3
e3
sp3
p4
sp4
e4
Parallel Execution
Main thread P1 P2
1. In-order commit.2. At any time, only two threads are executing. So the main thread doesn’t require a separate core.
Execution Model
7
Non-Speculative State (D space)Maintained by the main thread.
Speculative State (P space)Allocated by the main thread and used by parallel threads.
Results will be either committed to D space or discarded. Coordinating State (C space)
Version number for variables in D space.
Mapping table for variables in P space.
Main ThreadD space
D space
C spaceC space
C spaceC space
Parallel ThreadP spaceP space
P spaceP spaceParallel Thread
……
Memory State
8
Naïve schemeCopy-in: copy values from D space to P space when work assigned.
Copy-out: copy variable values from P space to D space when the speculation check succeeds.
Optimized schemeUse profiling to discover access pattern of variables in the speculative loop body. In-Out, Only-in, Only-out and Thread-local.
Unknown variables untouched in the profiling run. Copied on-the-fly through message passing.
Mapping tableMapping information of those variables. D space address, P space address, size, version and write-flag.
Updated when variables are copied into P space.
Referred to when variables are copied back to D space.
Copy Operations
9
Version number – maintained by the main threadFor each variable that is potentially read/written by parallel threads.
Version number is copied into mapping table when the corresponding variable is being copied into P space.
Miss-speculation check – performed by the main threadFor every entry in the mapping table, compare its version number with the one maintained by the main thread.
If all are same, the speculation succeeds. Perform the copy-out operations. Update the version numbers accordingly.
If any version number is different, the speculation fails because some earlier thread has changed this variable’s value: Re-execute the speculative body with the latest value. Value-based dependence check.
Rollback Free Recovery
Miss-speculation Check
10
Access Checks to consult the mapping table at:Loads and Stores
Pointer Assignments
Reducing Overhead of Access Checks
Stack & Global variables: Based upon classification. In-Out, Only-in, Only-out, Thread-local.
Heap: Optimizations beyond classification.
Locally created objects require no checks. Once object is copied, other fields accessed without checking. Copy-on-write only: No checks needed at Loads;
Since version number not copied on a read, miss-speculation detection implicitly carried out by another copied variable.
On-the-fly Copying
11
Reducing Thread IdlingScenario: an earlier thread finishes its task, but the main thread has not finished assigning tasks to later threads, and hence cannot handle this earlier thread. Performance fell when 4 or more parallel threads are used.
Solution: assign more work to each thread by loop unrolling.
Reducing Miss-speculation RateScenario: the value of a speculative variable being used by a thread is changed by an earlier thread, and hence the speculation fails. For benchmark 181.mcf, the miss-speculation rate becomes higher when
more threads are used.
Solution: delay copying of some variables - on-the-fly mechanism. Increases the chance of getting the latest version.
Other Enhancements
12
PrologueInput statements (e.g. fgets).
Loop counters.
EpilogueOutput statements (e.g. printf)
Statements highly-dependent on previous iteration. E.g. line_handled
while (){
line = read_one_line(input_file);
if (line cannot be parsed) {
error_num++;
}
else {
result = parse(line);
}
line_handled ++;
print(result);
}
An example from 197.parse
Speculative bodyThe remainder. Loop carried dependence on error_num rarely manifests itself.
Speculative Parallelization
13
while (){
Prologue code;
Speculative body code;
Epilogue code
}
Assign a new iteration
for (i=0; i < Num_Proc; i++) {
allocate P and C space for thread i;
Prologue code;
create thread i to execute thrd_func (i);
} reset i = 0;
while() {
while (speculation_check(i) == FAIL) {
update P and C space for thread i;
re-execute thrd_func (i);
}
commit result and execute Epilogue code;
Prologue code;
update P and C space for thread i;
ask thread i to execute thrd_func (i);
i= (i+1) % Num_Proc;
}
wait for all threads’ completion and execute Epilogue code;
Create thread and initialize their tasks
Handle misspeculation
In-order commit
Main Thread
14
while (){
Prologue code;
Speculative body code;
Epilogue code
}
void * thrd_func(i) {
while (1) {
wait for the “start” message;
Speculative body code;
send “finish” message;
}
}
Parallel Thread
Checks preceding/following Loads, Stores, Pointer Assignments
15
Profiling tool(Pin)
Dependence graph and access patterns
Compiler infrastructure(LLVM)
Binary and a small input
Source code
objdump SymbolsTransformation
Template
x86 binary
-native option
Dell PowerEdge 1900Two Intel Xeon Quadcores
3 GHz, 16 GB
Experimental Setup
16
Benchmarks5 SPEC benchmarks 197.parser, 181.mcf, 130.li, 256.bzip2 & 255.vortex.
1 MiBench benchmark CRC32- Best speedup achieved among all benchmarks.
Variables in speculative body (obtained via profiling)
Programs Only-In Only-Out Thread Local In-Out
197.parser 49 6 12 2
181.mcf 5 0 5 3
130.li 30 0 3 6
256.bzip2 12 8 11 1
255.vortex 76 5 4 6
CRC32 1 0 2 1
Dell PowerEdge 1900 server with two quad-core processors, 3GHz, &16 GB.
Experimental Setup
17
All benchmarks get the best speedup when 8 threads are used.
The highest speedups ranges from 3.7 to 7.8 across all benchmarks.
Execution Speedups
18
Thread Idling
19
Without delayed copying, miss-speculation rate of 181.mcf increases from 0.7% to 17.5% as the number of parallel thread increases from 2 to 8.
With delayed copying, miss-speculation rate of 181.mcf is below 10%.
The miss-speculation rate of other benchmarks is less than 2%.
Delayed Copying
Threads
20
Considered three schemes:1. All: all variables copied before parallel thread starts work
Unnecessary copying occurs.
2. On-the-fly: all variables copied on-the-fly via message passing Need to check every variable to see if it has been copied into P space
3. Opt. : profiling used to determine when to copy
Copy Optimization
21
The experiment shows the result when 4 threads are used.
Opt. outperforms other two schemes.
On-the-fly outperforms all when heap accesses dominate (bzip2, mcf).
Copy Optimization
22
Overhead breakdown per core when 8 threads are used.
No more than 7% of total instructions are used for operations related to the execution model.
Programs Copy
On Start
Copy
On-the-fly
Exception
Check
Miss-speculation
Check
Setup
197.parser 3.51% 0.33% 0.02% 1.76% 0.62%
181.mcf 0.08% 0% 0% 1.08% 0.07%
130.li 1.32% 0.25% 0.06% 1.03% 0.48%
256.bzip2 1.97% 0.13% 0.08% 2.81% 2.15%
255.vortex 5.28% 0.04% 0.01% 1.25% 0.39%
CRC32 0.02% 0% 0% 0.01% 0.32%
Overhead – Instruction Count
23
For most benchmarks, the space overhead is around 2-3x.
256.bzip2 - a large chunk of heap needs to be copied to P space.
Overhead – Memory Space
Top Related