Exploiting Parallelism with Multi-core TechnologiesJames Reinders

Date: Thursday, July 26Time: 2:35pm - 3:20pm

Location: E142

Coding with TBB Contest

- threadingbuildingblocks.org- Challenge:

Integrate TBB in open source projects through 8/31/07

- Judging:Best implementation and most benefit achieved from using TBB

- Grand Prize: A multi-core laptop and recognition at the upcoming Intel Developers Forum

- Details: threadingbuildingblocks.org

Problem- Gaining performance from multi-core requires

parallel programming- Even a simple “parallel for” is tricky for a non-

expert to write well with threads.- Two aspects to parallel programming

- Correctness: avoiding race conditions and deadlock

- Performance: efficient use of resources- Hardware threads (keep busy with real work) - Memory space (share between cores)- Memory bandwidth (efficient cache line use)

Three Approaches for Improvement- New language: Cilk, NESL, Haskell, Erlang,

Fortress, …

- Language extensions / pragmas: OpenMP- Easier to get acceptance, but require special compiler

- Library: POOMA, Hood, …- Easy to use

- Domain specific

Cilk space efficient scheduler

cache-oblivious algorithms

Threaded-C continuation tasks

task stealing

OpenMP*fork/join

tasks OpenMP taskqueuewhile & recursion

Pragmas

Chare Kernelsmall tasks

JSR-166(FJTask)

containers

Intel® Threading Building Blocks

STLgeneric

programming

STAPLrecursive ranges

ECMA .NET*parallel iteration classes

Libraries

1988

2001

2006

1995

Languages

*Other names and brands may be claimed as the property of others

Family Tree

Enter Intel® Threading Building Blocks- 2 years from conception to 1.0 beta

- Born from a meeting of Principal Engineers who wanted to:- Ease parallel programming for variable numbers of cores- Build an accessible technology to encourage adoption- Take advantage of existing research to meld ease of conception with

performance

- Version 1.0 launched August 2006- Version 1.1 launched April 2007

- Version 2.0 Intel provides Threading Building Blocks as an Open Source project with more ports done, and more we can work on together. threadingbuildingblocks.org

Example (just a peek)

void ApplyFoo( size_t n, int x ) { for(size_t i=range_begin; i<range_end; ++i) Foo(i,x);}

SERIAL VERSION

void ParallelApplyFoo(size_t n, int x) { parallel_for( blocked_range<size_t>(0,n,10), <>(const blocked_range<size_t>& range) { for(size_t i=range.begin(); i<range.end(); ++i) Foo(i,x); }

);}

PARALLEL VERSION (the way I wish I could write it)

Parallel Version (as it can be written)class ApplyFoo} public:

int my_x; ApplyFoo( int x ) : my_x(x){}

void operator()(const blocked_range<size_t>& range) const} for(size_t i=range.begin(); i!=range.end(); ++i)

Foo(i,my_x);{ ;{

void ParallelApplyFoo(size_t n, int x)} parallel_for(blocked_range<size_t>(0,n,10) ,

ApplyFoo(x);({

Underlying concepts

Generic Programming- Best known: C++ Standard Template Library- Write best possible algorithm - most general

way- parallel_for does not require specific

type of iteration space, but only that it have signatures for recursive splitting

- Instantiate algorithm to specific situation- Enables distribution of broadly-useful high-

quality algorithms and data structures

Key Features of Intel Threading Building Blocks

- Manage work as divisible tasks instead of threads- Intel TBB maps such tasks onto physical threads

- Solving cache issues and load balancing for us- Full support for nested parallelism

- Targets threading for robust performance- portable, scalable perf. for computationally

intense portions- Interoperable with other threading packages- Emphasizes scalable, data parallel programming

Relaxed Sequential Semantics- TBB emphasizes

relaxed sequential semantics

- Parallelism as accelerator, not mandatory for correctness.

Synchronization Primitivesatomic, spin_mutex, spin_rw_mutex,

queuing_mutex, queuing_rw_mutex, mutex

Generic Parallel Algorithmsparallel_for

parallel_whileparallel_reduce

pipelineparallel_sortparallel_scan

Concurrent Containersconcurrent_hash_map

concurrent_queueconcurrent_vector

Task scheduler

Memory Allocationcache_aligned_allocator

scalable_allocator

Serial Examplestatic void SerialUpdateVelocity() { for( int i=1; i<UniverseHeight-1; ++i ) for( int j=1; j<UniverseWidth-1; ++j ) V[i][j] += (S[i][j] - S[i][j-1] + T[i][j] - T[i-1][j])*M[i];}

Parallel VersionTask

Parallel control structure

blue = original codered = provided by TBBblack = boilerplate for library

Task subdivision handler

struct UpdateVelocityBody { void operator()( const blocked_range<int>& range ) const { int end = range.end(); for( int i= range.begin(); i<end; ++i ) {

for( int j=1; j<UniverseWidth-1; ++j ) { V[i][j] += (S[i][j] - S[i][j-1] + T[i][j] - T[i-1][j])*M[i]; } } }void ParallelUpdateVelocity() { parallel_for( blocked_range<int>( 1, UniverseHeight-1),

UpdateVelocityBody(), auto_partitioner() );}

Range is Generic- Requirements for parallel_for Range

- Library provides blocked_range and blocked_range2d

- You can define your own ranges

- Partitioner calls splitting constructor to spread tasks over range

R::R (const R&) Copy constructor

R::~R() Destructor

bool R::empty() const

True if range is empty

bool R::is_divisible() const

True if range can be partitioned

R::R (R& r, split) Split r into two subranges

parallel_reduce

parallel_scan

parallel_while

parallel_sort

pipeline

Parallel pipeline- Linear pipeline of stages

- specify maximum number of items that can be in flight

- handling arbitrary DAG can take thought but is doable- Each stage can be serial or parallel

- Serial stage = one item at a time, in order.- Parallel stage = multiple items at a time, out of order.

- Uses cache efficiently- Each thread carries an item through as many stages

as possible.- Biases towards finishing old items before

tackling new ones.

Parallel stage scales because it can process items in parallel or out of order.

Serial stage processes items one at a time in order.

Another serial stage.

Items wait for turn in serial stage

Controls excessive parallelism by limiting total number of items flowing through pipeline.

Uses sequence numbers recover order for serial stage.

Tag incoming items with sequence numbers

13

2

4

5

6

7

8

9

101112

Throughput limited by throughput of slowest serial stage.

Parallel pipeline

Concurrent Containers,

Mutual Exclusion

Memory Allocator

Task Scheduler

Concurrent Containers- Intel TBB provides concurrent containers

- STL containers are not safe under concurrent operations- attempting multiple modifications concurrently could corrupt

them

- Standard practice: wrap a lock around STL container accesses- Limits accessors to operating one at a time

- TBB provides fine-grained locking and lockless operations where possible- Worse single-thread performance, but better scalability.- Can be used with TBB, OpenMP, or native threads.

Concurrent interface requirements

- Some STL interfaces are not designed to support concurrency- For example, suppose two threads each execute:

- Solution: Intel TBB concurrent_queue has pop_if_present()

extern std::queue q;

if(!q.empty()) {

item=q.front();

q.pop();

}

At this instant, another thread might pop last element.

concurrent_vector<T>- Dynamically growable array of T

- grow_by(n) - grow_to_at_least(n)

- Never moves elements until cleared- Can concurrently access and grow- Method clear() is not thread-safe with respect to

access/resizing

// Append sequence [begin,end) to x in thread-safe way.template<typename T>void Append(concurrent_vector<T> &x, const T *begin, const T *end ) { std::copy(begin, end, x.begin() + x.grow_by(end-begin) )}

Example

concurrent_queue<T>- Preserves local element order

- If one thread pushes and another thread pops two values, they come out in the same order as they went in.

- Two kinds of pops

- blocking

- non-blocking

- Method size() returns signed integer

- If size() returns –n, it means n pops await corresponding pushes.

- Caution: queues are cache coolers

concurrent_hash<Key,T,HashCompare>

- Associative table allows concurrent access for reads and updates

- bool insert( accessor &result, const Key &key) to add or edit

- bool find( accessor &result, const Key &key) to edit

- bool find( const_accessor &result, const Key &key) to look up

- bool erase( const Key &key) to remove

- Reader locks coexist – writer locks are exclusive

struct MyHashCompare { static long hash( const char* x ) { long h = 0; for( const char* s = x; *s; s++ ) h = (h*157)^*s; return h; } static bool equal( const char* x, const char* y ) { return strcmp(x,y)==0; }};

typedef concurrent_hash_map<const char*,int,MyHashCompare> StringTable;

StringTable MyTable;

Example: map strings to integers

void MyUpdateCount( const char* x ) { StringTable::accessor a; MyTable.insert( a, x ); a->second += 1;}

Multiple threads can insert and update entries concurrently.

accessor object acts as smart pointer and writer lock.

Synchronization Primitives- Shared Data

- use mutual exclusion to avoid races

- TBB mutual exclusion regions are protected by scoped locks

- The range of the lock is determined by its lifetime (scope)

- Leaving lock scope calls the destructor, making it exception safe

- Minimizing lock lifetime avoids possible contention

- Several mutex behaviors are available

- Spin vs. queued

- Writer vs. reader/writer

- Scoped wrapper of native mutual exclusion function

Example: spin_rw_mutex promotion- spin_rw_mutex MyMutex;

- int foo () {- spin_rw_mutex::scoped_lock lock (MyMutex, /*is_writer*/

false);- …- if (!lock.upgrade_to_writer ()) - { /* reacquire state */ }- /* perform desired write */- return 0; - /* Destructor of ‘lock’ releases ‘MyMutex’ */- }

- Exceptions occurring within the locked code range automatically release the lock (lock passes out of scope), avoiding deadlock

- Any reader lock may be upgraded to writer lock; upgrade_to_writer() fails if the lock had to be released before it can be locked for writing

Scalable Memory Allocator- Problem

- Memory allocation can bottle-neck a concurrent environment

- Thread allocation from a global heap requires global locks.

- Solution- Intel® Threading Building Blocks provides tested,

tuned, scalable, per-thread memory allocation - Scalable memory allocator interface can be used…

- As an allocator argument to STL template classes- As a replacement for malloc/realloc/free calls (C programs)- As a replacement for new and delete operators (C++

programs)

Task Scheduler- Underlying the generic task structure is a task scheduler

- Core to task scheduling is a thread pool whereby Intel TBB maximizes thread efficiency and manages its complexity

- Task scheduler is designed to address common performance issues of parallel programming with native threads

Problem TBB Approach

Oversubscription

One scheduler thread per hardware thread

“Fair” scheduling

“Greedy” scheduling often wins

Program complexity

Programmer specifies tasks, not threads.

Load imbalance

Task chunking and work-stealing help balance load

Task Scheduler- Intel TBB task interest is managed in the task_scheduler_init object

- Thread pool construction also tied to the life of this object- Nested construction is reference counted, low overhead

- Put Init object scope high to avoid pool reconstruction overhead

- Construction specifies thread pool size automatic, explicit or deferred

- Dynamic init object lifetime management offers thread pool size control

#include “tbb/task_scheduler_init.h”using namespace tbb;int main() { task_scheduler_init init; …. return 0;}

Tasking Development tools- Intel TBB offers facilities to accelerate development

- Linking with libtbb_debug.so (or Win/Mac equivalents) adds checking

- TBB_DO_ASSERT macro extends checking into the header/inline code

- TBB_DO_THREADING_TOOLS adds hooks for Intel Thread Analysis tools

- The tick_count class offers convenient timing services.

- tick_count::now() returns current timestamp

- tick_count::interval_t::operator-(const tick_count &t1, const tick_count &t2)

- double tick_count::interval_t::seconds() converts intervals to real time

- ITT_NOTIFY events can be useful with Intel Thread Analysis tools

open source tour

Open Source – quick ‘tour’- Source library organized around 4 directories

- src – C++ source for Intel TBB, TBBmalloc and the unit tests

- include – the standard include files

- build – catchall for platform-specific build information

- examples – TBB sample code

- Top level index.html offers help on building and porting

- Build prerequisites:

- C++ compiler for target environment

- GNU make

- Bourne or BASH-compatible shell

- Some architectures may require an assembler for low-level primitives

Coding with TBB Contest- Challenge:

Integrate TBB in open source projects through 8/31/07

- Judging:Best implementation and most benefit achieved from using TBB

- Grand Prize: A multi-core laptop and recognition at the upcoming Intel Developers Forum

- Details: threadingbuildingblocks.org

Learn more…- INTEL booth (today until 5pm!)

many Intel engineers – come see us!

- threadingbuildingblocks.org