Intro to Parallel Programming

8/8/2019 Intro to Parallel Programming

1/47


2/47

Why Parallelism, Why Now? Parallelism w/ Visual Studio 2010 & .NET 4


3/47



4/47


5/47

[A]fter decades of single core processors, the high volumeprocessor industry has gone from single to dual to quad-core in just the last two years. Moores Law scaling shouldeasily let us hit the 80-core mark in mainstream processorswithin the next ten years and quite possibly even less.

-- Justin Rattner, CTO, Intel (February 2007)

If you havent done so already, now is the time to take a

hard look at the design of your application, determinewhat operations are CPU-sensitive now or are likely tobecome so soon, and identify how those places couldbenefit from concurrency.

-- Herb Sutter, C++ Architect at Microsoft (March 2005)
http://blogs.zdnet.com/OverTheHorizon/?p=13&tag=col1;post-13http://www.gotw.ca/publications/concurrency-ddj.htmhttp://www.gotw.ca/publications/concurrency-ddj.htmhttp://blogs.zdnet.com/OverTheHorizon/?p=13&tag=col1;post-13


6/47


7/47

Make it easier to express andmanage the correctness,

efficiency andmaintainability of parallelism

on Microsoft platforms fordevelopers of all skill levels

Enable developers toexpress parallelism

easily and focus onthe problem to be

solvedImprove the

efficiency andscalability of parallel

applications

Simplify the process

of designing andtesting parallelapplications


8/47



9/47

IEnumerable RaceCarDriver

var new List RaceCarDriver

foreach var in

if


10/47

IEnumerable RaceCarDriver

var new List RaceCarDriver

int Environment

int

var

try

using var new ManualResetEvent

for int

ThreadPool delegate

while true

RaceCarDriver

lock

if break

if

lock

if Interlocked ref

finally if is IDisposable IDisposable


11/47

var from in

where

orderby ascending

select

.AsParallel()

P


12/47

http://msdn.com/concurrency
http://msdn.microsoft.com/concurrencyhttp://msdn.microsoft.com/concurrency


13/47

Tools, Programming Models, Runtimes

ParallelPatternLibrary

Resource Manager

Task Scheduler

Task ParallelLibrary

Parallel LINQ

Managed NativeKey:

ThreadsOperatingSystem

Concurrency Runtime

Programming Models

ThreadPool

Task Scheduler

Resource Manager

DataStructures

DataStructures

Tools

Tooling

ParallelDebugger

ToolWindows

ProfilerConcurrency

Analysis

AgentsLibrary

UMS Threads

.NET Framework 4 Visual C++ 10Visual

Studio

IDE

Windows


14/47



15/47


16/47



17/47

Loops

for (int i = 0; i < N; i++)

{

work(i);

}

foreach (T e in data) work(e);

Parallel.For(0, N, i =>

{

work(i);

});

Parallel.ForEach(data, e => work(e));


18/47



19/47

Regions

A();

B();

C();

Parallel.Invoke(

() => A(),

() => B(),

() => C());


20/47

Regions

A();

B();

C();

var options = new ParallelOptions()

{

MaxDegreeOfParallelism = 2,

CancellationToken = myToken,

TaskScheduler = myScheduler

};

Parallel.Invoke(options,

() => A(),

() => B(),

() => C());


21/47

Tasks Concepts

Task

An asynchronous operation

Task

A Task that returns a result

Continuation

A Task that starts when oneor more other taskscomplete

FromAsync

A Task that wraps an APMimplementation

TaskCompletionSource

A Task you control manually

TaskScheduler

An extensible schedulinginfrastructure


22/47

Tasks Examples

TaskCompletionSource tcs = new TaskCompletionSource();

Task t2 = tcs.Task; // later may call tcs.{Try}Set*

Task t0 = Task.Factory.StartNew(() =>{

});

CancellationTokenSource cts = new CancellationTokenSource();

Task t3 = t2.ContinueWith(t => { }, cts.Token);

Task.Factory.ContinueWhenAny(new[] {t0, t3}, _ => { });

Task.WaitAll(t0, t3);

Task t1 = Task.Factory.FromAsync(

obj.BeginGetString, obj.EndGetString, ...,

TaskCreationOptions.AttachedToParent);

...Task t1 = Task.Factory.FromAsync(

obj.BeginGetString, obj.EndGetString, ...);


23/47



24/47


25/47

Parallel Tasks


26/47

Zoomcontrol Birds eye view

Parallel Stacks


27/47



28/47

LINQ enabled data sources

LINQTo Objects

Objects

LINQTo XML

XML

LINQ-enabled ADO.NET

LINQTo Datasets

LINQTo SQL

LINQTo Entities

Relational

OthersVisual Basic C#

.NET Standard Query Operators


29/47

var q = from i in arrayOfIntegers

where i % 2 == 0

select i;

var q = from i in arrayOfIntegers.AsParallel()

where i % 2 == 0

select i;


30/47

Aggregate(3)All(1)Any(2)AsEnumerable(1)

Average(20)Cast(1)Concat(1)Contains(2)Count(2)DefaultIfEmpty(2)Distinct(2)

ElementAt(1)ElementAtOrDefault(1)Empty(1)Except(2)First(2)FirstOrDefault(2)

GroupBy(8)GroupJoin(2)Intersect(2)Join(2)

Last(2)LastOrDefault(2)LongCount(2)Max(22)Min(22)OfType(1)OrderBy(2)

OrderByDescending(2)Range(1)Repeat(1)Reverse(1)Select(2)SelectMany(4)

SequenceEqual(2)Single(2)SingleOrDefault(2)Skip(1)

SkipWhile(2)Sum(20)Take(1)TakeWhile(2)ThenBy(2)ThenByDescending(2)ToArray(1)

ToDictionary(4)ToList(1)ToLookup(4)Union(2)Where(2)Zip(1)

In .NET 4, ~50 operators w/ ~175 overloads


31/47



32/47

Thread-safe, scalable collections IProducerConsumerCollection

ConcurrentQueue

ConcurrentStack

ConcurrentBag

ConcurrentDictionary

Phases and work exchange Barrier

BlockingCollection

CountdownEvent

Partitioning {Orderable}Partitioner

Partitioner.Create

Exception handling AggregateException

Initialization Lazy

LazyInitializer.EnsureInitialized

ThreadLocal

Locks ManualResetEventSlim

SemaphoreSlim

SpinLock

SpinWait

Cancellation CancellationToken{Source}


33/47



34/47

Schedulers

Policy-based characteristics

Multiple schedulers

Work-stealing

Cooperative Blocking

Runtime awareness of blockingevents

Dynamic ResourceManagement

Arbitration of competingrequests for processors

Dynamic reallocation based onload

Platform Scalability

Mapping of expressed

parallelism onto availableresources

Low overhead


35/47


36/47

Task

An asynchronous operation

Task group

A collection of tasks thatform a logical computationor sub-computation

Wait or cancel together

Parallel Loops/Regions

parallel_for

parallel_for_each

parallel_invoke

Concurrent containers

concurrent_queue

concurrent_vector

combinable


37/47

Tasks

Tasks Groups

auto t = make_task([]{

...

});

task_group group;

group.run(t);

group.run([]

{

...

});

group.wait();

group.cancel();


38/47


39/47

Buffer Blocks unbounded_buffer

overwrite_buffer

single_assignment

Action Blocks transformer

call

Control Blocks

choice

join

Functions

send(...) / asend(...)

receive(...) /try_receive(...)


40/47



41/47


42/47

Numberof cores

YourProcess

Idle time

Other

processes

CPU Utilization


43/47

Usage

Hints

Detailed thread analysis(one channel per thread)

ActiveLegend

Hideuninteresting

threads

Measure time forinteresting segments

Zoom in and out

Call Stacks

Threads


44/47

Each logical corein a swim lane

One color perthread

Cross-core

migration details

Migrationvisualization

Cores


45/47



46/47

For more information:

http://msdn.com/concurrency Parallel Computing Highlights

Parallel Computing Resources Download sample code

View MSDN Channel 9 Videos
http://msdn.com/concurrencyhttp://msdn.com/concurrency


47/47

Intro to Parallel Programming

Documents

Transcript of Intro to Parallel Programming