OpenMP - IPSltodi.est.ips.pt/aabreu/openMP.pdf · #pragma omp sections [clause ...] newline...

OpenMP

Antonio Abreu

Instituto Politecnico de Setubal

1 de Marco de 2013

Antonio Abreu (Instituto Politecnico de Setubal) OpenMP 1 de Marco de 2013 1 / 37

openMP – what?

It’s an Application Program Interface (API) that allows parallel programsto be explicitly and simply developed, in C/C++, for multi-platform,shared memory, multiprocessor computers (including Solaris, AIX, HP-UX,GNU/Linux, Mac OS X, and Windows platforms), supported by the majorcomputer hardware and software vendors (including AMD, IBM, Intel,Cray, HP, Fujitsu, Nvidia, NEC, Microsoft, Texas Instruments, OracleCorporation, and others.).

cores and memory

Multicore computers have a memory system where some memories areshared while others are not. The next figure makes this distinction clear.TLB stands for Translation Lookaside Buffer, which is an address cache.When making parallel programs one must know which memory is sharedand which memory is not.

Fork – join

OpenMP is based on multithreading, i.e., a form of parallelization wherebya master thread forks a specified number of slave threads, with theruntime environment allocating threads to different processors.

How many cores does my machine have?

In linux, the file /proc/cpuinfo contains a lot of information about thehardware of the machine. Typing less /proc/cpuinfo allows one to seeit all.

To see info about memory, see the contents of the file /proc/meminfo.The first number one wants to see is the one corresponding to MemTotal.

In order to use openMP, one has to have a propoer compiler. In linux,GCC 4.2 or higher supports openMP. To see the version of your (linux)compiler, type the command gcc -v.

parallel directive

#pragma omp parallel [clause ...] newline

structured_block

where clause can be

if (scalar_expression)

private (list)

shared (list)

default (shared | none)

firstprivate (list)

reduction (operator: list)

copyin (list)

num_threads (integer-expression)Antonio Abreu (Instituto Politecnico de Setubal) OpenMP 1 de Marco de 2013 6 / 37

Hello world

#include <stdio.h>

#include <omp.h>

int main(void)

#pragma omp parallel

int ID = omp_get_thread_num();

printf("Hello (%d)\n",ID);

printf("world (%d)\n",ID);

printf("! (%d)\n",ID);

return 0;

Compile with gcc -fopenmp hello.c -o hello

Hello (0)

world (0)

Hello (1)

world (1)

Hello (2)

world (2)

Hello (3)

world (3)

The code between the curly brackets (after the pragma directive) is set toexecute in a predetermined number of threads.

After the first curly bracket there is a fork, i.e., the master thread createsa team of parallel threads, and after the second curly bracket there is ajoin, i.e., the master thread continues execution after all the slave threadsend. The second curly bracket constitutes a barrier, of which only themaster thread passes.

The number of threads is typically set to the number of cores in themicroprocessor; it can be set by the command lineexport OMP_NUM_THREADS=4.

omp_get_thread_num() is a function that returns the Id of the respectivethread. The master thread has Id 0 and makes part of the thread team.

We observe an ordered output, but sometimes this may not happen; infact there is a race condition because the four threads share the standardoutput.

Note that openMP is not necessarily implemented identically by allvendors. Also, it does not provide check for data dependencies, dataconflicts, race conditions, or deadlocks. In particular, it does not guaranteethat input or output to the same file is synchronous when executed inparallel. It is up to the programmer to synchronize input and output.

Synchronization Constructs – barriers

#include <omp.h>

#include <stdio.h>

#include <stdlib.h>

int main (int argc, char *argv[]) {

int th_id, nthreads;

#pragma omp parallel private(th_id)

th_id = omp_get_thread_num();

printf("Hello World from thread %d\n", th_id);

#pragma omp barrier

if ( th_id == 0 ) {

nthreads = omp_get_num_threads();

printf("There are %d threads\n",nthreads);

return EXIT_SUCCESS;

Hello World from thread 1

There are 4 threads

Barriers are a synchronization primitive. This means that all threads in theteam wait for the last one to reach the barrier. At that moment, allthreads in the team resume execution in parallel. If there is a thread thatdoes not reach the barrier, all threads in the team wait, and the processhangs without any work being produced.

If we comment the barrier pragma in the code above the output will be,

There are 4 threads

Explain why.

If we add the code

printf("Bye from thread %d\n", th_id);

after the if, what would be the output?

Workshare directives – for

#pragma omp for [clause ...] newline

for_loop

where clause can be,

schedule (type [,chunk])

ordered

private (list)

firstprivate (list)

lastprivate (list)

shared (list)

collapse (n)

nowait

parallel for example

#include <omp.h>

#define CHUNKSIZE 100

#define N 1000

main ()

int i, chunk = CHUNKSIZE;

float a[N], b[N], c[N];

/* Some initializations */

for (i=0; i < N; i++)

a[i] = b[i] = i * 1.0;

#pragma omp parallel shared(a,b,c,chunk) private(i)

#pragma omp for schedule(dynamic,chunk) nowait

for (i=0; i < N; i++)

c[i] = a[i] + b[i];

} /* end of parallel section */

The for pragma asks the compiler to create threads from the N iterationsof the for loop.

The clause schedule informs the OS (operating system) about how toschedule those threads. In this case, the scheduling policy is dynamic,which means that threads are dynamically assigned on afirst-come-first-serve basis.

In this case each thread will execute chunk (i.e., 100) iterations of thetotal of 1000 in the loop.

The nowait clause makes the implied barrier at the end of the fordirective to be ignored. Put differently, if there was not such a clause, allteam threads stop at the end of the for primitive, and only thread 0would continue past this point.

In the following program, which for cycle is executed in parallel: the first,or both? Before answering, note that the clauses parallel and for arecombined in a single one. This is valid.

#include <stdio.h>

int main(int argc, char *argv[])

const int N = 100;

int i, a[N];

#pragma omp parallel for

for (i = 0; i < N; i++)

a[i] = 2 * i;

for (i = 0; i < N; i++)

printf("%d ",a[i]);

return 0;

Workshare directives – sections

#pragma omp sections [clause ...] newline

#pragma omp section newline

structured_block

#pragma omp section newline

structured_block

where clause can be,

private (list)

firstprivate (list)

lastprivate (list)

nowait

section directive example

#include <stdio.h>

#include <omp.h>

int main(void)

#pragma omp parallel sections

#pragma omp section

printf("hello from thread %d\n",omp_get_thread_num());

#pragma omp section

printf("Bye from thread %d\n",omp_get_thread_num());

A few executions

First execution

hello from thread 0

Bye from thread 0

Second execution

hello from thread 0

hello from thread 1

hello from thread 3

Bye from thread 0

Third execution

hello from thread 2

hello from thread 1

hello from thread 0

Bye from thread 0Antonio Abreu (Instituto Politecnico de Setubal) OpenMP 1 de Marco de 2013 22 / 37

Another example

#include <stdio.h>

#include <omp.h>

int main(void)

int i=0;

#pragma omp parallel sections if (i==1)

#pragma omp section

printf("Bye from thread %d\n",omp_get_thread_num());

}Antonio Abreu (Instituto Politecnico de Setubal) OpenMP 1 de Marco de 2013 23 / 37

Unique result

hello from thread 0

Bye from thread 0

Since the condition is false, the team of threads is not created; but themaster thread stands. Note that the assigned work (three blocks of code)is executed serially; so the if clause permits to parallelize work or not(i.e., to seriallize it), and the decision is made at runtime.Also, there is an implicit barrier at the end of each section. This explainswhy Bye from ... (in the last two examples) is always the last messageto print.

Clause reduction

At the creation of a team of threads the variables in list are created asprivate. At the end of the threads in the team, operator is applied to thevariables in list, a process known as reduction, and the final result iswritten back to the variables in list, now seen as global shared variables.Variables in list must be scalar; not arrays or structures.

#include <stdio.h>

#include <omp.h>

int main(void)

int t=0;

omp_set_num_threads(4);

#pragma omp parallel reduction(+:t)

t = omp_get_thread_num() + 1;

printf("local %d\n", t);

printf("reduction %d\n", t);

Result

local 1

local 2

local 3

local 4

reduction 10

The function of omp_set_num_threads() is self explanatory. Asexpected, it cannot be called from a parallelized block of code.

Synchronization Constructs – atomic

Used to identify a memory location that should not be modifiedsimultaneously by more than one thread in the team. In other words, itprovides an atomic access to the memory location.

#pragma omp atomic

<statement_block>

The directive applies only to a single statement.

Synchronization Constructs – single

Used when there is a block of code that must be executed by a singlethread in the team. Note that by no means this implies that the code ismade atomic. It may happen that other threads (outside this team) accessthe same memory location, thus creating a race condition.

#pragma omp single [clause[[,] clause] ...]

statement_block

Threads in the team that do not execute this directive, wait at the end ofthe code block, unless a nowait clause is specified.

Synchronization Constructs – master

Used to identify a block of code that must executed only by the masterthread.

#pragma omp master

statement_block

Synchronization Constructs – critical

Specifies a block of code that must be executed by only one thread at atime. In other words, if the code in a critical region is executing, no otherthread with that code will run in parallel.

#pragma omp critical [(name)]

statement_block

Different critical regions with the same name are treated as the sameregion. All unnamed critical regions are treated as the same region.

Example

#include <omp.h>

main()

int x;

x = 0;

#pragma omp parallel shared(x)

#pragma omp critical

x = x + 1;

} /* end of parallel section */

Synchronization Constructs – flush

This directive identifies a point at which a consistent view of memory mustexist, i.e., thread-visible variables are written back to memory is responseto this directive.

#pragma omp flush [ (list) ]

Remember the first figure of these course notes. This directive forces thedata in the data cache of each core to be written to the shared unifiedcache memory (and not necessarily to the main memory; that decision ismade by the virtual memory system).

openMP functions about threads

#include <stdio.h>

#include <omp.h>

int main(void)

printf("omp_get_max_threads=%d\n",omp_get_max_threads());

omp_set_num_threads(2);

printf("omp_get_num_procs=%d\n",omp_get_num_procs());

#pragma omp parallel

printf("omp_get_thread_num=%d\n",omp_get_thread_num());

omp_get_max_threads=4

omp_get_num_procs=2

omp_get_thread_num=0

omp_get_num_procs() returns the number of processors in the machine.

Synchronization – locks

omp_lock_t lck;

omp_init_lock(&lck);

#pragma omp parallel private (tmp,id)

id = omp_get_thread_num();

tmp = do_lots_of_work(id); // critical region wrt tmp

omp_set_lock(&lck);

printf(%d %d",id,tmp); // atomic access to id and tmp

omp_unset_lock(&lck);

tmp = do_more_lots_of_work(id); // critical region wrt tmp

omp_destroy_lock(&lck);

Bibliography

wikipedia

http://openmp.org

https://computing.llnl.gov/tutorials/openMP/

http://msdn.microsoft.com/

http://publib.boulder.ibm.com

OpenMP - IPSltodi.est.ips.pt/aabreu/openMP.pdf · #pragma omp sections [clause ...] newline...

Documents

Transcript of OpenMP - IPSltodi.est.ips.pt/aabreu/openMP.pdf · #pragma omp sections [clause ...] newline...

OpenMP on GPUs, First Experiences and Best Practiceson-demand.gputechconf.com/gtc/2018/presentation/s... · #pragma omp taskwait Launch kernel asynchronously, annotating the dependency

Introduction to Matlab#pragma omp parallel sections {#pragma omp section funcA(); #pragma omp section funcB();} /*-- End of parallel region --*/ • The most common use of the sections

Shared memory programming with OpenMP · #pragma omp parallel Parallel region, teams of threads, structured block, interleaved execution across threads int omp_get_thread_num() int

THOMAS APPLENCOURT–APL@ANL - Argonne Leadership Computing … · 2019-10-03 · ECP Provided Programming Models. OPENMP 5 ... #pragma omp target update [clause[[,] clause],…]

SIMD Programming - 東京大学 · 2018-11-08 · simd pragma 1 void axpy_omp(float a, float * x, float c, long m) {2 #pragma omp simd 3 for (long j = 0; j < m; j++) {4 x[j] = a *

OpenMP - RWTH Aachen Universityhpac.rwth-aachen.de/teaching/pp-17/07.OpenMP-3.pdf · Loopconstruct-Clauses #pragma omp for [clause [, clause] ...] Thefollowingclausesapply: private,

OpenMP Accelerator Offloading using OpenCL with SPIR-Vllvm.org/devmtg/2018-04/slides/Schuermann-OpenMP Accelerator Offloading.pdf · #pragma omp target • map code section to a device

Newline BLACK

Loop Optimizations in LLVM: The Good, The Bad, and The Ugly#pragma _CRI unroll #pragma _CRI fusion #pragma _CRI nofission #pragma _CRI blockingsize #pragma _CRI interchange #pragma

Pragma Corporate Finance · Pragma Corporate Finance . PRAGMA Table of Contents 2 Page Section 1 Pragma Overview 3 Section 2 Recent Transactions Completed by Pragma 11

OpenMP: Vectorization and #pragma omp simdhpac.cs.umu.se/teaching/pp-16/material/08.OpenMP-4.pdf · 2016. 12. 8. · The OpenMP simd pragma I Uni es the enforcement of vectorization

Newline Cast Support and user guide - DisplayNote guide...Newline Cast -Support and user guide Newline Cast receiver Newline Cast receiver is a software application for Windows that

OpenMP programming · #pragma omp barrier synchronization, must be encountered by all threads in a team (or none) #pragma omp ordered {a block of codes} another form of synchronization

GNAT Reference Manual - GNU Project · iv 2.134 Pragma Post Class ::::: 69 2.135 Pragma Rename Pragma ::::: 70 2.136 Pragma Pre ::::: 70

Software to examine cores · You could put #pragma omp for just in front of second loop, which is pretty good.! Alternatively, you can use #pragma omp for collapse(2) in front of

NEW014 - Newline LaundryGuide v05 - Newline Anglia

ALVATORE I GIROLAMO DPHPC: Introduction to OpenMP · spcl.inf.ethz.ch @spcl_eth What is it? A set of compiler directives and a runtime library #pragma omp parallel num_threads(4)

«OpenMP Programming Model - Tasks» Panos Hadjidoukas · • #pragma omp parallel • #pragma omp sections • #pragma omp parallel for • “parallel for” is great for for-loops,

Shared Memory Programming With OpenMPjburkardt/classes/acs2_2008/... · 2011-10-06 · the string pragma omp followed by the name of the directive. # pragma omp parallel # pragma

OpenMP on Ranger and Stampede (with Labs)...• The following OpenMP directives prevent this race condition: #pragma omp critical – for a code block (C/C++) #pragma omp atomic –

Introduction to Matlab#pragma omp parallel sections {#pragma omp section funcA(); #pragma omp section funcB();} /-- End of parallel region --/ • The most common use of the sections