November  2010  

Fabrizio  Ferri  (fabferri@microso0.com)  

Windows  HPC  2008  R2  

Agenda  Windows  HPC  2008  R2  session  

!  Windows  HPC  2008  R2  Architecture  

!  Basic  funcDonality  

!  Same  basic  examples  to  run  custom  code  (demo)  

!  Example  of  uDlizaDon  of  HPC  in  academic  environment  

Windows  HPC  Server  2008  R2  Suite  

HPC PACK 2008 R2 Enterprise

Windows Server 2008 R2 x64

#1 DVD

#1 DVD

!  h"p://www.microso..com/hpc    

Head  node

Brokers

Compute  nodes

3  

Win  HPC  2008  R2:  Architecture  (1/2)  !   HPCS  cluster    ==    Windows  Server  2008  R2(64-­‐bit)  +  HPC  Pack  

!   HPC  Pack  ==  set  of  HPCS  services  

HW (real or virtual)

Windows Server 2008 R2

HPC Pack R2

SW (apps, run-time libraries)

Head  node

Brokers

Compute  nodes

SQL

Job submission Scheduling

Cluster mgmt

Job execution Node mgmt

HPC Pack

Windows  HPC  2008  R2:  Architecture  (2/2)  

IT enterprise infrastructure

Compute Node

AD DNS

DHCP

Cluster HPC

Admin  /  User  Cons  WDS  

Job Scheduler

Management  

NAT  

Node  Manager  

Management  

MPI

Enterprise  Network  

Private  Network

Application Network

…… Head Node

Compute Node

1….100

SQL Scheduler

Store

•   Job  ValidaGon  •   Resource    AllocaGon      and  Controller  

Win Desktops to submit jobs

Node  Manager  

Management  

5  

COW:  Cluster  of  worstaDon  !  The  ability  to  add  Windows  7  workstaDons  as  compute  nodes:  

–  Windows  7  Professional  or  Enterprise,  32  or  64-­‐bit  (requires  AcDve  Directory  support)  

–  Desktops  and  cluster  has  to  belong  to  the  same  AD  Domain  

!  Not  deploying  the  workstaDon  OS  

!  Two  policy  to  join  workstaDon  to  cluster:  

− Manual  

−  AutomaDc   Time  of  day  scheduling  for  WorkstaDon  availability  

 Draining  interval  for  graceful  preempDon    

COW  

Define when the availability of the workstation based on a week

Compute  node  diskless:  iSCSI  boot  !   Rapid  deployment  with  controlled  OS  image  

!  

!  

!    

Diskless client

Windows Storage Server iSCSI Software Target

DHCP

Windows DHCP server with TFTP

TFTP

iSCSI  boot  workflow  

Windows  Storage  Server  2008  R2  

Windows  HPC  2008  R2:  how  it  works  – A batch-oriented software system with mainframe-

like job scheduler – An SOA-oriented software system for compute-

intensive service calls

Head  node

... Broker  nodes

Compute  Nodes

...

... Call Call Call

Result

Job task Job

task

Job task

R R

HPCS-based cluster client  

ActiveDir

9  

Core  vs.  Socket  vs.  Node  !   Processes  are  scheduled  onto  processors −  Q:  What  is  meant  in  HPCS  by  the  term  processor?    A:  1  core  =1  processor

−  Core: 1  independent  execution  core

−  Socket: 1  physical  CPU  chip

−  Node: 1  compute  node  (computer  with  CPU(s),  RAM,  HD,  NICs,  OS)

 #1 dual quad-core server = 1 node, 2 sockets, 8 cores  All processors share the same RAM, hard disks, NICs  Cores may have independent caches, may not

Socket Quad-core Intel processor

node socket

C C

C C

socket

C C

C C

1 core = 1 processor 10  

MCRA:  Node/Socket/Core  !  MulGlevel  Computer  Resource  AllocaGon:  Core  based,  Socket  based,  Node  based  

!  Windows  HPC  Server  can  help  your  applicaDon  make  the  best  use  of  mulD-­‐core  systems  allow  smart  resources  allocaDon  

Node 1

S0 P0 P1

P2 P3 S1

P0 P1

P2 P3

S2 P0 P1

P2 P3

S3

P0 P1

P2 P3

Node 2

S0 P0 P1

P2 P3 S1

P0 P1

P2 P3

S2 P0 P1

P2 P3

S3 P0 P1

P2 P3

J1

J3 J2

J1:  /numsockets:3  /exclusive:  false  J3:  /numcores:4  /exclusive:  false  

J2:  /numnodes:1  

J3

J3 J3

J1

J1

11  

Job/Task/Process  

Job  

Task  

Proc  

Task   Task  

Task   Task  

Task   Task  

Task  

Task  

Proc   Proc   Proc  

Task   Task  

Task  

Task  

one  Job  one  Task  

Parametric  sweep  applicaGon  

Serial    ApplicaGon  

MPI  ApplicaGon  

one  Job  mulGple  Tasks  

Task  flow  

!   A  job  is  a  set  of  tasks  

!   A  task  is  an  executable  program  

!   A  program  yields  one  or  more  processes  !   A  job  can  contain:    

−  basic  tasks  or/and  −  parametric  tasks  

In the case of mpiexec, you have a task that yields one or more processes.

12  

Run  a  job  by  XML  Job  Template  !   Jobs  can  be  saved  as  template  to  launch  same  applicaDon  later  

!   Just  loaded  by  user  (eventually  set  input  case  and  number  of  process)  

!   A  task  is  any  executable  program  

!   User  specifies  command-­‐line  to  run  program  

!   Examples:  −  C:\Apps\HPCApp.exe arg1 arg2 −  mpiexec MPIApp.exe arg1 arg2 −  copy \\headnode\Public\file.txt C:\Apps

HPCApp.exe arg1 arg2 …

Run  jobs  by  Powershell  

New  -­‐HpcJob  –Scheduler  hn  –Name  “Test"  -­‐RunTime  "0:4:00"  -­‐Priority  "AboveNormal“  

$MyJob  =  Get-­‐HpcJob  –Scheduler  hn  –Name  “Test“  –state  configuring  

$MyJob  =  Get-­‐HpcJob  –Scheduler  hn  –Name  “Test“  –state  configuring  

$MyJob  |  Add-­‐HpcTask  –Scheduler  hn  -­‐WorkDir  "%UserProfile%\Documents"  -­‐Commandline  "dir“  

$MyJob  |  Submit-­‐HpcJob  –Scheduler  hn    

MyJob.Refresh()  

$MyJob  |  Get-­‐HpcTask  |  fl  

1

14  

2

3

!    HPC  Server  provides  a  plug-­‐in  for  Microso0  PowerShell    

4

5

Job  View  and  Task  Progress  (via  HPC-­‐API)  

15  

Job  Scheduling  Policies  !  The  job  scheduler:  

− Single  job  queue  − Priority-­‐based,  first-­‐come  first-­‐served  (higher  priority  may  preempt  lower)  

− Allocates  as  many  resources  N  as  possible  (fastest  nodes  with  most  RAM  first)  

− Starts  1..N  tasks  running  on  those  resources  − Job  can  acquire  more  resources  ("grow"),  lose  resources  as  tasks  finish  ("shrink")  

!  HPCS  job  scheduler  is  performance-­‐oriented:  

− If  job  requests  Min  ..  Max  resources,  will  assign  as  close  to  Max  as  possible  

− Jobs  do  not  share  resources  —  given  a  core,  no  other  job  uses  that  core  

16  

Scheduling  policy:  Exclusive/non-­‐Exclusive  

!  Resource  allocaDon…  − exclusive  vs.  non-­‐exclusive  use  of  a  node  − preempDon  of  a  job  by  a  higher-­‐priority  job  

17  

Scheduling  policy:  Backfill  policy  

processors Job #1

32 processors

Job #2 64 processors

time

Job #3 (32 processors)

64 processor

Hour 1

Hour 2

Hour 3

Consente ai job aventi un breve tempo di esecuzione di saltare la coda

!  GOAL:  increase  the  uDlizaDon  of  cluster,  without  delaying  the  execuDon  of  my  high  priority  work  

!  The  user  specify  Dme,  resources  required  to  execute  the  job;  if  job  scheduler  idenDfied  available  resources,  small  jobs  can  jump  the  queue  

18  

Service  Balanced  Scheduling  !   GOAL: my cluster runs a number of jobs, and I want to dynamically provide

resources to the jobs that need them most !   Balanced allows to jobs with multiple task to increase (grow) and decrease

(shrink) resource allocation over time !   Priority Bias allow to highest priority jobs to take resource away from low-

priority job

Job 1 Tasks Job 2 Tasks Job 3 High Priority

processors  

time 19  

3D  Tomography  for  Art  ConservaDon  

!  ApplicaDon  of  XRAY  tomography  to  Art  conservaDon  !   The  preservaDon  of  Italy’s  vast  cultural  heritage  is  an  intense,  mulD-­‐disciplinary  effort.  

!   Parallel  compuDng  techniques  for  3D  tomography  require  experDse  in  Physics,  Art,  Computer  Science  

!   PROJECT:  hpp://www.xraytomography.com/  !   Reference:  Dop.  Ssa  Rosa  Brancaccio/Prof.  Levi  Giuseppe  

Radiographies 3D Tomography

Virtual Cuts

20  

CT  system  with  EBCCD:  sequence  of  OperaDons  !  ProjecGons  are  the  radiographs  of  the  object  at  different  angles  

!  Sinograms  are  images  that  contain  all  informaDon  to  reconstruct  1  slice  at  fixed  height  

!  The  process  is  computaDon-­‐intensive  and  complex,  but  each  slice  can  be  processed  independently  

FFT  of  sinograms  

SelecGon  1  line  in  FFT  space  

Filter  line  

Reverse  FFT  

Reverse  projecGon  of  result  into  normal  space  

5  steps  

Electron-­‐Bombarded    Charge  Coupled  Device  

An  EBCCD  camera  collects  the  X-­‐ray  image  on  a  30  x  30  cm  screen  of  scinGllaGng  material.  The  image  represents  the  radiographic  projecGon  of  the  object.  

CT  system  setup  

X-ray tube globe on a turntable

detector  and  the  translaGon  axes  

(31000  radiographs)  Pixel size: ~= 800µm Pixel number for a whole projection: ~4500x280

Globo  ‘DanD’  (~1567)    The  Map  Room  (“Sala  delle  Carte  Geografiche”)  within  Palazzo  Vecchio,  with  the  

ancient   large   globe   created   by   Egnazio   DanG   around   1567,   on   assignment   of  

Cosimo  I  de’  Medici,  duke  of  Florence.  

Globo  ‘DanD’:  14  slices  in  a  planar  projecDon  

Perspective distortions must be corrected before performing the CT

reconstruction.

3D  CT  ReconstrucDon  14  secGons  in  planar  projecDon  

280 slices  

CT  ReconstrucDon  on  a  Windows  HPC  cluster  !   The  problem  lends  itself  to  a  parametric  sweep  approach  because  of  data  parallelism:  −  Compute  nodes  in  HPC  cluster  execute  same  process  on  different  projecDons,  mostly  asynchronously  

!   The  analysis  code  was  simply  recompiled  with  Visual  Studio  2008  

!   Analysis  of  the  same  images  on  a  5-­‐nodes  Windows  HPC  cluster  achieved  an  impressive  speedup  (>20)  compared  to  standard  processing.  

System Tasks Bit Total time

(dd:hh:mm:ss) Speed

Rate Factor

Unibo single 32 20:15:24:18 1

HPC 240 64 00:17:23:16 28.49

HPC 797 64 00:17:15:43 28.70

HPC 1594 64 00:17:14:18 28.74

Times  to  reconstruct  all  12  secGons  as  funcGons  of  tasks  number.  

Windows  HPC  2008  R2  info  !   Win  HPC  portal:  hpp://www.microso0.com/hpc/en/us/default.aspx  

!   Win  HP  community:  hpp://windowshpc.net/Pages/Default.aspx  

!   HPC  (manual  technet):  hpp://technet.microso0.com/en-­‐us/library/ee783547(v=WS.10).aspx    

!   MS  HPC  Pack  2008  SDK:  hpp://msdn.microso0.com/en-­‐us/library/cc853440(VS.85).aspx    

!   MPI:  hpps://compuDng.llnl.gov/tutorials/mpi/  

!   MPI  code  example:  hpp://code.msdn.microso0.com/LoperyMPI  

!   Lizard:  Microso0  HPC  Pack  2008  R2  Tool  Pack  hpp://www.microso0.com/downloads/en/details.aspx?FamilyID=3313856B-­‐02BC-­‐4BDD-­‐B8B6-­‐541F5309F2CE&displaylang=en    

!   Windows  Performance  toolkit:  hpp://www.microso0.com/whdc/system/sysperf/per0ools.mspx  

!   MPI  Cluster  Debugger  

hpp://www.microso0.com/downloads/details.aspx?displaylang=en&FamilyID=d3993532-­‐bdf8-­‐4024-­‐b164-­‐db2ee8a851f5  

hpp://visualstudiogallery.msdn.microso0.com/en-­‐us/90w60d4-­‐0b8c-­‐472f-­‐8135-­‐683d3c45f45a  26