11

Parallel Performance AnalysisParallel Performance Analysiswith Open|SpeedShop with Open|SpeedShop

Trilab Tools-WorkshopTrilab Tools-Workshop

Martin Schulz, LLNL/CASCMartin Schulz, LLNL/CASC

LLNL-PRES-426152

Features, Capabilities, StatusOpen Source Performance Analysis Tool FrameworkOpen Source Performance Analysis Tool Framework

Most common performance analysis steps Most common performance analysis steps all in one toolall in one tool ExtensibleExtensible by using plugins for data collection and representation by using plugins for data collection and representation

Flexible and Easy to useFlexible and Easy to use User access through User access through GUIGUI, , CommandCommand LineLine, and , and PythonPython ScriptingScripting

Several Instrumentation OptionsSeveral Instrumentation Options All work on All work on unmodifiedunmodified applicationapplication binariesbinaries Offline Offline andand online data collection online data collection // attach attach to running applicationsto running applications

Target: Cluster systems and MPPsTarget: Cluster systems and MPPs LinuxLinux ClustersClusters with x86, IA-64, Opteron, and EM64T CPUs with x86, IA-64, Opteron, and EM64T CPUs Ports of data collection to more exotic platforms soonPorts of data collection to more exotic platforms soon

Status & AvailabilityStatus & Availability Version 1.9.3 just released / working on several large lab codesVersion 1.9.3 just released / working on several large lab codes Installed on all Trilab TLCC systems / more to comeInstalled on all Trilab TLCC systems / more to come

Results

Experiment Workflow

Ru

nApplication

“Experiment”

Results can be displayed using several “Views”

Process Management

Panel

Consists of one or more data “Collectors”

Stored in SQL database

Experiment Types in O|SS

Sampling ExperimentsSampling Experiments Periodically interrupt run and record locationPeriodically interrupt run and record location Report statistical distribution of these locationsReport statistical distribution of these locations Good overview at modest overheadGood overview at modest overhead pcsamp, usertime, hwc, hwctimepcsamp, usertime, hwc, hwctime

Tracing Experiments Tracing Experiments Gather and store individual application events, Gather and store individual application events,

e.g., function invocations (MPI, I/O, …)e.g., function invocations (MPI, I/O, …) Detailed information at higher overheadDetailed information at higher overhead io, iot, mpi, mpit, mpiotf, fpeio, iot, mpi, mpit, mpiotf, fpe

Three InterfacesExperiment Commands expAttach expCreate expDetach expGo expView

List Commands list –v exp list –v hosts list –v src

Session Commands setBreak openGui

import openss

my_filename=openss.FileList("myprog.a.out")my_exptype=openss.ExpTypeList("pcsamp")my_id=openss.expCreate(my_filename,my_exptype)

openss.expGo()

My_metric_list = openss.MetricList("exclusive")my_viewtype = openss.ViewTypeList("pcsamp”)result = openss.expView(my_id,my_viewtype,my_metric_list)

GUIGUI

Command Line

Command Line

Python ModulePython Module

Performance Analysis in O|SS

Parallel codesParallel codes MPI and Threaded codesMPI and Threaded codes Apply any experiment to all tasksApply any experiment to all tasks Choose aggregate vs. task specific viewsChoose aggregate vs. task specific views

Pre-canned experimentsPre-canned experiments Load balanceLoad balance Cluster analysisCluster analysis

Custom comparisonsCustom comparisons Multiple experiments, metrics, ranks, …Multiple experiments, metrics, ranks, …

Availability

Platform Ver. Usage Doc. POC

LLNL/TLCCOCF

1.9.3.4 use openss https://computing.llnl.gov/code/oss.html

Martin Schulzschulzm@llnl.gov

LLNL/TLCCSCF

1.9.3.4 use openss https://computing.llnl.gov/code/oss.html

Martin Schulzschulzm@llnl.gov

LLNL/Dawn soon soon Martin Schulzschulzm@llnl.gov

SNL/Glory 1.9.3.4 see below http://www.openspeedshop.org/wp/documentation/

Mahesh Rajanmrajan@sandia.gov

LANL/Red 1.9.3.4 Mod: hpc-toolsMod: openss

man opensshttp://hpc.lanl.gov/open_speed_shop

consult@lanl.gov

LANL/YellowYellowrail, Turing

1.9.3.4 Mod: hpc-toolsMod: openss

man opensshttp://hpc.lanl.gov/open_speed_shop

consult@lanl.gov

LANL/TurquoiseLobo, Cerrillos and Coyote

1.9.3.4 Mod: hpc-toolsMod: openss

man opensshttp://hpc.lanl.gov/open_speed_shop

consult@lanl.gov

Usage for SNL/Glory:module switch mpi mpi/mvapich-1.1_intel-11.1-f064-c064module load /home/jgalaro/privatemodules/openspeedshop-1.9.3.4OR module load /home/jgalaro/privatemodules/openspeedshop-1.9.3.4-mvapichOR module load /home/jgalaro/privatemodules/openspeedshop-1.9.3.4-openmpi

Usage Instructions

Option 1: Shortcut commandsOption 1: Shortcut commands osspcsamp “<command>”osspcsamp “<command>” Command can be sequential or parallelCommand can be sequential or parallel ossusertime/osshwc/ossio/ossmpi/…ossusertime/osshwc/ossio/ossmpi/…Option 2: Graphical User InterfaceOption 2: Graphical User Interface Launch GUI: Launch GUI: openssopenss Wizards to run experimentsWizards to run experiments View performance resultsView performance resultsOption 3: Command lineOption 3: Command line openss –cliopenss –cli Use Use helphelp command to get details command to get details

Advanced Concepts

MPI Application

O|SS

post-mortem

OfflineMPI Application

O|SS

MRNet

Easy setup

Low overhead

No additional resources

Higher portability

Online analysis

Intermediate updates

Attach to running code

Optional aggregation

Availability and Contact

Open|SpeedShop website:Open|SpeedShop website: Tutorials & User GuidesTutorials & User Guides

http://www.openspeedshop.org/

Lab Instructions:Lab Instructions: https://computing.llnl.gov/code/oss.html

Demo Session 2Demo Session 2 Open|SpeedShop 101Open|SpeedShop 101 Test O|SS on your codeTest O|SS on your code Different views and other GUI optionsDifferent views and other GUI options