Visualisation Services at SCD/Hartree Center193.62.125.70/CIUK-2016/MartinTurner.pdf ·...
Transcript of Visualisation Services at SCD/Hartree Center193.62.125.70/CIUK-2016/MartinTurner.pdf ·...
CIUK December 2016
Visualisation Services at SCD/Hartree Center
Aim to have the human back into the Experimental-Simulation-
Computational-Loop in order to more efficiently use STFC
imaging facilities and HPC cycles
Martin Turner
Erica Yang, Srikanth Nagella
Ron Fowler, Evgueni Ovtchinnikov
Emma Tattershall, Barry Searle
Rob Allan,
George Leaver, Simon Hood
Tom Christy, David Hughes
+ thanks to many others
http://www.ccpi.ac.uk/
Collaborative Computational Project in Tomographic Imaging has been funded for
the next phase 2015-2020 and will use a range of computational HPC facilities.
Visualisation Matters
� Sites @ STFC
� Laying Fibre
� Use Cases
� Adding GPUs
� Adding login Nodes
� CCPi links
Viglen Cluster uses etc…
TSB Space Applications Catapult
University of Reading
Visualisation Matters Example
SCD contracted Dr Lee
Margetts and Ms Louise
Lever at the University of
Manchester to help add
transient thermal modelling
capability to the open source
parallel finite element
software ParaFEM. This
enabled Dr Llion Evans at
the Culham Centre for
Fusion Energy and his team
to use it to study plasma
facing wall components.
Visualisation Matters Example
SCD contracted Dr Lee
Margetts and Ms Louise
Lever at the University of
Manchester to help add
transient thermal modelling
capability to the open source
parallel finite element
software ParaFEM. This
enabled Dr Llion Evans at
the Culham Centre for
Fusion Energy and his team
to use it to study plasma
facing wall components.
Visualisation Matters Example
SCD contracted Dr Lee
Margetts and Ms Louise
Lever at the University of
Manchester to help add
transient thermal modelling
capability to the open source
parallel finite element
software ParaFEM. This
enabled Dr Llion Evans at
the Culham Centre for
Fusion Energy and his team
to use it to study plasma
facing wall components.
Visualisation Matters:Creating a discussion space
Visualisation Matters:Discovering insight
Visualisation Matters:Commercial Example
Software demonstration by Thomas Salvini from
construqtive.com that turned our curved wall into an ipad-
mini controlled dashboard for selected and changeable
visual analytics output. Initial test with the Clarity3 software
in its vanilla format.
Use Case: Hartree development“having an argument”
Different users using ParaView etc.;
� multi-modality and stereoscopic display for Matthieu
Chavent (University of Leeds Researcher in Molecular
Visualisation), Biophysical Modelling, Biological
Complexity
Use Case: The human-in-the-computational-loop: still
important for supercomputing
Training and steering example: Stefano Rolfo and Charles Moulinec presenting
and running CFD code in the Hartree training room, and then in the Hartree
Visualisation Facility.
IBM Blue Gene/Q are highly ranked on the green
computing list (STFC system was #30 on the November
2014 Top500 Green List at 2,178 mflops per watt) the
computational cycles are not free and still limited
to a certain size of cores.
Use Case: Software for the Future: integrating ParaView and OpenFOAM
One of the most useful aspects of ParaView is its ability to run in client-server mode so
that the data of a simulation can remain on the parallel file system it was generated on
and only the required visualisation geometry is sent to the client. This avoids the need to
transfer many GigaBytes of data to the client, which can be slow and inconvenient.
iCSF (interactive) vs CSF (batch)
� iCSF - run interactively
– A backend node becomesyour workstation
– Start application GUI
• E.g., MATLAB, Rstudio, Stata, SAS
� CSF - run in batch
– No direct access tobackend nodes
– Batch system runs jobson backends without GUIs Login
node
Jobqueue
/home
/scratch
Backend batch nodes
= GPU
home shared
RDS (Isilon)
Loginnode
/home
Backend interactive nodes
What is the CSF?
� Computational Shared Facility
� A general purpose batch compute cluster
Loginnode
Job
queue
/scratch
home
Backend compute nodes
Connect to the login
node remotely.
No direct access
to backend
compute nodes.
Two main storage
areas visible to all
nodes.
100s of compute
nodes (~9300 cores)
GPU compute nodes
Compute nodes
connected together
by local network for
parallel apps & files.
All work (“jobs”) submitted
to batch system's job queue
18home shared
RDS (Isilon)
Hardware: Compute Nodes
� CSF has a variety of compute nodes
– We’ll cover which to use, why and how later
19
Intel: 2 x 6-core Xeon
(Westmere)
64 & 96 GB RAM (one with 512GB)
GigE or InfiniBand networking64 / 96GB
Intel: 2 x 6-core Xeon
(Sandybridge)
64 & 256 GB RAM
GigE networking64/256GB
Intel: 2 x 8-core Xeon
(Ivybridge)
64 GB RAM
GigE networking64 GB
AMD: 4 x 8-core Opteron
(Magny-Cours)
64 GB RAM
InfiniBand networking64 GB
AMD: 4 x 16-core Opteron
(Bulldozer)
128 GB RAM
InfiniBand networking128 GB
128 GB
Intel: 2 x 12-core Xeon
(Haswell)
128 GB RAM
InfiniBand networking128GB
What is the iCSF?
� interactive Computational Shared Facility
� Non-batch compute cluster
Loginnode
/scratch
home
Backend compute nodes
Connect to the login
node remotely.
Two main storage
areas visible to all
nodes.
interactive compute
nodes shared by
multiple users.
20home shared
RDS (Isilon)
login node forwards
you to a backend
node.
256GB 256GB 2TB
64GB
Part of the Computationally
Intensive Research Ecosystem
21
Similarities of iCSF and CSF
� Both are a collection of servers (compute nodes) linked
together to form two large systems
� Operating system on both is the same – Scientific Linux.
– Need to use the command line a little on iCSF,
– CSF more command-line heavy
� They share the same home directory (similar to your P-
Drive - a private area for your files)
� Run work on CSF in batch, analyse/Viz in a GUI on iCSF
� To use both you must be part of a group/faculty that has
contributed hardware
Differences between iCSF and CSF
� iCSF (aka incline - interactive computational linux env)– The ‘i’ means interactive. No batch. Run apps straightaway.– Has a login node, but you are automatically routed to a compute node.– Suitable for GUI applications - start your app and use as you would on a
desktop.– Ideal for code development and testing.– Instant access to compute resources.– Typically running one job/app at a time (possibly a few together)
� CSF (computational shared facility)– Accessed and used via a login node. No direct access to the compute nodes.
– All work must be submitted to a batch system.
– Very command-line based.
– Hand your work over to the system, go for a coffee or on holiday/conference, come back and collect results.
– Large scratch filesystem so that running jobs don’t run out of disk space
Starting apps on the iCSF
� You need to load a modulefile to use the installed software. Run at inclineNN prompt, EG:
module load apps/gcc/rstudio/0.98.983module load apps/binapps/stata/14
module load apps/binapps/matlab/R2014amodule load apps/binapps/sas/9.4
� Then start it (the & lets you keep using the terminal):rstudio &
xstata
matlab &
sas &
� You should see a familiar looking GUI/window
� The software pages tell you which modulefile you need:– http://ri.itservices.manchester.ac.uk/icsf/software/
0. Your Windows desktop 2. X2GO connected to nyx3,4(gives you a Linux desktop to work from)
1. X2GO
4. Log in to iCSF from nyx3 desktop5. Run matlab on iCSF(windows appear on nyx3 desktop)
6. X2GO(matlab keeps running!)
3. Start a terminal on nyx3 desktop Hartree Centre ADD Login Nodes
Two Quadro 5000
65GB 32 core : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
NVIDIA-SMI 361.45 Driver Version: 361.45.11
There are 2 extra login nodes currently,
� one running EnginFrame and DCV and the other
� for general visualisation use, e.g. with Virtual GL, VisIt, ParaView, etc.
NICEDesktop Cloud Visualization (DCV)
End users:
• Intuitive, application-centric Web interface to start, control and re-connect to a session
• Single sign-on for batch and interactive applications
• All data transfers from and to the remote visualization farm are handled by EnginFrame
• Built-in collaboration, to share sessions with other users
• The load and usage of the visualization cluster is monitored in the browser
• The solution also delivers significant added-value for the system administrators:
• No need of SSH / SCP / FTP on the client machine• Easy integration into identity services, Single Sign-On (SSO), Enterprise portals• Automated data life cycle management• Built-in user session sharing, to facilitate support• Interactive sessions are load balanced by a scheduler (LSF, GridEngine or Torque/MOAB) to achieve
optimal performance and resource usage• Better control and use of application licenses• Monitor, control and manage users’ idle sessions
NICE (DCV)
� Coupled with these third party remote visualization engines (which specialize in delivering high frame-rates for 3D graphics), the NICE offering for Remote Visualization solves the issues of user authentication, dynamic session allocation, session management and data transfers.
� When using NICE DCV, the scene geometry and graphics state are rendered on a central server, and pixels are sent to one or more remote displays.
� This approach requires the server to be equipped with one or more GPUs, which are used for the OpenGL rendering, while the client software can run on "thin" devices.
Remote visualisation on HartreeA Constelcom Ltd. project
Initial results using ParaView client-
server and TurboVNC/VirtualGL
Hartree iDataPlex
� Fast access to data on GPFS file system
� High performance graphics nodes each with:
– 2 nVidia quadro 6000 cards
– 64 GBytes memory
– Access to X11 server for remote rendering
� Security access – only ssh access allowed
– Use ssh port forwarding
ParaView X11/GLX forwarding
• Requires client support for GLX extensions
• Does not user remote GPU
• Slow response due to verbose X11 protocol
• Lack of X11-extensions may crash program
• Unusable on slow connections
% ssh –X user@gfxlogin8
$ paraview &
$
ParaView client server
� Client paraview; pvserver on remote
� Communication via ssh port forward
� Rendering: simple on client, complex on remote
# client connects to server over tunnel
% paraview –url=cs://localhost
Advantages:
• Fast GUI response
• Large memory on
server
• Parallel processing
# forward port & start remote servers% ssh –L11111:localhost:11111 gfxlogin8$ DISPLAY=:0 mpirun –n 4 pvserver …
ParaView turboVNC/VirtualGL
� All processing on remote, any VNC client on local
� VirtualGL to exploit remote GPU
# port forward, start vncserver
% ssh –L5902:localhost:5902 gfxlogin8
$ vncserver :2
# run vnc client locally
% vncviewer :2
# in vncviewer shell
$ vglrun paraview &
# optional
$ DISPLAY=:0 mpiexec –n 4 pvserver
Advantages:
• Good on slow links
• Clients on Ipad,
Android
Diamond Light Source
ISIS Neutron Source
Central Laser Facility
STFC Facilities
Visualisation
& Computing
Use Case: IMAT beamline: linking the human-visualisation to the facility
imaging capture process
The IMAT (Imaging and Materials Science & Engineering) beamline, due for use in 2016-
7, is going to be the first neutron imaging instrument at ISIS that will offer unique time-of-
flight tomography-driven diffraction techniques.
This will capitalise on the latest image reconstruction procedures and event mode data
collection schemes involving a strong visualisation component. ParaView has been tested
and will incorporate fast parallel image reconstruction algorithms with on-the-fly image
processing and visualisation to inform and guide experiments.
SCARF Configuration
The hardware for
SCARF is purchased
on a yearly basis,
with each new
addition added to the
same set of
computational job
queues so as to give
the users a seamless
experience.
Backend VM nodes
Horizon VMware software which has integrated support for
NVIDIA Grid K2.
� Target VMs are Windows only.
� Users have to download a horizon client and …
� will be able to get the VM desktop to their PC/Laptops.
ULTRA is underpinned
by an array of high-end
computing
technologies, including
high speed data
acquisition, high
throughput data
transfer, cluster
computing, parallel
rendering and remote
visualisation to enable
end-to-end fast parallel
reconstruction
workflows for facility
users.
Summary
� Connect to your data storage
� Add memory to login nodes
� … then lots of cores
� … and a GPU card
� Have graphics facilities capable of 50-100GB
� Learn how to be intelligent >2TB data is okay
� There are solutions with GPUs
� … and login nodes make a great location
Contacts
http://tinyurl.com/STFCVis