Neutron Science TeraGrid Gateway Update

Vickie Lynch, Meili Chen, John Cobb

Oak Ridge National Laboratory

AUS Technical Presentation

October 8, 2009

2 Managed by UT-Battellefor the U.S. Department of Energy

Outline

• Sinergia Startup Project

• Fitting Service

• Transition to SNS

3 Managed by UT-Battellefor the U.S. Department of Energy

Sinergia Startup Project

• Collaboration between the University of Zürich, ETH Zürich and Oak Ridge National Laboratory (ORNL)

• Swiss Sinergia grant supports PostDoc at Zürich and graduate student at ORNL

4 Managed by UT-Battellefor the U.S. Department of Energy

Diffuse Scattering Modeling

• 3D fitting of large datasets

• Needed for Single Crystal instruments– Topaz (SNS)– Snap (SNS)– MaNDi (SNS)– Corelli (SNS)– Four-circle Diffractometer

(HFIR)– Imagine (HFIR)

5 Managed by UT-Battellefor the U.S. Department of Energy

Sinergia July 09 Kickoff in Zurich

• Organized by Hans-Beat Bürgi

• Michel Chodkiewicz (PostDoc at Zürich) will develop new C++ diffuse scattering code over 3 years

• Tara Clark (Grad student at ORNL) working with Vickie Lynch will get previous code running on TeraGrid and test

• Jürg Hauser from University of Bern will provide a reference dataset with less noise

6 Managed by UT-Battellefor the U.S. Department of Energy

Original diffuse scattering code

• Used for publications:– Weber T.,Bürgi H.B., Determination and refinement of disordered crystal

structures using evolutionary algorithms in combination with Monte Carlo methods, Acta Crystallographica A 58, 526-540 (2002)

– H.B. Bürgi, J. Hauser, T. Weber, R.B. Neder, Supramolecular Architecture in a Disordered Perhydrotriphenylene Inclusion Compound from Diffuse X-ray Diffraction Data. Crystal Growth & Design, 5 2073-2083 (2005)

• Master/slave code was written mainly in Perl with telnet to communicate between workstations (Calls Fortran and C codes for FFTs)

• Much file communication between master and slave– Code assumed unique disk space for each slave

• One run for over 200 generations took ~30 days on ~10 workstations

7 Managed by UT-Battellefor the U.S. Department of Energy

VOP code by Phil Bentley

• Vitess Optimisation Program written for optimizing parameters in instrument simulations

• Genetic algorithm with tournament selection; Swarms

• Written in C++ using MPI for parallel computing (Allreduce to find min; Bcast to sent best parameters)

• Command line input for Vitess with parameters to be optimized preceded by @

• Limits file read by code

• Written to maximize flux from Vitess, but easily modified to minimize 2

8 Managed by UT-Battellefor the U.S. Department of Energy

Code port to TeraGrid

• Replaced master telnet code with VOP MPI code

• VOP calls Perl slaves with pipes

• Unique directories created for each slave

• Ported and ran successfully on NSTG

• Needed faster turnaround and 1048 clones

• Got Kraken Startup allocation

• Popen for Perl slaves did not work with CNL

9 Managed by UT-Battellefor the U.S. Department of Energy

Code port to Ranger

• Tested that popen worked on Ranger– Transferred part of Startup

to Ranger

• Too much file communication in Perl slaves for 16 cores/node

• Put OpenMP directives in FFT codes

• Using 4 slaves/node with 4 OpenMP threads/slave

10 Managed by UT-Battellefor the U.S. Department of Energy

VOP results with Perl slaves

11 Managed by UT-Battellefor the U.S. Department of Energy

Sinergia accomplishments

• Add differential evolution option to vop master

• Compared differential evolution, genetic algorithm and swarm for disordered crystal data. Genetic algorithm converged fasted, but each generation takes twice as long using tournament selection

• Added hybrid OpenMP directives

• Have code running on NSTG and Ranger

• Have Levenberg-Marquardt nonlinear least squares driver for slaves for further optimization

• Running using reference dataset on Ranger now (160 cores)

12 Managed by UT-Battellefor the U.S. Department of Energy

Future Sinergia work

• Install PGPLOT Perl module and helper scripts

• Testing with reference dataset– Number of individuals– Number of clones– Optimize mutation and crossover constants

• Scale to thousands of cores

• Automatically switch between genetic algorithms and least squares using history of chisq

• Write slave code in C++, not Perl

• Communicate between modules with memory, not files

13 Managed by UT-Battellefor the U.S. Department of Energy

Fitting Service

• Fast data fitting, the major and the most time consuming procedure of the data analysis, is the key of high performance data analysis and real-time data processing.

• Uses NL2SOL or Dakota to fit experimental data– http://www.ornl.teragrid.org/fitting/FitSrv_method.doc

• Instrument scientists writes functions – Read data– Model to fit data– http://www.ornl.teragrid.org/fitting/FitSrv_Portal_uInterface.pdf

14 Managed by UT-Battellefor the U.S. Department of Energy

Fitting Service results

• Fitting service used for experimental reflectometer data is shown

• NL2SOL results show 5 times smaller Chisq than previous fitting code used for this data

15 Managed by UT-Battellefor the U.S. Department of Energy

Fitting Service uses TeraGrid

• New fitting service– going into production– available from the

development portal– will be “monitored” for

health in the future

• Calculation scales linearly on TeraGrid on Mercury at NCSA

• See TG09 slides by Meili Chen for more details.

16 Managed by UT-Battellefor the U.S. Department of Energy

Fitting Service GUI in Portal

17 Managed by UT-Battellefor the U.S. Department of Energy

NSTG Cluster Transition to SNS

• Cluster: dual processor 3.06 GHz Intel Xeon nodes

• 14 nodes with 2.5GB of memory are compute nodes

• 4 nodes with 4GB memory are dedicated to GridFTP

• Transitioning to Scientific Linux 5.2

• Move planned from ORNL to SNS machine room (~ 3 miles)

• Cluster will link SNS with TeraGrid