T7/High Performance Computing

K. Ko, R. Ryne, P. Spentzouris

2

Accelerators are Crucial to Scientific Discoveries in High Energy Physics, Nuclear Physics, Materials Science, Biological Science

“Starting this fall, a machine called RHIC will collide gold nuclei with such force that they will melt into their

primordial building blocks”

“A new generation of accelerators capable of generating beams of exotic radioactive nuclei aims to simulate the element-building process in stars and shed light on nuclear structure”

“Biologists and other researchers are lining up at synchrotrons to probe materials and molecules with hard

x-rays”

“Muon Experiment Challenges Reigning Model of Particles”

“Violated particles reveal quirks of antimatter”

3

Contributions of accelerators to applied science and tech have huge economic impact and greatly benefit society

• Medical isotope production

• Electron microscopy

• Accelerator mass spectrometry

• Medical irradiation therapy

• Ion implantation

• Beam lithography

• Transmutation of waste

• Accelerator-driven energy production

Particle accelerators are among the largest, most complex, and most important scientific

instruments in the world

Given the complexity and importance of accelerators, it is imperative that the most advanced HPC tools be brought to bear

on their design, optimization, commissioning, and operation

Particle accelerators helped enable some of the most remarkable discoveries of the 20th century (“century of physics”)

We may now be on the brink of revolutionary advances in particle physics that will change how we view the universe

5

Objective of HPC Working Group

• …understand the modeling needs of current and future accelerator technology

• …identify HPC hardware and software technologies required for such modeling

• …outline a plan for the development of these technologies

6

T7 Sessions

• Dedicated HPC tools HPC simulation of Beam Systems HPC simulation of Electromagnetic Systems

• Joint Advanced Accelerators Beam Dynamics Proton Drivers Muon-based Systems Particle Sources

HPC needs of present and next-generation accelerators

8

High-Intensity Proton Drivers

• Present codes inadequate for full 3D modeling Extrapolation predicts ~year long jobs on 128 processors Good progress from SNS effort

• Important physics 3D space charge Long bunches Impedances e-p instability

• Code development underway

• Code comparison effort

• Experiments TBD

9

Linear Colliders

• Electromagnetic modeling NLC

– Complicated 3D structures– High accuracy, large problems, unstructured grids

• Beam Dynamics– Space-charge and collective effects in damping rings– Linac & beam delivery simulations needed with full bunch train(?)

including component fluctuations + beam tuning and feedback systems

– Damping ring IP full simulation expected to take ~1yr/processor

10

VLHC

• HPC simulation of instabilities Electron-cloud Resistive wall, TMCI

• HPC simulation of beam-beam effects

• Dynamic aperture

• Closed-orbit correction

• Energy deposition

• Full system simulations including component fluctuations + feedback

11

Muon-Based Systems

• Ionization cooling Accurate modeling of muon/matter

interactions Energy loss, multiple scattering Typically done with portions of HEP

packages (need to parallelize) Need for code interfacing System optimization involving HPC

software components Space-charge effects in rings

[we should be so lucky ]

1PE: 500 particles, 460 sec

128 PE: 500,000 particles, 4400 sec

12

• Space charge Particle & field managers Direct Vlasov

• Beam-beam

• Electron-cloud Full dynamics of beam and electrons

• CSR

• Impedances/wakes

• Multi-bunch effects (pipeline model)

• Collisions First-principles Fokker/Planck

Summary: Application of HPC to Beam Phenomena

13

Application of HPC to Electromagnetic Modeling

• Eigenmode Complicated 3D structures High-accuracy (individual cells) Very large scale (EM system simulation) Unstructured grids, micron-scale variation

• Time-domain Frequency response

• Direct wakefield calcuations• Additional phenomena (surface physics,…)• Particle dynamics (dark current,…)• Trapped modes• Structure heating

14

HPC Simulation of Advanced Accelerator Concepts

• Multiple models Fully explicit 3D Vlasov/Maxwell Ponderomotive, quasi-static,…

• Moving windows• Multiple species• Ionization• Multiple scales (lasers)

• CPU estimates >= 100,000 hrs for a GeV plasma accelerator stage

Laser off Laser on

PIC Simulation

Experiment

(a) (b)

(c) (d)

SciDAC(Scientific Discovery through Advanced

Computing)

Includes DOE/HENP project to develop a comprehensive terascale capability

in accelerator simulation

DOE Office of Science initiative in advanced computing

16

HPC Tools

• David Keyes, TOPS (Terascale Optimal PDE Solvers)

• Lori Freitag, TSTT (Terascale Simulation Tools & Tech)

• Phil Colella, Adaptive Mesh Refinement for PDEs

• Viktor Decyk, Parallel Particle Simulation

• Esmond Ng, Parallel Linear Algebra & Eigensolvers

• Horst Simon, Future architectures

• Performance Enhancement (ATLAS,…)

• Code integration (Common Component Architecture,…)

• Visualization

17

Theory+Experiment+Computation

Physicalsystem

Physicalmodel

Mathematicaldescription

Algorithms

Softwareimplementation

{{Theory

Computation(HPC)

PhysicalExperiment

(diagnostics!)

Verification& Validation

ComputerSimulation

18

• Present accelerators: Maximize investment by optimizing performance expanding operational envelopes increasing reliability and availability

• Next-generation accelerators better designs feasibility studies Facilitate important design decisions completion on schedule and within budget

• Accelerator science and technology help develop new methods of acceleration explore beams under extreme conditions

Summary: HPC will play a major role

19

Let’s get back on the Livingston curve