Presented by

XGC: Gyrokinetic Particle Simulation of Edge Plasma

CPES TeamPhysics and Applied Math

Computational Science

2 Klasky_XGC_0611

Computational Science

California Institute of TechnologyJ. Cummings

Lawrence Berkeley National LaboratoryShoshani

Oak Ridge National LaboratoryR. Barreto, S. Klasky, P. Worley

Princeton Plasma Physics LaboratoryS. Ethier, E. Feibush

RutgersM. Parashar, D. Silver

University of California - DavisB. Ludäscher, N. Podhorszki

University Tennessee - KnoxvilleM. Beck

University of UtahS. Parker

CPES TeamPhysics and Applied Math

New York University Chang, Greengard, Ku, Park, Strauss, Weitzner, Zorin

Oak Ridge National LaboratorySchultz, D’Azevedo, Maingi

Princeton Plasma Physics LaboratoryHahm, Lee, Stotler, Wang, Zweben

Columbia UniversityAdams, Keyes

Lehigh University Bateman, Kritz, Pankin

University of ColoradoParker, Chen

University of California - Irvine Lin, Nishimura

Massachusetts Institute of TechnologySugiyama, Greenwald

Hinton AssociatesHinton

3 Klasky_XGC_0611

Physics in tokamak plasma edge Plasma turbulence

Turbulence suppression (H-mode)

Edge localized mode and ELM cycle

Density and temperature pedestal

Diverter and separatrix geometry

Plasma rotation

Neutral collision

ITER (www.iter.org)

Edge turbulence in NSTX (@ 100,000 frames/s)

Diverted magnetic field

4 Klasky_XGC_0611

XGC development roadmap

Black: Achieved • Blue: In progress • Red: To be developed

Full-f neoclassical ion root code (XGC-0)

Buildup of pedestal along ion root by neutral ionization

Full-f ion-electron electrostatic code (XGC-1)

- Whole edge

Neoclassical solution

Turbulence solution

Full-f electromagnetic code (XGC-2)

Study L-H transition

Multi-scale simulation of pedestal growth in H-mode

XGC-MHD coupling for pedestal-ELM cycle

5 Klasky_XGC_0611

XGC 1 code Particle-in-cell code

5-dimensional (3-D real space + 2-D velocity space)

Conserving plasma collisions (Monte Carlo)

Full-f ions, electrons, and neutrals

Gyrokinetic Poisson equation for neoclassical and turbulent electric field

PETSc library for Poisson solver

MPI for parallelization

Realistic magnetic geometry containing X-point

Particle source from neutral ionization

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

6 Klasky_XGC_0611

Routine Time % Peak Performance

Total 100% 6%

Poisson 7% ~1.5%

Pushing 37% ~8%

Charging 50% ~4%

Peak performance of XGC1 on Jaguar 131M ions and 131M electrons, 200K nodes

Peak performance with 2048 cores, using strong scaling results

Working with team members to increase peak performance to 18%

7 Klasky_XGC_0611

Scalability of XGC1 on Jaguar:Near linear scaling for strong, linear scaling for weak scaling

XGC Strong Scaling :131M ions and electrons, 200K grid

XGC Weak Scaling :50K ions and electrons/core

100 1,000 10,0001.E+06

1.E+07

1.E+08

1.E+09

Spee

d (#

of p

artic

le x

ste

p/s)

Number of Cores

1.E+04

1.E+05

1.E+06

1.E+07

100 1,000 10,000

Number of Cores

Spee

d (#

of p

artic

le/s

)

8 Klasky_XGC_0611

Neoclassical potential and flow of edge plasma from XGC1

Electric potential Parallel flow and particle positions

9 Klasky_XGC_0611

Phs-0: Simple coupling:

with M3D and NIMROD

XGC-0 grows pedestal along neoclassical root

MHD checks instability and crashes the pedestal

The same with XGC-1 and 2

Phs-2: Kinetic coupling:

MHD performs the crash

XGC supplies closure information to MHD during crash

Phs-3: Advanced coupling:

XGC performs the crash

M3D supplies the B crash information to XGC during the crash

XGC-MHD coupling plan

Black: Developed • Red: To be developed

10 Klasky_XGC_0611

Data replication

XGC-M3D code couplingCode coupling framework with Kepler-HPC

XGC on Cray XT3

End-to-end system 160p, M3D runs on 64PMonitoring routines here

Ubiquitous and transparent data access via logistical networking

User monitoring Data replication

Post-processing

40 Gb/s

Data

arch

iving

 

11 Klasky_XGC_0611

M3D equilibrium and linear simulationsnew equilibrium from eqdsk, XGC profiles

Equilibrium poloidal magnetic flux

Linear perturbed poloidal magnetic flux, n = 9

Linear perturbed electrostatic potential

12 Klasky_XGC_0611

Contact

Scott A. KlaskyLead, End-to-End SolutionsCenter for Computational Sciences(865) 241-9980klasky@ornl.gov

12 Klasky_XGC_0611