Tutorial: Computation for Advanced Accelerators

Thomas M. Antonsen Jr,

IREAP, University of Maryland

College Park MD, 20742

AAC 2012Work supported by NSF, and DOE

Computation has become integral to Accelerator Science

Why is Computation Important ?

- Understanding of basic physical processes- Understanding and diagnosing particular experiments- Designing improved experiments- Optimizing designs

Areas of Progress

Improved Physics Models

Find better descriptions of what’s going on inside a an accelerator

More Efficient Algorithms

Solve the governing equations with fewer operations and/or less memory usage.

New Computing Architectures

Restructure your code to take advantage of new hardware.

Examples of Areas where Computation is Important

Particle trajectories in analytic fields*Space charge fields* Magnetic fields due to currents in coils

RF fields in cavities and waveguidesRadiation generated by energetic particles Multipactor

Plasma-Based Accelerationand more

*See R. Ryne http://uspas.fnal.gov/materials/09UNM/ComputationalMethods.pdf

Meshes and Grids

Unstructured mesh

Structured mesh

Calculations are carried out on digital computersTime is discretized

Space may also be discretized - meshes or grids

If field is a dynamical variable (Maxwell’s Equations) a mesh is needed

If field describes pair-wise interaction forces (Coulombs law) it can be calculated with or without a grid.

Tree Code Calculation of Coulomb interaction without grids J. Barnes and P. Hut, “A hierarchical force-calculation algorithm,” Nature, vol. 324,

pp. 446–449, 1986

Direct application of Coulomb’s law to N particles requires O(N2) operations. Not practical for large N.

Tree code: Divide domain into cellsEach cell has fewer than NL particles

Particles within same cell interact pairwise

Particles in different cells interact through multipole moments

O(NlogN) operations possible

Mesh Types

Structured Mesh

Complex geometries may be represented using a multi-block representation:

Unstructured Mesh

conformal meshes: boundary of mesh conforms with physical boundary

Existing Discretization Techniques

Finite Integration TechniqueApplicable to orthogonal Cartesian meshDegrees of freedom are integrals of fields over appropriate mesh elements

Suffers due to non-conformal representation of geometry

Finite Element MethodApplicable to curvilinear mesh(structured or unstructured)

Field described by interpolating basis functions in each cell

Solves integral (weak) form of 2nd order differential equations

Evaluates matrix elements using overlap integrals of basis & test functions

Large number of coupling terms

Basis & test functions

1

-½ 0 ½

Fields and Structured Mesh for FDTD solution of Maxwell’s Equations

Electric fields defined on edgesMagnetic fields defined on faces Electric fields defined at integer time stepsMagnetic fields defined at half integer time steps

• Widely used• Simple (explicit)• Second order accurate• Stable if t <cdx

Yee Mesh(1966)

Leap Frog Explicit FDTDSolutions to Maxwell’s Equation

time differencing

Alternately apply Faraday’s law to faces

Explicit: knowing E at time t allows for direct calculation of H at time t+t/2

Finite Differencing Plane Waves

Assume all fields vary as

Dispersion relation for Plane waves

ContinuousFDTD

Stability - Courant Freidricks Lewy Condition (CFL)

Left side < Maximum of Right Side

Stable if

CFL Condition

Dispersion Relation

Dispersion and Accuracy

Dispersion in discrete system is periodic.

Departures from continuous system are quadratic in t, .

Waves have phase velocity <c

Numerical Cerenkov Radiation - Requires filter

More complicated evaluation of curl

Conducting Boundaries on Structured Grids

When boundary does not coincide with mesh FDTD must be modified.

Without modification errors of order are introduced.

Boundary

S. Dey, R. Mittra, A locally conformal finite-difference time-domain (FDTD) algorithm modeling modeling three-dimensional perfectly conducting objects, IEEE Microwave Guided Wave Lett. 7 (1997) 273

Problem: If cells are small CFL condition is violated

Separate modification to compute surface losses.

ADI (Alternating Dimension Implicit) Solutions to Maxwell’s Equation)

• Split time steps• Recently developed• Complex (implicit)• Not subject to Courant condition, t > cdx

time differencing

Second half of time step has x, y, and z permuted

Scalar: Douglas, Peaceman, and Ratchford (1955)

ME: Namiki T (1999), Zheng, Chen, Zhang (2000)

Leapfrog ADI-FDTD equationsS. Cooke et al (2009)

Electric and magnetic field terms are offset in timeElectric fields only appear on the half time stepsMagnetic fields only appear on the full time steps

Still need to solve six tri-diagonal equations per time step3 for the E-components, 3 for the H-componentsADI-FDTD uses 6 for the E-components (3 components, 2 half steps)

No additional explicit equations are necessary

Tri-diagonal operators “curl” terms

PIC Algorithm - From Plasma Physics via Computer Simulation C. K. Birdsall and A. B. Langdon

FDTD ME in here

Integration of Equations of Motion

Basic Leap Frog:

Modified for Lorenz Force:“Boris Push”

Half step with E

Full step with B

Half step with E

p and x known on staggered time grid

Interpolation of Fields and Accumulation of Currents and Charges

Forces interpolated from grid to location of particles

Particle contribution to current density accumulated on grid. Algorithm should conserve charge.

E and J are known on the same grid, but are staggered in time

J.Villasenor and O.Buneman. Computer Physics Communications, 69. 306-316, (1992)

Charge is not computed,only current is used.

Space charge fields occur only through integration of Maxwell’s equations.

Charge-conserving methodsdo this exactly – no need fordivergence-cleaning.

Advance H

Assign external current sources

Add new particles

Advance P, then X(accumulating J)

Remove dead particles

Advance E

HPEXJ

IVEC 2012 21

FDTD-PIC Time-stepping loop S. Cooke NRL

Plasma Based Accelerators

• Direct Acceleration in Modulated Channels

Physical Processes

Important

• Relativistic Motion

• Plasma Wave Generation / Cavitation

• Laser Self Focusing / Scattering

• Ionization

Not Important

• Turbulence - most plasma particles interact for a short time, several plasma periods, then leave

Our job is “relatively” easy.

Simulation has played an important role in the development of the field.

Guiding our understanding Designing new experiments

Relevant Time Scales (LWFA)

• Laser Period =800 nm TL = 2.7 fs

• Driver Duration ~ Plasma Period TD = 50 fs

• Propagation Time (driver evolution, L= 1 cm) TP = 3.3 104 fs

(driver evolution, L= 1 m) TP = 3.3 106 fs

• Disparity in time scales leads to both complications and simplifications

Propagation Time >> Driver Duration >> Laser Period

TP >> TD >> TL

Propagation Time >> Driver Duration >> Laser Period

TP >> TD >> TL

MODELS APPROACHES / APPROXIMATIONS

• LaserFull EM - Laser Envelope

• Plasma

Particles - Fluid

Full Lorenz force - Ponderomotive

Dynamic response - Quasi-static

Hierarchy of Descriptions

1. Full Format Particle in Cell (PIC)

- Relativistic equations of motion for macro-particles- Maxwell’s Equations on a grid- Most accurate but most computationally intensive

• LaserFull EM - Laser Envelope

• Plasma

Particles - Fluid

Full Lorenz force - Ponderomotive

Dynamic response - Quasi-static

Windows and Frames

z=0 z=L

t=L/cLab Frame - # Grids

NLF = (L/dz)2 ~ (L/)2

Moving Window - # Grids

NMW = (LD/L) NLF

grid must resolve

Three versions:- Lab Frame- Moving Window- Boosted Frame

- LD Propagation distancePulse length

dt=dz/c

Boosted FrameJ-L Vay, PRL98, 130405 (2007)

z=0 z=L

t=L/c

Boosted frame - # grids NBF = -2 NMW

Optimum based on 1Dmakes simulation “square”

Boost -

z'=0z' =-LD

t' =L/c

dz' = dz

Qualifications:-no backscatter. In boosted frame gives upshift. Requires smaller dz'-transverse grid size sets limit on dt' < dx' /c= dx /c

Lab Framedt' = dt

Reported speedups limited by various factors: • laser transverse size at injection, • statistics (trapped injection),• short wavelength instability (most severe).

Osiris: trapped injection

Vorpal: external injection w/ beam loading

Warp: external injection wo/ beam loading

Using conventional PIC techniques, 2-3 orders of magnitude speedup reported in 2D/3D by various groups

J-L Vay

2. Full EM vs. Laser Envelope Driver Duration >> Laser Period

TD >> TL

• Required Approximation for Laser envelope:

laser pulse >> 1, rspot >>

p /laser <<1

• Advantages of envelope model:

-Larger time steps

Full EM stability: t < x/c

Envelope accuracy: t < 2 x2/c

- No unphysical Cherenkov radiation

- Further approximations

• Advantages of full EM: Includes Stimulated Raman back-scattering

Also direct acceleration

Laser Envelope Approximation

• Laser Frame Coordinate: = ct – z

• Laser + Wake field: E = E laser + E wake

• Vector Potential: Alaser = A 0(,x,t) exp ik 0 + c.c.

• Envelope equation:

Necessary for: Raman ForwardSelf phase modulationvg< c

Drop(eliminates Raman back-scatter)

Validity of Envelope Equation

Extended Paraxial approximation- Correct treatment of forward and near forward scattered radiation - Does not treat backscattered radiation- If grid is dense enough can treat ~

Extended Paraxial– v g() = c / 1 +

k2c2 + p

2

2 2

True: v g() = c 1 –

k2 c2 + p

2

2

1 / 2

k2c2 , p

2 < < 2 Requires :

3. Full Lorenz Force vs. Ponderomotive Description

• Ponderomotive Equations

dpdt

= q Ewake +v Bwake

c + Fp

Fp = – mc 2

2

q A laser

mc 2

2

= 1 +

p 2

m2c2 +q A laser

mc 2

2

dpi

dt= q E +

vi Bc

dxi

dt= vi

• Full Lorenz:

E = Elaser + Ewake

x(t) = x(t) + x(t)

• Separation of time scales

x(t) Elaser < < Elaser

• Requires small excursion

Ponderomotive Guiding Center PIC Code: TurboWAVE

Fields are separated into high and low frequency components. The low frequency component is treated as in an ordinary PIC code. The high frequency component is treated using a reduced description which averages over optical cycles.

Deposit Sources j and

Advance Fields(Maxwell’s Equations)

Lorentz Push

Deposit Sourcenq2/<m>

Advance Laser(Envelope Equation)

Ponderomotive Push

+

Low Frequency Cycle High Frequency Cycle

D.F. Gordon, et al. , IEEE TRANSACTIONS ON PLASMA SCIENCE , V 28 , 1224-1232 ( 2000 )

35

INF&RNO(INtegrated Fluid & paRticle simulatioN cOde)

C. Benedetti at al. (LBNL)

2D cylindrical + envelope for the laser (ponderomotive approximation) full PIC/fluid description for plasma particle (quasi-static approx. is also available) switching between PIC/fluid modalities (hybrid PIC-fluid sim. are possible) dynamical particle resampling to reduce on-axis noise 2nd order upwind/centered FD schemes + RK2/RK4 (& Implicit) for time integration linear/quadratic shape functions for force interpolation/charge deposition high order low-pass compact filter for current/field smoothing “BELLA”-like runs (10GeV in ~ 1m) become feasible in a few days on small machines

FLUID PIC

4. Quasi - Static vs. Dynamic WakeP. Sprangle, E. Esarey and A. Ting, Phys. Rev. Lett. 64,

2011 (1990)

ddt

= t

+ c – vz

+ v

Laser Pulse PlasmaWake

Plasma electron

c

Trapped electron

c - vz

Electron transit time:

e = pulse

1 – vz / c

Electron transit time << Pulse modification time

Advantages: fewer particles, less noise (particles marched in ct-z)

Disadvantages: particles are not trapped

Quasi Static Simulation Code WAKE P. Mora and T. M. Antonsen Jr. - Phys Plasma 4, 217 (1997)

Particle trajectories

r

=ct-z

Two Time Scales1. “fast time” t ~ TD ~ p

-1

particle trajectories and wake fields determined

2. “slow time”

laser pulse evolvesdiffractionself-focusingdepletion

QuickPIC: 3D quasi-static particle code

New Features• Particle tracking• Pipelining• Parallel scaling to 1,000+

processors• Beta version of enhanced

pipelining algorithm: enables scaling to 10,000+ processors and unprecedented simulation resolution down to nm

Description

• Massivelly Parallel, 3D Quasi-static particle-in-cell code

• Ponderomotive guiding center for laser driver

• 100-1000+ savings with high fidelity• Field ionization and radiation reaction

included• Simplified version used for e-cloud

modeling• Developed by UCLA + UMaryland + IST

Examples of applications• Simulations for PWFA experiments,

E157/162/164/164X/167 (Including Feb. 2007 Nature)

• Study of electron cloud effect in LHC.• Plasma afterburner design up to TeV• Efficient simulation of externally injected

LWFA• Beam loading studies using laser/beam

drivers

Chengkun Huang: huangck@ee.ucla.eduhttp://exodus.physics.ucla.edu/http://cfp.ist.utl.pt/golp/epp

QUASI-STATIC CODE STRUCTURE

Laser

Particles and Wake

“slow time - t”

“fast time =ct-z”

PARTICLES CONTINUEDconstant of motion

• Hamiltonian::

• Algebraic equation:

• Weak dependence on “t” in the laser frame

• Introduce potentialsTransverse Dynamics

LOCAL FREQUENCY MODIFICATIONM. Tsoufras, PhD Thesis, UCLA

neIntensity

Frequency shift is proportional to propagation distance

Relative frequency shift is unity at the dephasing distance

Assume complete cavitation

Quasi Static Modeling of Pulse DepletionW. Zhu Wednesday WG1

Comparison of Spectra; Modified Paraxial, Full-wave, and FDTD-PIC

Conclusions

• Numerical simulation of Laser-Plasma interactions is a powerful tool

• A variety of models and algorithms existfirst principlesreduced modles

• Field is still advancing with new developmentsBoosted Frame Calculations speed-up ~ 2

3D Parallel Quasi-static speed-up ~ [p2

GPUs

WAKE - CavitationP. Mora and T. Antonsen PHYSICAL REVIEW E, Volume: 53 R2068

(1996)

Intensity Density Trajectories

Complete cavitationSuppression of Raman instability Stable propagation for 30 Rayleigh lengths

Wake simulation of pulse Wake simulation of pulse propagation in corrugated plasma channelspropagation in corrugated plasma channelsSee WG #1 B. Layer Wed. PM, J. Palastro Thu. AM

150 m

3 cm ~ 90 corrugations ~ 40 TR

t=0 TR t=15 TR t=30 TRUpper: corrugatedLower: smooth

Suppression of Raman Instability

density

intensity

=800 nmwch=15 mao=.35t=1 ps

no=7x1018 cm-3

=.9m=.035 cm

QUICKPIC

Quasi-Static Plasma Particles

Dynamic Beam Particles

Wake Fields

Laser Field

Axial (dt) Axial (dt)Transverse (dt) Transverse (dt)

LaserAdvance

Axial (dt/2)

push particlespush particlesPlasma Particle Advance

time-st t+dt

Second Order Accurate Split-Step Scheme

time-tt t+dt

e- driver e+ driver

e- driver with ionization laser driver

Verification : Full PIC vs. Quasi-static PICBenchmark for different drivers:

QuickPIC vs. Full PIC

100 to 10000 CPU savings with “no” loss in accuracy

Excellent agreement with full PIC.

100 to 10000 times savings in CPU needs

No noise issues and no unphysical Cerenkov radiation

Iteration of Electromagnetic Field

Parallel electric field

Transverse electric field

Electromagnetic portion

Ampere’s law

Iterate to find

Equation of motion

Quasi-Static Field Representation

Lorenz

Pro:

Simple structure

Compatible with 2D PIC

Con:

A carries “electrostatic” field

Transverse Coulomb

Pro:

A = 0 in electrostatic limit

Con:

non-standard field equations

Particle Promotion in WAKES. Morshed PoP, to be published

Osiris

Wake

“Plasma particles” for which quasi-static violated are promoted to “beam particles”

54

Combine a half step forward with a half step back:

Subtracting equations gives a new scheme:

New “Leapfrog” ADI-FDTD Formulation

Tri-diagonal operator “curl H” term

Compare to FDTD:

S.J.Cooke, M.Botton, T.M.Antonsen, Jr., B.Levush, “A leapfrog formulation of the 3-D ADI-FDTD algorithm” Int. J. Numer. Model. 2009; 22:187–200

Simple source terms

TurboWAVE Framework

TurboWAVE BaseGrid geometry, regions, domain decomposition, diagnostics, input/output…

Numerical FrameworkLinear algebra, elliptical solvers, fluid advection…

PICEM pusher,

ponderomotive pusher, gather/scatter

“SPARC”Multi-species

hydrodynamics

Nonlinear Optics

Anharmonic Lorentz model, Quasistatic modelField Solver Modules

Explicit, envelope, direct fields, coulomb gauge, electrostatic.

Lindman boundaries, simple conducting regions, PML media.

ChemistryArbitrary species,

reactions

NRL Plasma Physics Division

Images from TurboWAVE

Gridded Gun ModelingContours are equipotential lines.Colors represent current density.Black circles represent grid wires.

Blowout WakefieldRed = ion richBlue = electron rich

ve

k0

Utilizes 2D electrostatic PIC (slab or cylindrical) with conducting regions

Utilizes 3D electromagnetic PIC

NRL Plasma Physics Division

Speedup of 50,000 shown for meter-scale parameters

Rigorously tested, showing second-order convergence and correct laser group velocity

Benchmarks against scaled simulations show excellent agreement

Envelope model in VORPAL enables full 3D simulations of meter-scale LPA stages

[1] B. Cowan et al., Proc. AAC 2008

[2] P. Messmer and D. Bruhwiler, PRST-AB 9, 031302 (2006)

[3] D. Gordon, IEEE Trans. Plasma Sci. 35, 1486 (2007)

[4] P. Mora and T. M. Antonsen, Phys. Plasmas 4, 217 (1997)

Ben Cowan

Charge Conserving Algorithm

Full Format: PIC Algorithm

VORPAL

OSIRIS 2.0

New Features in v2.0• Bessel Beams • Binary Collision Module• Tunnel (ADK) and Impact

Ionization• Dynamic Load Balancing• PML absorbing BC• Optimized Higher Order

Splines• Parallel I/O (HDF5)• Boosted Frame in 1/2/3D

osiris framework• Massivelly Parallel, Fully Relativistic

Particle-in-Cell (PIC) Code • Visualization and Data Analysis

Infrastructure• Developed by the osiris.consortium:

UCLA + IST• Widely used: UCLA, SLAC, USC,

Michigan, Rochester, IST, Imperial College, Max Planck Inst.

• Examples of applications• Mangles et al., Nature 431 529 (2004).• Tsung et al., Phys. Rev. Lett., 94

185002 (2004)• Mangles et al., Phys. Rev. Lett., 96,

215001 (2006)• Lu et al., Phys. Rev. ST: AB, 10, 061301

(2007)• I.A. Blumenfeld et al., Nature 445, 241

(2007)

Frank Tsung: tsung@physics.ucla.eduRicardo Fonseca: ricardo.fonseca@ist.utl.pthttp://exodus.physics.ucla.eduhttp://cfp.ist.utl.pt/golp/epp

~300x fasterthan lab

simulation

+10GeV self-injection in nonlinear regimeControlled self-guided a0=5.8

Laser pulse

Laser pulse

Injected electrons

Injected electrons

Smooth accelerating

field

Smooth accelerating

field

Boosted frame7000x256x256

cells ~109 particles

3x104 timestepsΥ=10

7-12 GeV1-2 nC

7-12 GeV1-2 nC

Samuel F. Martins et al. Nature Physics V6, April 2010Nature Physics V6, April 2010