Status of PY-ORBIT: Benchmarking and Noise Control in PIC ... · Python ORBIT (PY-ORBIT) •...

Status of PY-ORBIT: Benchmarking and Noise Control in PIC Codes

J. Holmes, S. Cousineau, and A. Shishlo

54th ICFA Beam Dynamics Workshop HB2014 Michigan State University November 12, 2014

Python ORBIT (PY-ORBIT) •  PY-ORBIT is a collection of computational beam dynamics models for accelerators, designed to work

together in a common framework.

•  It was started as a “friendly” version of the ORBIT Code, written using publicly available supported software.

–  Users interface with code via Python scripts.

–  Computationally intensive code in C++ (mostly, PTC is in Fortran).

–  Wrappers make C++ routines available to users at Python level.

–  Uses MPI for multiprocessing.

•  PY-ORBIT source code is publicly available via Google Codes:

–  svn checkout https://py-orbit.googlecode.com/svn/trunk py-orbit --username [email protected]

–  It is not difficult to develop your own extensions to PY-ORBIT. We welcome responsible participation in developing PY-ORBIT models.

•  PY-ORBIT people at present:

–  Owners: Andrei Shishlo (ORNL), Sarah Cousineau (ORNL), Jeff Holmes (ORNL)

–  Committers: Sabrina Appel (GSI), Oliver Boine-Frankenheim (GSI), Hannes Bartosik (CERN), Timofey Gorlov (ORNL)

–  Additional user: Sasha Molodozhentsev (KEK)

What Does PY-ORBIT Do? •  Most things that ORBIT does for rings and transfer lines

–  Single particle tracking •  Native symplectic tracker •  PTC tracking •  3D field tracker •  Linear tracking with matrices (coming)

–  Space charge •  Longitudinal •  2D potential and direct force •  Full 3D (not parallel) and 3D ellipses (mostly for linacs and transfer lines) •  2.5D (Recently added by Hannes Bartosik)

–  Impedances •  Longitudinal •  Transverse dipole (partially completed)

–  Injection, painting, RF cavities, collimation, apertures •  Linac Modeling

–  RF cavities –  Magnets –  Full 3D space charge (not parallel) and 3D ellipses

•  Laser Stripping, Nonlinear Optics, Electron Cloud

Appealing Features •  Entire source code is available. Uses standard Python, C++, and

Fortran (PTC). Extra libraries only for FFTs and PTC. •  User is free to develop specialized or extended models to suit

his/her own needs. •  With permission of owners, users’ models can be incorporated

into public version. •  Many examples demonstrate use of models in scripts. Some

documentation in Google Code wikis. •  Bunch class is easily extendable:

–  Basic bunch has macroparticle coordinates in 6D. –  User can add various properties:

•  Particle ID tag •  Spin •  Species, ionization number, excited state, etc.

Remaining Issues •  Code is not yet complete. Missing:

–  Linear and second order tracking with matrices (coming)

–  Transverse dipole impedance (in progress) –  Alignment errors

–  Electron Cloud –  Other?

•  Documentation is incomplete: –  Although lots of examples illustrate use of modules and making of scripts, –  Some models are documented in Google Code wikis, but the rest needs to be

done.

•  No “professional” full-time support: –  PY-ORBIT development has been carried out by working accelerator physicists

(owners) trying to model and solve problems.

–  We all have to work at our “day jobs”.

6 Managed by UT-Battelle for the U.S. Department of Energy

Space Charge Models for Long Times •  Space charge physics has been successfully incorporated into computational

particle tracking studies of linacs, transfer lines, accumulator rings, and rapid cycling synchrotrons.

•  Computations for these machines all involve tracking particles on short to moderate time scales.

•  With the advent of higher beam intensities, calculations incorporating space charge effects must now be undertaken for storage rings.

•  Storage ring calculations require following beams for far longer times than are necessary for linacs, accumulator rings, or rapid cycling synchrotrons. –  This will place more severe requirements on the speed and accuracy of the

physics models. –  Progress for modeling collective effects, such as space charge, is necessary.

•  Issues in choosing a model: –  Representation of beam –  Numerical properties (discretization) –  Computational requirements vs available resources


State of the Art a Decade Ago S. Cousineau, J. A. Holmes, J. Galambos, A. Fedotov, J. Wei, and R. Macek, Physical Review Special Topics – Accelerators and Beams 6, (2003) 074202 .

transport of beams in rings, and uses MAD files to createthe accelerator lattice. The code features sophisticatedinjection painting capabilities, rf cavity models, spacecharge and impedance modeling, and a complete set ofdiagnostics. The details of the code are given in Ref. [13],and a good description of the space charge implementa-tion is given in Refs. [12,14]. In the present simulations, anumber of effects are ignored, including strip-foil scat-tering, magnet errors, chromatic effects, transverse im-pedances, and higher order magnetic fields. Therefore,space charge and longitudinal impedance provide theonly nonlinearities.

For all of the simulations performed, 50 macroparticleswere injected on every turn, resulting in approximately1:6! 105 macroparticles by the end of accumulation. Afast-Fourier transform (FFT) method on a 64! 64 rect-angular grid was used to calculate the nonlinear trans-verse space charge of the beam [14]. Transverse spacecharge kicks numbering 371 were applied per turn of thebeam, yielding an average integration step size of 24 cm.The step size, grid size, and number of macroparticleshave been optimized for numerical convergence [15].Because the synchrotron motion in the PSR beam isvery slow, the longitudinal space charge is representedby a single space charge kick per turn. The beam isdistributed longitudinally over 64 bins, and a combinedspace charge and impedance algorithm adopted fromMacLachlan [16] is applied.

Figure 1 shows an example of a benchmark for theexperiment presented in Table I. The experimental verti-cal beam profile recorded on the wire scanner is shownalong with the simulated beam profile at the same point inthe extraction channel. The centers of the profiles werealigned, and both profiles were normalized to the sametotal beam intensity; neither the shape nor the width of

either profile was altered. The agreement between theexperimental and simulated curves is very good, withabout 3% difference in the width of the beams. Figure 1also includes, for comparison, a simulated profile gener-ated without space charge. The profile shape for this lattercase is determined exclusively by the painting schemeemployed, and a large difference from the experimentalmeasurement is evident. Including the space charge inter-actions of the beam is therefore critical for accuratemodeling of the experiment.

Figure 2 shows the longitudinal benchmark for thesame experiment, where again the centroids of the pro-files have been aligned and the vertical axes normalizedto the integrated beam intensity. An inductive insert inthe PSR lattice is modeled using a longitudinal imped-ance algorithm, implemented in conjunction with thelongitudinal space charge routine. The profile simulatedwith the complete longitudinal package (space charge andimpedance) is in close agreement with the experimentalprofile. Also shown in the figure is a simulated profilegenerated without the impedance of the inductive insert.The profile underestimates the amount of central peakingin the experimental measurement. Thus it is important toinclude the relevant longitudinal nonlinearities (imped-ance and space charge) in the simulation.

The success of these benchmarks lends credibility tothe ORBIT code as a tool that can accurately portray therelevant physics in the high intensity PSR beam. Theresults of ORBIT simulations will now be used extensivelyto probe the parameter space of the beam not availablethrough experimental measurement. We hereafter refrainfrom presenting benchmarks of the simulations, as thistopic is covered quite thoroughly in Ref. [12]; however, itshould be noted that for all experiments discussed, the

-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

-40 -30 -20 -10 0 10 20 30 40

Arb

itrar

y U

nits

Y (mm)

FIG. 1. (Color) Benchmark of vertical experimental profilemeasurements with PIC simulations. The solid (red) curve isthe experimental profile. The dotted (blue) curve is the PICresult with space charge. The (pink) dashed line is the PICresult without space charge.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

-4 -3 -2 -1 0 1 2 3 4

Arb

itrar

y U

nits

! (rad)

FIG. 2. (Color) Benchmark of longitudinal profile measure-ments with PIC simulations. The solid (red) curve is theexperimental profile. The dashed (blue) curve is the PIC resultwith longitudinal space charge and impedance. The (pink)dotted line is the PIC result with longitudinal space chargebut without other impedance.

PRST-AB 6 S. COUSINEAU et al. 074202 (2003)

074202-3 074202-3Transverse profiles: experiment and simulation with and without space charge in PSR ring.

Simulation, no space charge

Simulation, space charge

Experiment


Choice of Model •  Do we calculate space charge from assumed simple beam distribution or directly from tracked

particle bunch?

•  Representation of beam –  Assumed simple distributions will miss details, such as real beam profiles and nonuniformites, and cannot accurately depict

many processes, such as injection into accumulator rings. –  Direct use of the tracked macroparticle bunch can, in principle, faithfully represent real beam distributions.

•  Numerical Properties –  With assumed simple distributions, applied forces are smooth and collisions eliminated. Numerical noise comes from time

discretization only. –  Direct use of the macroparticle distribution for space charge force calculation involves discretization errors in space and

time that lead to numerical diffusion and emittance growth. The implications of controlling this will be discussed.

•  Computational requirements –  Forces from assumed simple distributions can be evaluated quickly, and it may be necessary to track relatively few particles.

–  Because of numerical issues, the direct use of macroparticles is typically more demanding on computer resources. Even so, in previous work on short to medium time scales, accurate calculations could be performed on workstations and small clusters in reasonable time.

•  Therefore, methods of choice have used the tracked distribution directly to obtain the force. –  Particle-in-cell (PIC) and fast multipole (FMM) methods are of order O(~M), where M is the macroparticle number. –  Methods come in various dimensionality (1D longitudinal, 2D transverse, 2.5D transverse and optionally longitudinal, or full

3D). –  Complexity increases with dimension. Use simplest model containing desired physics.


Bunch-Based Forces in Time •  2D or 2.5D bunch-based models are appropriate candidates for long time scale

calculations (storage rings). Valid for long bunches LBunch >> rBeamPipe. •  Most calculations use FFT-based PIC methods.

–  Discretization errors due to time, coarseness of distribution , gridding.

•  An alternative is the Fast Multipole Method (FMM) –  Discretization errors due to time, coarseness of distribution, but not gridding.

–  Accurate in principle to machine precision.

•  Based on the above properties, we programmed and tested fast multipoles.

•  References on numerical solution of Poisson’s equation: –  R. W. Hockney and J. W. Eastwood, “Computer Simulation Using Particles”, Institute of

Physics Publishing, Bristol, 1988.

–  J. Demmel, NSF-CBMS Short Course on Parallel Numerical Linear Algebra, especially Lectures 24-27, http://www.cs.berkeley.edu/~demmel/cs267-1995/

–  L. Greengard and V. Rokhlin, “A Fast Algorithm for Particle Simulations”, Journal of Computational Physics 73, (1987), 325.


Fast Multipoles: The Method •  Create series of levels, each with grid of 2n×2n square cells, n = N->2.

•  Grids of successive levels align. •  Each cell of level n < N contains 4 child cells from n+1 level.

•  Grid of cells must cover beam and boundary (if specified). •  At finest level, N, bin macroparticle charges to center of containing cell.

•  Expand force/potential of each charge as series about center. •  Can keep as many terms as necessary – machine precision in principle. •  Series converges outside of cell corner radii.

•  Successively shift multipole expansions to centers of parent cells. •  n = N-1 -> 2. •  Accumulated, shifted multipole series converge outside of parent cells.

•  Expand multipole series from non-adjacent parent cells as Taylor series about center of cells.

•  Again, can keep as many terms as necessary – machine precision in principle. •  n = 3 -> N. •  Pick up contributions of non-adjacent sibling cells (same level) that were missed.

•  At finest level, N: •  Apply Taylor series kicks to macroparticles. •  Sum force kicks from same and adjacent cell macroparticles pairwise. Smoothing?!

•  Treat boundary •  Calculate beam potential at boundary points as sum of Taylor series and adjacent

Cell macroparticles. •  Use method of Fred Jones (ACCSIM) to add kicks due to conducting wall BC.

•  Parameters: N levels, K terms, eps (smoothing parameter for pairwise). •  Others: boundary, longitudinal discretization, etc.


Fast Multipoles vs FFT

No smoothing -> pairwise collisions. Smaller time steps, more macroparticles -> increase computer time.

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

y-pr

ime

{mra

dian

}

x-prime {mradian}

FMM, eps = 1e-06FFT - 128x128



Some pairwise smoothing -> less Scattering agreement with FFT.

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

y-pr

ime

{mra

dian

}

x-prime {mradian}

FMM, eps = 3e-02FFT - 128x128



RMS emmitance: FFT and FFM with varied smoothing.

0.998

1

1.002

1.004

1.006

1.008

1.01

0 20 40 60 80 100

RM

S Em

ittan

ce {m

m-m

radi

an}

Turn

FMM, eps = 1e-06FMM, eps = 3e-02

FFT - 128x128



Property FFT FMM Numerical order (work) n n

Poisson solution Grid and interpolate Up to machine precision

Poisson domain noise Particle distribution Grid size

Interpolation scheme

Particle distribution Expansion order

Binary smoothing parameter

Longitudinal noise Grid size Interpolation scheme Particle distribution

Grid size Interpolation scheme Particle distribution

Time step discretization noise

Yes Yes Anomalous pairwise forces

Based on initial experiments with 105 macroparticles, FMM takes anywhere from 10-50 times more computer time, depending on the number of levels and order of expansion, than FFT with 128×128 cells.


Numerical Discretization Effects •  Discretization in time and graininess of the charge/current distribution -> diffusion.

•  Spatial meshes and smoothing also effect discretization.

•  Boine-Frankenheim, Hofmann, Struckmeier, and Appel studied numerically generated collisions and emittance growth in 2D space charge calculations for FODO lattices.

–  Based on earlier theoretical studies using Fokker-Planck analysis.

–  Derived an empirical scaling law for collisionality, or entropy or emittance growth, in space charge calculations.

•  ν = collision frequency, N = beam intensity (number of real particles), M = number of macroparticles, Δ = smoothing size (grid spacing), λ = cutoff scale length.

•  Numerical entropy growth is proportional to beam intensity squared, is inversely proportional to number of macroparticles, and is decreased by smoothing.

•  References on collisions and numerically-induced emittance growth: –  Jurgen Struckmeier, Phys. Rev. E 54, 830, (1996).

–  O. Boine-Frankenheim, I. Hofmann, J. Struckmeier, S. Appel, Nucl. Instr. and Meth. in Phys.Research A., in press.

ν ∝N 2

M× (1− Δ

λ)


Observations of the Empirical Scaling

•  Numerical diffusion in PIC codes leads to emittance growth in time. The growth is dependent on the number of macroparticles and the grid spacing.

100K particles, no space charge and space charge with various grids.

256x256 grid and various numbers of particles.

0.9995

1

1.0005

1.001

1.0015

1.002

1.0025

0 100 200 300 400 500 600 700 800 900 1000

RM

S Em

ittan

ce {m

m-m

rad}

Turn

Ex, 100K particles, 256 x 256 GridEx, 100K particles, 64 x 64 Grid

0.9995

1

1.0005

1.001

1.0015

1.002

1.0025

0 100 200 300 400 500 600 700 800 900 1000R

MS

Emitt

ance

{mm

-mra

d}Turn


Ex, 1M particles, 256 x 256 Grid


What to Do to Reduce Collisions? •  Refinement: More macroparticles, smaller time steps…

–  Feasible to some extent with the help of modern computers: large clusters, supercomputers, GPUs, etc.

–  Still expensive and “brute force”.

•  Reduce discretization effects –  For FFT solvers, try different binning or smoothing algorithms for distributing space charge

from the numerical particle distributions to the grid. Like low pass filters, can alleviate grid and distribution discretization effects.

–  Simplified or analytic distributions, based on statistically calculated parameters of the macroparticle distribution, provide another type of low pass filter to reduce grid and distribution discretization effects. Such methods can be used to evaluate space charge forces quickly.

–  Questions for the latter approaches: How much physics is lost in the simplification? How important is that physics? To what problems are analytic approaches applicable.

•  We have just begun exploring these approaches in the ORBIT Code.

MEBT 38 mA. PARMILA vs. pyORBIT

0 50 100 150 200 250 300 350 4000.0

0.1

0.2

0.3

0.4

0.5

X rm

s, c

m

pos(cm)

X rms, Parmila 3D PICNIC X rms, Parmila SCHEFF X rms, pyORBIT

Horizontal Beam Sizes in SNS MEBT. Space Charge 38 mA.

0 50 100 150 200 250 300 350 4000.0

0.1

0.2

0.3

0.4

Y rm

s, c

m

pos(cm)

Y rms, Parmila 3D PICNIC Y rms, Parmila SCHEFF Y rms, pyORBIT

Vertical Beam Sizes in SNS MEBT. Space Charge 38 mA.

-50 0 50 100 150 200 250 300 350 4000

10

20

30

40

Z rm

s, d

eg

pos(cm)

Z rms, Parmila 3D PICNIC Z rms, Parmila SCHEFF Z rms, pyORBIT

Longitudinal Beam Sizes in SNS MEBT. Space Charge 38 mA. Water Bag 3D, 38 mA 2,000 macro-particles for Ellipse SC 20,000 macro-particles for 3D FFT 32 x 32 x 32 grids Bunchers RF is on, 900 phases Timing: Parmila SHEFF – about 4 sec PARMILA 3D PICNIC – 8 sec pyORBIT 1 Ellipsoid – 1.6 sec


Benchmarking

•  ORBIT has been extensively benchmarked, including all its space charge models, through its 15 year tenure. –  Benchmark tests include comparisons with analytic results, with

experimental observations, and with other codes. –  PY-ORBIT has been thoroughly benchmarked with ORBIT as models have

been completed and tested.

•  Formal benchmark tests –  An excellent suite of benchmarks on several time scales that involves

space charge induced resonance trapping can be found on Giuliano Franchetti’s web site at http://web-docs.gsi.de/~giuliano/ •  ORBIT has successfully completed 8 of the 9 benchmark tests on this site. One case is

under study. •  Some of the longer time scale tests, as posed, are accompanied by significant

numerically-induced emittance growth.


Numerical Diffusion Effects Are Apparent in Individual Particle Behavior

•  Individual particle behavior is difficult predict, due to numerically-induced collisions.

Calculated tunes of individual particles are sensitive their diffusion.

4.24

4.26

4.28

4.3

4.32

4.34

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Qx

x/sigma

One Turn1024 Turns


Numerical Diffusion Effects Are Apparent in Individual Particle Behavior

•  Individual particle behavior is difficult predict, due to numerical noise.

Individual particle orbits diffuse in time.

For long times, convergence criteria are hard to satisfy.

0

10

20

30

40

50

60

70

0 2000 4000 6000 8000 10000 12000 14000

Hor

izon

tal E

mitt

ance

{mm

-mr}

Turns

s0 = 2.51*Sigmass0 = 2.50*Sigmas


2.5D PIC Space Charge Model •  “Comparison Between Measurements, Simulations, and Theoretical Predictions of the Extraction Kicker

Transverse Dipole Instability in the Spallation Neutron Source,” J.A. Holmes, S. Cousineau, V. Danilov, and L. Jain, Phys. Rev. Special Topics – Accelerators and Beams 14, (2011), 074401 .

predicted range of dominant unstable mode numbers fromthe extraction kicker impedance.

The experimental results for this case have been presentedin Ref. [6], and the growth rate of the 6 MHz (n ! 12)harmonic was used with Eq. (3) to infer the extractionkicker impedance at that frequency. The resulting predictionof 28 k!=m is in excellent agreement with the laboratory-measured impedance of 30 k!=m. We have now performeda precise simulation of this case, using ORBIT and matchingall known experimental details as closely as possible.

VI. SIMULATION OF OBSERVED EXTRACTIONKICKER INSTABILITY

We used the actual ring settings with !x ! 6:23 and!y ! 6:20 and zero chromaticity. The ring rf cavitieswere turned off and a continuous coasting beam of7:7" 1013 protons at 860 MeV was injected for 850 turnsand stored up to 10 000 turns. The injected beam rmsenergy spread was taken to be 0.5 MeV, consistent withobservation, and the nominal SNS transverse injectionpainting was employed. Tracking was carried out usingsymplectic single particle transport, the laboratory-measured longitudinal and transverse impedances for theextraction kickers, and the 3D space-charge model. Inaddition, the ORBIT foil scattering model was activatedand a complete set of apertures was included to incorporatebeam losses during accumulation and storage. The numberof macroparticles in the simulation was 4:25" 106, aboutdouble the number required for convergence of the 3Dspace-charge model.

One of the more impressive results presented in Ref. [6]was the above-mentioned agreement between the extrac-tion kicker impedance calculated from the observedgrowth rate of the instability using Eq. (3) and that mea-sured in the laboratory. There are 14 extraction kickersdistributed over a length of about 10 m in the SNS ring. Thevertical beta function at the kickers varies from a minimumof 6.4 m to a maximum value of 13.5 m, with an averagevalue of 9.3 m. The measurement gave an experimentalgrowth time of 1036 turns. In estimating the value of28 k!=m using Eq. (3), a value "s0 ! 6 m, which is closeto the average for the entire ring, was assumed. If, instead,the average value of "s0 ! 9:3 m at the extraction kickersis used, the estimated impedance is 18 k!=m, which islower than the lab value of 30 k!=m. However, Eq. (3)was derived under the assumptions that we are far fromthreshold and that the energy distribution is a delta func-tion, thus ignoring Landau damping. Both these assump-tions overestimate the growth of a real beam with energyspread. Therefore, we expect the experimental growth timeto be longer than that from Eq. (3), and its use in Eq. (3)should result in an impedance prediction that is somewhatlower than the actual value.

The ORBIT simulation was carried out with a singleextraction kicker impedance node using the experimentally

measured impedance values from Fig. 3 and placed at aposition among the extraction kickers where the beta func-tion satisfies"s0 ! 9:3 m. The result, shown in Fig. 8, is an

exponential growth time for the n ! 12 harmonic that iscompletely consistent with themeasured time of 1036 turns.This impressive result is an important testimony to thenecessity of getting all the details correct when performinga quantitative comparison between experiment and simula-tion. In reaching the result of this simulation, we made anumber of false starts [8–10]. Erroneous assumptions in-cluded the use of chopped beams, the use of (too large)impedances from a previous design of the extractionkickers, and the placement of the impedance node at thegeometric center of the extraction kickers rather than at alocation with the average beta function. With hindsight,mistakes such as these appear to be foolish. However, insimulating a complicated particle accelerator, there aremany details, each of which can affect the results. As

0.01

0.1

1

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

n=12

Y H

arm

onic

{m

m}

Turn

T = 1036 Turns

FIG. 8. Vertical n ! 12 harmonic (in blue) versus turn numberin the ORBIT extraction kicker instability simulation. The red linedepicts an exponential growth time of 1036 turns.

FIG. 9. Evolution of simulated turn-by-turn vertical harmonicspectrum of the extraction kicker instability.

COMPARISON BETWEEN MEASUREMENTS, SIMULATIONS, . . .Phys. Rev. ST Accel. Beams 14, 074401 (2011)

074401-9

impedance, the transverse impedance (we used 32 modes)as shown in Fig. 3, and variations on the single particletransport and presence of space-charge forces. In all cases,thresholds were obtained in terms of impedances by multi-plying the measured extraction kicker impedance by vary-ing coefficients. The results are shown in Table III, bothwithout and with space charge. The cases are as follows:(1) linear MAD tracking; (2) symplectic tracking, correctedchromaticity; and (3) symplectic tracking, natural chroma-ticity. The results of case 3 show, as in the coasting beamcalculations, that chromaticity leads to significant stabili-zation. We also see from cases 1 and 2 that, if space chargeis neglected, SNS at zero chromaticity is predicted to beunstable at the extraction kicker impedance. The importantcolumns are those including space charge. Unlike thecoasting beam case in Table II, for which space charge isstrongly destabilizing, the effect of space charge on theSNS bunched beam is stabilizing for the zero chromaticitycase and somewhat destabilizing at natural chromaticity.Most important, these calculations predict that SNS will bestable with at least a factor of 2 to spare over the measuredextraction kicker impedance. Although the stability ofbunched beams is an area of active theoretical research[33,34,36,37], we do not consider an analytic treatmenthere.

From these studies, we conclude that, for coastingbeams, the ORBIT code benchmarks very well with analyticresults of instability thresholds, including the effects ofphase slip, chromaticity, and space charge. These beamsare stabilized due to Landau damping by increasing theenergy spread and/or the chromaticity. For coasting beams,when the linear treatment of space charge is appropriate, ittends to be destabilizing, as the imaginary space-chargeimpedance shifts the beam away from the stabilizingLandau-damped portion of the stability diagram. Also,coasting beam results are found to be less stable than thoseof bunched beams for otherwise similar cases. The coast-ing beam model predicts instability for SNS ring energydistributions and intensities, while realistic simulation with3D space charge shows that bunched beams are stable,even for zero chromaticity. Unlike coasting beams,bunched beams are not significantly destabilized byspace-charge effects. The greater stability of bunchedbeams is due to several factors, including the coupling ofmany modes and the spread of betatron tunes along thelongitudinal coordinate due to vacuum chamber and bunchfactor effects. Ultimately, real bunched-beam dispersionrelations will be required to describe analytically our par-ticular SNS Ring situation.

V. THE OBSERVED EXTRACTIONKICKER INSTABILITY

Section IV shows that the Spallation Neutron Sourceaccumulator ring was designed and constructed to be stableat the full intensity of 1:5! 1014 protons. Bunched-beam

stability calculations using ORBITwith the extraction kickerimpedance show longitudinal stability up to 8!1014 protons [42], while Table III shows that transversestability at 1:5! 1014 protons is predicted for up to 3 to 4times the known impedance. However, for coasting beamsin SNS, Table II shows that analytic and ORBIT calculationsfor mode number n " 10 and 1:5! 1014 protons predictvertical instability when Re#Z$> 0 k!=m at zero chroma-ticity and when Re#Z$ % 250 k!=m at natural chromatic-ity. Because the measured impedance of the extractionkickers in the vicinity of n " 10 is Re#Z$ & 30 k!=m, itis appropriate to use coasting beams and corrected chro-maticity to look for this instability experimentally.The extraction kicker instability has been observed

under these conditions in the course of high intensitybeam studies. The scenario was the following: The ringtunes were set at !x " 6:23 and !y " 6:20. The chroma-ticity was corrected to zero and the rf buncher cavities wereturned off. The choppers were also turned off so that acontinuous coasting beam was accumulated. An 860 MeVbeam of 7:7! 1013 protons, more than 12 "C, was in-jected for 850 turns and subsequently stored until thebeam was lost in the ring. The evolution of the beam wasfollowed for 10 000 turns. The observed instability beganat about 1200 turns and saturated somewhat beyond4000 turns. It was active in the transverse vertical directionwith dominant harmonic at 6 MHz and noticeable excita-tion in the 4 ! 10 MHz range, as shown in Fig. 7.Interpreting this to be a ‘‘slow’’ mode, the frequency isconsistent with dominant harmonic n " 12, and excitationin the range 10 ' n ' 16. Because the SNS ring frequencyis roughly 1 MHz and the betatron tunes are slightly abovesix, mode numbers for the ‘‘slow mode’’ correspondroughly to the observed instability frequency in MHzplus six. The observed signal range agrees well with the

FIG. 7. Evolution of experimental turn-by-turn vertical har-monic spectrum of the extraction kicker instability.

J. A. HOLMES et al. Phys. Rev. ST Accel. Beams 14, 074401 (2011)

074401-8

predicted range of dominant unstable mode numbers fromthe extraction kicker impedance.

The experimental results for this case have been presentedin Ref. [6], and the growth rate of the 6 MHz (n ! 12)harmonic was used with Eq. (3) to infer the extractionkicker impedance at that frequency. The resulting predictionof 28 k!=m is in excellent agreement with the laboratory-measured impedance of 30 k!=m. We have now performeda precise simulation of this case, using ORBIT and matchingall known experimental details as closely as possible.

VI. SIMULATION OF OBSERVED EXTRACTIONKICKER INSTABILITY

We used the actual ring settings with !x ! 6:23 and!y ! 6:20 and zero chromaticity. The ring rf cavitieswere turned off and a continuous coasting beam of7:7" 1013 protons at 860 MeV was injected for 850 turnsand stored up to 10 000 turns. The injected beam rmsenergy spread was taken to be 0.5 MeV, consistent withobservation, and the nominal SNS transverse injectionpainting was employed. Tracking was carried out usingsymplectic single particle transport, the laboratory-measured longitudinal and transverse impedances for theextraction kickers, and the 3D space-charge model. Inaddition, the ORBIT foil scattering model was activatedand a complete set of apertures was included to incorporatebeam losses during accumulation and storage. The numberof macroparticles in the simulation was 4:25" 106, aboutdouble the number required for convergence of the 3Dspace-charge model.

One of the more impressive results presented in Ref. [6]was the above-mentioned agreement between the extrac-tion kicker impedance calculated from the observedgrowth rate of the instability using Eq. (3) and that mea-sured in the laboratory. There are 14 extraction kickersdistributed over a length of about 10 m in the SNS ring. Thevertical beta function at the kickers varies from a minimumof 6.4 m to a maximum value of 13.5 m, with an averagevalue of 9.3 m. The measurement gave an experimentalgrowth time of 1036 turns. In estimating the value of28 k!=m using Eq. (3), a value "s0 ! 6 m, which is closeto the average for the entire ring, was assumed. If, instead,the average value of "s0 ! 9:3 m at the extraction kickersis used, the estimated impedance is 18 k!=m, which islower than the lab value of 30 k!=m. However, Eq. (3)was derived under the assumptions that we are far fromthreshold and that the energy distribution is a delta func-tion, thus ignoring Landau damping. Both these assump-tions overestimate the growth of a real beam with energyspread. Therefore, we expect the experimental growth timeto be longer than that from Eq. (3), and its use in Eq. (3)should result in an impedance prediction that is somewhatlower than the actual value.

The ORBIT simulation was carried out with a singleextraction kicker impedance node using the experimentally

measured impedance values from Fig. 3 and placed at aposition among the extraction kickers where the beta func-tion satisfies"s0 ! 9:3 m. The result, shown in Fig. 8, is an

exponential growth time for the n ! 12 harmonic that iscompletely consistent with themeasured time of 1036 turns.This impressive result is an important testimony to thenecessity of getting all the details correct when performinga quantitative comparison between experiment and simula-tion. In reaching the result of this simulation, we made anumber of false starts [8–10]. Erroneous assumptions in-cluded the use of chopped beams, the use of (too large)impedances from a previous design of the extractionkickers, and the placement of the impedance node at thegeometric center of the extraction kickers rather than at alocation with the average beta function. With hindsight,mistakes such as these appear to be foolish. However, insimulating a complicated particle accelerator, there aremany details, each of which can affect the results. As

0.01

0.1

1

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

n=12

Y H

arm

onic

{m

m}

Turn

T = 1036 Turns

FIG. 8. Vertical n ! 12 harmonic (in blue) versus turn numberin the ORBIT extraction kicker instability simulation. The red linedepicts an exponential growth time of 1036 turns.

FIG. 9. Evolution of simulated turn-by-turn vertical harmonicspectrum of the extraction kicker instability.

COMPARISON BETWEEN MEASUREMENTS, SIMULATIONS, . . .Phys. Rev. ST Accel. Beams 14, 074401 (2011)

074401-9

Left: Growth of dominant harmonic matches measured and predicted values.

Right: Experimental (top) and simulated (bottom) spectral evolution.


Back Up Slides: •  Back Up Slides:


Longitudinal PIC Space Charge Model •  “Space-Charge-Sustained Microbunch Structure in the Los Alamos Proton Storage Ring”, S. Cousineau, V.

Danilov, J. A. Holmes, and R. Macek, Physical Review Special Topics – Accelerators and Beams 7, (2004), 094201.

structure, visible as the high frequency density (color)variations, persists for at least 800 turns after the end ofinjection. The microbunch structure is even more longlived in the unchopped beam case, remaining strongthroughout the entire storage, up until the beam is ex-tracted from the ring at 1950 turns. The longer wave-length (lower frequency) waves seen at the end of thestorage in both the chopped and unchopped beam arerelated to the ferrite instability, which we do not considerhere.

A more precise method of characterizing the behaviorof the 201 MHz microbunch structure is by monitoringthe strength of the corresponding harmonic (harmonicnumber 72) of the longitudinal beam signal during injec-tion and storage. In order to eliminate possible contribu-tions from higher order harmonics of the ferriteinstability, a bandpass filter was applied to the data beforethe FFTs were taken; the bandpass region was taken from175 through 210 MHz, sufficient to exclude the nearestharmonics of the ferrite instability. Figures 3(a) and 3(b)show the analysis for the chopped and unchopped beamdata presented in Figs. 1 and 2, respectively. The resultsare surprising: For the chopped beam conditions (Fig. 1),

the harmonic is strong throughout the injection period,during which the beam is constantly supplied at the linacfrequency. However, instead of weakening after the in-jection has ceased, the harmonic strength increases sig-nificantly during the first 300 turns of storage, beforefalling off to a constant, lesser strength for the remainderof the storage. The effect is even more pronounced forunchopped beam conditions (Fig. 2), where the harmonicstrength increases for a full 1000 turns after the end ofinjection, and then drops sharply before increasing againuntil the end of storage. It is not possible to know how longthe microbunch structure would have persisted had theunchopped beam been allowed to store for a longer periodof time. Regardless, from the standpoint of single micro-bunch considerations, it is astonishing that the linacmicrobunch structure amplifies during the beam storage,where energy spread and space-charge effects would haveintuited the opposite response.

To investigate the intensity dependence of the micro-bunch structure during beam storage, we compare single-turn longitudinal profiles for two data sets with differentbeam intensities but otherwise identical experimentalconditions. The two plots shown in Fig. 4 correspond to

FIG. 2. (Color) Waterfall plot of the wall current monitorsignal for the unchopped beam experiment. The time in asingle pulse is plotted on the horizontal axis, and the turnnumber is plotted on the vertical axis. Beam injection ends at559 turns. The high frequency variations in color indicate themicrobunch structure of the beam.

FIG. 1. (Color) Waterfall plot of the wall current monitorsignal for the chopped beam experiment. The time in a singlepulse is plotted on the horizontal axis, and the turn number isplotted on the vertical axis. Beam injection ends at 559 turns.The high frequency variations in color indicate the microbunchstructure of the beam. The beam gap region is displayed in blue.


094201-3 094201-3

role of space charge in the phenomenon, in the nextsection we reproduce the chopped beam experimentswith detailed particle-in-cell (PIC) tracking simulations.

III. SIMULATION

ORBIT is a PIC code developed for realistic simulationsof beams in rings and transport lines [11,12]. The codehas been successfully benchmarked with experimentaldata for a number of cases [13,14]. For the simulationspresented here, we invoke ORBIT’s 1D longitudinal track-ing package, which includes an algorithm for modelingthe effects of longitudinal space charge and user-definedexternal impedances [15]. One combined space chargeand impedance kick is applied per turn of the beam,adequate for modeling the slow longitudinal phase spaceevolution of the PSR beam. Although initially the persis-tent microbunch structure was observed in simulationswith the ferrite inductor impedance included, no corre-lation was found between the sustained structure and theinductor impedance. To demonstrate that space charge isthe relevant effect, in the simulations presented here theimpedance of the inductive inserts is intentionally omit-ted. High numerical resolution is attained by slicing thebeam longitudinally into 256 bins and using 8! 106

macroparticles.

The experimental parameters for the chopped beamcase, listed in the previous section, were used in the ORBITsimulations. To allow for direct comparison with theexperimental data, turn-by-turn longitudinal beam pro-files were plotted with the same resolution as the wallcurrent monitor data (0.5 ns). The simulations were per-formed both with and without space charge, and theresults are shown in waterfall plot format in Figs. 5(a)and 5(b), respectively. When space charge is included, weobserve the same microbunch dynamics as in the experi-mental data: The 201 MHz structure is locked in placeand sustained for at least 800 turns after injection. In thecase where no space charge is present, however, the mi-crobunch structure decoheres due to energy spread effectswithin a few tens of turns after the end of injection.

In a system where energy spread effects are negligibleand space charge alone governs the longitudinal dynam-ics, the steady-state solution for the distribution is alwaysd!=d" " 0, where ! is a line density and " is theindependent variable along the length of the beam.However, in the present system, both space charge andenergy spread effects are non-negligible, and evidencefrom both the experimental and simulated data points tothe existence of another steady-state solution.

To understand the dynamics of the system, we inves-tigate the longitudinal phase space of the beam using the

FIG. 5. (Color) (a) Waterfall plot of simulated wall current monitor signal for chopped beam experiments. Space-charge effects areincluded. (b) Same as in (a) but with no space charge included.


094201-5 094201-5

Experiment Simulations

Space Charge No Space Charge


2D PIC Space Charge Model •  “Comparison of Simulated and Observed Beam Profile Broadening in the PSR and the Role of Space Charge”, J.

D. Galambos, S. Danilov, D. Jeon, J. A. Holmes, D. K. Olsen, F. Neri, and M. Plum, Physical Review Special Topics – Accelerators and Beams 3, (2000), 034201.

•  “Resonant Beam Behavior Studies in the Proton Storage Ring”, S. Cousineau, J. A. Holmes, J. Galambos, A. Fedotov, J. Wei, and R. Macek, Physical Review Special Topics – Accelerators and Beams 6, (2003) 074202 .







-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

-40 -30 -20 -10 0 10 20 30 40

Arb

itrar

y U

nits

Y (mm)


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

-4 -3 -2 -1 0 1 2 3 4

Arb

itrar

y U

nits

! (rad)



074202-3 074202-3







-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

-40 -30 -20 -10 0 10 20 30 40

Arb

itrar

y U

nits

Y (mm)


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

-4 -3 -2 -1 0 1 2 3 4

Arb

itrar

y U

nits

! (rad)



074202-3 074202-3 In addition to single particle tunes crossing the inco-herent stop band limit, the behavior of the beam envelopedeviates from the zero-space-charge case, as predicted bySacherer. The beam envelope can be studied by calculat-ing the second moment of the beam distribution over oneturn of the ring. Figure 6 shows the one-turn secondmoments of the fully accumulated, high intensity beam,taken at three different longitudinal locations with re-spect to the rms longitudinal bunch length (!l !1:66 rad, or 23.8 m): one at approximately 0:01!l, one atapproximately 0:25!l, and one at approximately !l. Themoment is normalized by the zero-space-charge beamenvelope,

!!!!!!!!!!!!!!!!

"y"s##yq

, such that the vertical axis representsa fractional deviation from this envelope. We observe that

the longitudinal center of the beam undergoes as much asa 20% deviation from the zero-space-charge envelope.The envelope executes about four oscillations per turn,consistent with Sacherer’s predictions for the periodic,coherent half-integer envelope resonance. Notice also thatthe moment amplitudes drop very quickly outside of thelongitudinal center. This behavior is detailed in Fig. 7,where the strength of the m ! 4 envelope harmonic issampled via a Fourier transform of the second momentsof the beam over the longitudinal coordinate range. Asexpected, the distribution of strengths is strongly peakedat the longitudinal center of the beam, with the harmonicstrength dropping to less than half of its peak valuewithin 0.2 rad, or 0:12!l, on either side of the longitudinalcenter. Therefore, the envelope resonance excitation of thePSR beam is heavily concentrated in the central, highdensity portion of the bunch. The effect of the sharplypeaked longitudinal density distribution is discussed inmore detail in Sec. IVA.

Figure 6 demonstrates that the resonant motion of thebeam envelope is dominated by the periodic component,which is a fixed point in the envelope phase space.However, a real beam does not sit exactly on the fixedpoint, but instead executes small oscillations about thispoint [11,17]. The tune of the small oscillations isbounded above by the zero-space-charge upper limit,$e ! 2$y, and decreases with increasing space charge.In the PIC simulations, the amplitude of the oscillationis too small to observe with second moment analysis. It isin fact much more practical to measure the tune of themismatched envelope by solving the envelope equationsof motion. For our calculations, we use an envelope modelwhich has been derived from an rms standpoint and doesnot include higher order moment contributions [18]. Themodel has been implemented into the ACCSIM particletracking code [19], where the envelope motion including

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

1.25

1.3

0 10 20 30 40 50 60 70 80 90

Nor

mal

ized

Sec

ond

Mom

ent

Distance [m]

FIG. 6. (Color) Vertical second moment of the beam, normal-ized to

!!!!!!!!!!!

"y#yp

, taken over one turn near the end of accumu-lation of 4:37$ 1013 protons. The curves correspond tolongitudinal slices of beam at approximately 0:01!l (red solidline), 0:25!l (blue dashed line), and !l (pink dotted line).

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

-3 -2 -1 0 1 2 3

n=4

Har

mon

ic S

tren

gth

Longitudinal Coordinate [rad]

FIG. 7. Strength of the $e ! 4:0 envelope harmonic versuslongitudinal coordinate. The harmonic strengths are obtainedvia FFT of the second moments of the beam at specificlongitudinal locations.

FIG. 5. (Color) Incoherent particle tune shifts versus longitu-dinal coordinate after the accumulation of the full intensityPSR beam (red points), the half intensity beam (blue points),and the quarter intensity beam (green points), obtained fromPIC simulations.


074202-5 074202-5

Transverse profiles: experiment and simulation with and without space charge

Longitudinal profiles: note bunch factor.

Incoherent tunes versus longitudinal position.


Choice of Independent Variable

•  Time or position. –  Time

•  Natural choice for evaluation of space charge. •  Requires knowledge of accelerator environment for the entire bunch ->

complicated for long bunches spanning multiple lattice elements.

–  Position •  Since all particles are at the same place, relative transverse positions of

longitudinally separated particles are incorrect for space charge evaluation. The particles exist at different times. This effect grows with longitudinal particle separation, but forces decrease with separation.

•  Convenient from the standpoint of accelerator environment, especially for long bunches.

•  To work, space charge forces must depend mostly on local, rather than distant, longitudinal separation.


Applicability of Model Depends on Information Sought

•  Bulk quantities, such as RMS emittances, can be obtained for short to medium time scales with moderate numbers of particles, but numerical diffusion leads to slow growth over long times. Even for short times, it is necessary to be sure the numerics converge.

256x256 grid with various numbers of particles.

0.999

0.9995

1

1.0005

1.001

1.0015

1.002

1.0025

1.003

0 100 200 300 400 500 600 700 800 900 1000

RM

S Em

ittan

ce {m

m-m

rad}

Turn


Ex, 1M particles, 256 x 256 GridEy, 100K particles, 256 x 256 GridEy, 333K particles, 256 x 256 Grid

Ey, 1M particles, 256 x 256 Grid

100K Particles

333K Particles

1000K Particles

Set of Ellipsoids

pyORBIT Space Charge Solver can use arbitrary number of ellipsoids

Electric Field of Uniformly Charged Ellipse

inside

outside

Status of PY-ORBIT: Benchmarking and Noise Control in PIC ... · Python ORBIT (PY-ORBIT) •...

Documents

Transcript of Status of PY-ORBIT: Benchmarking and Noise Control in PIC ... · Python ORBIT (PY-ORBIT) •...