NEW FRONTIERS IN LEADERSHIP COMPUTING

High-Performance Computing Modeling AdvancesAccelerator Science for igh-Energy Physics

James Amundson, Alexandru Macridin, and Panagiotis Spentzouris I Fermilab National Accelerator Laboratory

Particle accelerators are essential or advancing o r understanding of matter, energy, space, and time. Becausethey exhibit many physical effects o mu tiple scales, advanced computational tools are essential for accuratelymodeling them. The authors focus he e on Synergia, an accelerator simulation package capable of handling theentire spectrum of beam dynamics smulations.

he US Department of Energy (DOE)’sOffice of High-Energy Physics (HEP)promotes a broad, long-term particlephysics program by supporting current

operations, experiments, and research as well asdevelopment for future operations at three interrelated frontiers of particle physics1:

• The energy frontier directly explores the universe’s fundamental constituents and architecture through the highest-energy particlebeams.

• The intensity frontier enables a second, unique,investigation of fundamental interactions througha combination of intense particle beams and highly sensitive detectors.

• The cosmic frontier reveals the nature of darkmatter and dark energy by using particles fromspace to explore new phenomena.

These scientific frontiers form an interlockingframework that addresses fundamental questionsabout the laws of nature and the cosmos. The development and deployment of high-performancecomputing (HPC) accelerator modeling capabilities is essential to meeting these grand scientificHEP challenges because it enables and catalyzesadvancement in accelerator science. In the nextdecade, for example, the HEP community willexplore the intensity frontier by designing high-intensity proton sources for neutrino physics andrare process searches, such as Proton ImprovementPlans (PIP) I and II at Fermilab.

The design, cost optimization, and successfuloperation of modern accelerators require the optimization of many parameters, as well as the understanding and control of many physics processes.This can only be accomplished by employing high-fidelity computational accelerator models that efficiently utilize HPC resources. A comprehensivepicture of current HPC accelerator modeling capabilities in the US can be obtained by reviewingthe codes developed under the Scientific Discoverythrough Advanced Computing (SciDAC) programwithin the Community Petascale Project for Accelerator Science and Simulation (ComPASS).2These codes obtain good scalability and parallelefficiency on thousands to hundreds Gf thousandsof processors and are routinely used to performsingle-physics, single-scale simulations on a fewthousand processors. Their applications have enabled large multiscale multiphysics simulations ofthe most challenging accelerator science projectsand demonstrated the impact of large-scale simulations in accelerator science.3

Synergia is an accelerator simulation packagethat can handle the entire spectrum of beam dynamics simulations. Here, we present the Synergiabeam dynamics framework and discuss its currentapplications in the design of high-intensity protonaccelerators.

HEP’s Intensity FrontierThe intensity frontier addresses central questionsin particle physics that aren’t directly accessiblewith current or planned accelerators at the energy

32 computing in Science & Engineering 1521-9615/14/$31.OO © 2014 IEEE Copublished by the IEEE cs and the AlP NovemberlDecember 2014

frontier. Experiments at the intensity frontier studyare processes that indirectly probe higher-massscales and exotic physics using intense beams ofparticles such as neutrinos, muons, kaons, and nuclei, providing powerful probes of new phenomena.

In the US, the key to long-term leadership atthe intensity frontier is the PIP at Fermilab (stagesI and II),~’~ which is building a multimegawattproton accelerator that will produce intense beamsof neutrinos and muons (and possibly kaons andheavy nuclei with further upgrades). The culmination of these PIP efforts will be the delivery of anew superconducting radio frequency (RF) Linacand major improvements to Fermilab’s Boosterand Main Injector accelerators. High-fidelitysimulations are an important component for thecampaign’s success and cost-effectiveness.

Every proton in the domestic HEP experimental program will be accelerated by the existing (andnow 40-year-old) Fermilab Linac and Booster accelerators until new machines become operational toreplace them. The leading replacement candidate—the proposed superconducting linear accelerator—is anticipated for completion no sooner than 2020to serve demands for beams at 3 gigaelectronvolts(GeV) and lower energy and no sooner than wellinto the next decade to serve demand for beams athigher energy. The domestic HEP program for thenext 15 years therefore depends on the viability andvitality of the Linac and Booster, which Fermilabhas established a charge to assure, promising “2.25x i0’~ protons/hour (at 15 Hz) by January 1,2016” (more than two times the beam rate in current operations) while “ensuring a useful operatinglife of the p~ton source through 2025.~~4.6

Booster intensities and repetition rate are currently limited by radiation due to uncontrolledlosses. These losses are a problem both because ofprompt radiation levels and equipment activation.To ensure that we’ll achieve the PIP’s target intensities, it’s necessary to understand and suppressbeam instabilities as well as to develop and studytechniques to control and minimize beam losses(examples include optimized operating parameters,an improved collimation system, a second harmonic radio frequency system, and so on). High-fidelitysimulations that incorporate all the relevant physics effects will help guide accelerator scientists asthey undertake these tasks.

SynergiaRealistic accelerator simulations require treatmentof many devices and physical effects, and cover a

broad range of size and detail. Modern particle accelerators are complex devices, typically consistingof thousands of components. The settings of thesecomponents are often varied during a run eitherthrough preprogrammed operating cycles or in response to active feedback. The beams in these accelerators are usually bunched longitudinally with0(1012) particles in a bunch (the longitudinal di

rection is defined along the direction of the beampropagation). These bunches have transverse dimensions on the order of millimeters and a typicallongitudinal extent of meters.

Synergia (https://web.fnal.gov/sites/Synergia/SitePages/Synergia%2OHome.aspx) is an accelerator simulation package designed to take advantageof computational resources ranging from desktopmachines to computing facilities. It utilizes particle-in-cell (PlC) methods to combine advanced independent-particle dynamics with state-of-the-art collectiveeffects. The current version (2.1), which we describehere, is a hybrid Python/C++ implementation: all corecomputations are done in C++, while end-user simulations are described using Python. The combinationprovides efficiency and great flexibility.

Accelerator Simulation Problem DefinitionThe split-operator technique is the core mathematical concept for combining independent-particleand collective effects in Synergia. When the Hamiltonian for a system can be split into independent(i) and collective (c) components,

H=H1+H~. (1)

The split-operator approximation for the time-step evolution operator, M1~ is given by

Mfi,11 (t) = M~ (t)M, +12J L~2J

(2)

where M~ and M~ are the evolution operators corresponding to H~ and R~, respectively. Synergia abstracts this technique by defining each simulationas a series of steps, each of which is defined by anordered set of operators; a set of steps through an entire accelerator is called a turn. Typical simulationsof circular accelerators may consist of thousands to100,000 turns—real accelerator cycles can meanmillions of turns. Accelerator operations often require changing the accelerator parameters such asaltering magnet settings and RF phase shifts, possibly in response to beam conditions. Synergia canalso handle linear accelerators, through which the

www.computer.org/cise 33

NEW FRONTIERS IN LEADERSHIP COMPUTING

beam only passes once. Then, the (poorly named,in this context) number of turns is simply one.

A Synergia simulation consists of propagatinga single bunch or train of bunches through agiven number of turns. Along the way, varioususer-selected (or user-defined) diagnostics can beperformed on the bunches to monitor beam stateas it propagates through the machine. Operatorsact on a bunch of macroparticles representing thebeam bunch, propagating them forward in time.The simulation parameters are defined by a briefPython or C++ program written by the end user.This program may use existing Synergia classesor include custom extensions to those classes. Theentire state of the simulation, including end-userextensions, can be checkpointed or resumed atany point. This checkpointing mechanism allowsboth for recovery from hardware failures and thechaining of multiple jobs in time-limited queues tocomplete one long simulation.

Synergia consists of a core set of C++ classesthat are exposed to Python via Boost.Python (www.boost.org/doc/libs/ 1_56_O/libs/python/doc). Theindependent-particle dynamics are handled by theCHEF C++ libraries.7 Both Synergia and CHEFwere developed at Fermilab.

Independent-Particle PhysicsMost modern particle accelerators are dominatedby independent-particle physics, which means thatthe forces felt by beam particles due to externallyapplied fields (magnets, accelerating structures,and so on) are much larger than the forces dueto interactions with other particles in the beam.The CHEF libraries, which we use in Synergia,provide a comprehensive set of tools for linear andnonlinear independent-particle accelerator physics.

CHEF not only models particle propagationthrough accelerators, but it also performs symbolicalgebra calculations on the transfer maps forthe propagation, allowing the user to performanalyses such as nonlinear map analysis of resonantstructures. In Synergia, we take advantage of thisability by incorporating two different types ofparticle propagation: direct numerical integrationand polynomial map application. In the latter, thesix-dimensional final phase space coordinates of aparticle u1 are related to the initial coordinates ui by

Uf M(u~),

where M is a polynomial in the phase spacecoordinates u. This approach is useful both in

terms of efficiency for some calculations and forcomparison with other accelerator simulationpackages, many of which are limited to fixed-orderpolynomial maps for particle propagation.

Collective EffectsSynergia includes a general treatment of collectiveaccelerator physics effects via the split-operatorapproximation described earlier. Synergia 2.1 includes implementations of the two most importanteffects for high-intensity proton accelerators—space charge and wake fields. Space charge effectsarise from the electromagnetic repulsion betweenlike-charged particles in the beam, and wake fieldeffects occur when leading portions of the beam induce currents in the beam pipe or other structuresthat give rise to residual electromagnetic fields feltby the passing particles. Synergia’s design lets theuser implement additional collective effects and/ordifferent approximations or implementations of theexisting space charge and wake field models.

Space charge. Calculating the effects of spacecharge requires solving the Poisson equation,

(4)

where c~5 is the scalar electric potential and p isthe charge density due to the beam particles. In abunched beam, the longitudinal separation betweenbunches (typically on the order of meters) is usually much greater than the transverse dimensionsof the beam (typically on the order of millimeters),so bunch-to-bunch space charge effects can beneglected.

Synergia includes space charge solvers at different levels of approximation. The simplest solveruses an analytic approximation for the field dueto a two-dimensional Gaussian charge distribution with open boundary conditions. All the othersolvers obtain numerical solutions to the Poissonequation on a discrete grid. The solvers includetwo- and three-dimensional approaches and multiple boundary conditions. Figure 1 shows a spacecharge solve with macroparticles and the resultingscalar field.

Wake fields. The effects of induced wake fieldsin beam pipes with horizontal and vertical mir

(3) rot symmetry can be summarized using theexpressions8

qQW1I(z) (5)

34 November/December 2014

—qQ(Wx(z)X + W~(z)x)

cL~.p qQ(Wy(z)Y+ W~(z)y),

where Q, X, Y (q, x, y) represent the charge andhorizontal and vertical displacements of the leading (trailing) particle, and Wx,y,11 represents thewake functions that need to be calculated for a given geometry. The momentum of a trailing particlegoing through a lattice element is kicked, that is,instantaneously changed, by a term proportional toa wake function that depends only on the distancez between the leading and the trailing particle.Wake field effects include both intra- and inter-bunch interactions.

Parallel PerformanceThe Synergia design includes parallelism at itscore. Simple Synergia simulations can be run ona desktop machine with a single core, but userscan easily utilize a range of parallel resourcesranging from multicore desktops to 100,000+core supercomputers. Figure 2 demonstrates codescalability. We find excellent strong scaling behavior over a range of a little less than a thousandon a Blue Gene/Q machine and weak scaling inthe number of macroparticles (not shown) andbunches (Figure 2b).

Scalability in Synergia is limited by the communication requirements of collective effects.The scaling we’ve achieved to date relies on twocomplementary techniques: communicationavoidance and hybrid message passing interface(MPI)/OpenMP optimizations. In communication avoidance, we perform multiple, redundantfield solves to reduce the size of the necessarycommunication steps. The hybrid MPI/OpenMPoptimizations involve reducing the number ofMPI processes while using OpenMP threads totake advantage of the multiple cores available foreach process. Here, communication is reducedbecause of the smaller number of MPI processes.The use of OpenMP is optional; once it’s enabled,the number of threads per process can be chosenat runtime.

In preparation for the next generation of HPCtechnologies, we’re actively working on adaptingSynergia for GPUs and Intel’s MIC architecture.The prototype GPU implementation is able to takeadvantage of multiple GPUs using communicationavoidance a single field solve fits naturally on acurrent GPU. The most difficult portion of the collective calculations in both the OpenMP and GPU

Figure 1. Macroparticles with space charge (scalar electric) field. Thenumber of macroparticles has been reduced by a factor of 1,000 for clarity.An animated version of this figure is available at http://compacc.fnal.gov/-.amundson/animation-synergia.m4v.

Bunches

_____________________ 64 128 256 512 1,024

Figure 2. Code scalability. (a) Strong scaling of a single-bunch space chargesimulation on the Argonne Leadership Computing Facility’s (ALCF’s) Mira,a Blue Gene/Q machine. The space charge calculation uses a 32 x 32 x1,024 grid and 100 million macroparticles. (b) Weak scaling of a multiple-bunch space charge simulation on ALCF’s Intrepid, a Blue Gene/P machine.The space charge calculation uses a 32 x 32 x 1,024 grid and 100 millionmacroparticles per bunch; the largest simulation has a total of over13 billion macroparticles.

implementations is the charge deposition calculation. In this case, multiple threads need to write toa single grid in memory, resulting in locking issues.After extensive experimentation, we’ve settled ona red-black scheme for interleaved writes. Figure 3shows our preliminary results after running a fullSynergia benchmark on Tesla and Kepler GPUs incomparison with results from a single Intel Xeon

(7)

U,

0

=0U,

0

a)E

1=

1e4

30..~

1e3 25

201e2

15

10lel — ideal

~ actual5

leO iii 016 64 256 1k 4k 16k 8,192 32,768 131,072

Number of cores BG/P (Intrepid) cores

www.computer.org/cise 35

NEW FRONTIERS IN LEADERSHIP COMPUTING

29.Os 26.4s12.Os

Figure 3. The overall simulation time for one turn of a particle beam ondifferent platforms. Running a full Synergia benchmark on Tesla and KeplerGPUs offers promising results.

processor and a cluster of Intel Xeons. The resultsare promising so far. We plan to release productionversions of Synergia for GPUs and MICs next year.

Modeling the Fermilab BoosterThe Fermi National Accelerator Laboratory(FNAL) Booster accelerator is an approximately150-meter-diameter proton synchrotron with aninjection energy of 400 megaelectron volts (MeV)and an extraction energy of 8 GeV. The Boosteraccelerator is made up of 96 dipole-quadrupolecombined function magnets (focusing [F] and defocusing [D]) in a series of 24 repeating periods.The magnets’ vacuum chamber has a quasi-flatgeometry consisting of two parallel planes alongthe horizontal direction. The combined functionmagnets cover about 60 percent of the machinelength—the rest of the machine is made up ofstraight metallic beam pipe sections. A consequence of the presence of bare laminations is theformation of very large wake fields. Since the machine runs at low energy (injection F/final F0.4 GeV/8 GeV), the space charge is also strong.The Booster accelerator runs with an average repetition rate of 9 Hz at an intensity of 4.5 x 10~~protons per batch, which is about two times largerthan the originally designed intensity. A batchconsists of 84 bunches. The next generation ofFermilab neutrino experiments require even higher proton output, approximately 6 x 1012 protonsper pulse at 15 Hz.

Two intensity-dependent effects are importantin the Fermilab Booster: space charge and wakefields in the laminated magnets. We’ve developeda detailed model of wake fields in laminated structures in Synergia and validated it with experimental results.9 The wake field and space chargecalculations are inherently coupled through theboundary conditions in the space charge field, soour model has to include space charge with compatible boundary conditions. These collective effects in turn affect particle propagation throughthe accelerators, often creating instabilities thatresult in particle losses and degradation of beamquality. Simulations can provide insight into theinstabilities’ mechanism and the necessary guidance to improve beam loss, but all the relevantphysics, including accurate single-particle dynamics (determined by accelerator description accuracy) and accurate multiparticle dynamics (spacecharge, single and multibunch wake field effects),should be properly considered.

There’s a long history of successful Synergia applications in support of the high-intensity FNALphysics program. Synergia simulations were used tostudy emittance growth and beam halo generationin the Fermilab Booster during the second collidermode run of the FNAL program, which began in2001. Our models also enabled the first-ever simulation of Linac microbunch capture, debunching,and acceleration, including beam position feedback,three-dimensional space charge,1° and multibunchimpedance effects.9 These simulations providedguidance to machine operators to reduce losses,maximize intensity; and commission the Boostercollimators, which were essential to the neutrinoprogram’s success during the second Tevatron run.Our current work aims to support the PIP program, which has much more demanding intensityrequirements, by running simulations of a Boosteraccelerator model that incorporates all the essentialsingle- and multiparticle dynamic effects. To validateour model’s accuracy, we compared our simulationresults with beam experiments. To demonstrate thecomputation’s scale, the results presented next usedroughly 20 million core-hours, primarily runningl6,384-core jobs with 1012 particle steps per job.

Model ConstructionThe accelerator’s single-particle dynamics are determined by the positions and strengths of thedifferent magnets and accelerating cavities thatcomprise the accelerator lattice. To ensure agreement between the lattice model and the real lattice,

00

0

0(0bi~(0

Xeon X5550n Wilson ClusterD TeslaCl060x1C Tesla C 1060 x4

Kepler K20 xlKepler K20 x4

96s

455

oc’4

74.6s

36November/December 2014

Logarithmic scale

100

‘ ‘

0 100 200 300 400 01234 30 60 90

z(m) z(m)(b) (c)

-0.030

x

E-0.06 ~

Figure 4. Impedances and wakes. (a) Transverse impedances Zxand Z~for the laminated F magnet and the metallicstraight section in the Fermilab Booster. The impedance in the laminated magnets (see inset, logarithmic scale) isthree to four orders of magnitude larger than in the metallic pipe and has a very different frequency behavior. Atsmall frequencies, the horizontal impedance is larger than the vertical one. (b) Horizontal and (c) vertical wakes. At ashort distance, the vertical wake is about two times larger than the horizontal one, but the situation changes at largerdistances on the order of the few bunch lengths (1 bunch length = 5.654 m) relevant for instabilities.

we determined the parameters of the dipole andquadrupole correctors in the model lattice by usingOrbit Response Measurement to fit the measureddata.” For a realistic simulation, it’s important tohave an accurate estimate of wake functions. Wakecalculation requires solving the electromagneticproblem for the chamber in which the beam propagates—specifically, the solution depends on chamber geometry and boundary conditions for theelectromagnetic field at the vacuum chamber walls.

Due to the exposure of the laminations, theimpedances (which are related to the wake function via a Fourier transform) and wakes are ordersof magnitude larger in the combined functionmagnets than in the metallic pipes. We calculatedthe impedances and wake fields of the Booster’slaminated magnets at injection energy. The detailsof impedance calculations for flat laminated chambers in the ultrarelativistic limit appear elsewhere,9as do the nonultrarelativistic effects.8 Figure 4ashows the horizontal and vertical impedances forthe laminated F magnet and the transverse impedance in the straight metallic pipe. Besides the

fact that the impedance in the gradient magnets isthree or four orders of magnitude larger than theone in the metallic pipe, the frequency behavior isalso quite different.

Model Validation and AnalysisThe measurements in the Booster accelerator exhibit many unconventional effects. Our simulationswere able to capture very well the experimentallyobserved behavior.

For example, the vertical coherent tune thatis, the frequency of the bunch centroid’s oscillations—is strongly suppressed at large intensity dueto the strong wakes and the large space charge.However, the horizontal tune increases slightlywith beam intensity. This difference between thehorizontal and the vertical tune behaviors is a result of the fiat geometry of the combined magnet’svacuum chamber’2

Using our simulation, we calculated thecoherent tune by doing a Fourier transformof the beam centroid position as a function ofpropagation length. Figure 5a shows the spectral

300

E

200 ‘~ ~

I 0.01

100

Z,~ straight section— ReZ~ F magnet—. ImZ~ F magnet— ReZ~ F magnet—. ImZ~ F magnet

0 200 400f(MHz)

—I

(a)

0

f(MHz)

-0.09

www.computer.orglcise

NEW FRONTIERS IN LEADERSHIP COMPUTING

0.005

0.003be0)a(0

C-)0)0.

C,)

00.6 0.65 0.7 0.75

Tune

A• A

234 510

Particles per bunch x 10

Figure 5. Simulations and comparisions with data. (a) Fourier transform of the beam centroid horizontal andvertical displacements at intensity 4 x 1010 p per bunch for the full machine (84 bunches). When the collectiveeffects are neglected (red and black), the spectral weight (given in arbitrary units, or a.u.) exhibits sharp peaks forfrequencies corresponding to the bare tunes. The spectral weight shows small positive horizontal (blue) and largenegative vertical (green) tune shifts when the collective effects are present. Note the wide spectral features whenthe collective effects are included. (b) Coherent tune shift versus intensity in the Fermilab Booster at injection. Thesimulation results are compared with two experimental measurements, one in 2010 and the other in 2012. Thehorizontal tune increases slightly with intensity, while the vertical tune shows a strong decrease.

features. When no collective effects are included, the spectrum shows sharp peaks at frequencies corresponding to the bare tunes. With thecollective effects taken into account, the spectral weight shows small positive horizontal andlarge negative vertical tune shifts. Aside fromthat, the spectral features are broad, indicatingan interaction between multiple modes. Figure5b compares calculated tune shifts with experimental data.

A rather puzzling effect observed in the Booster accelerator is the presence of horizontal instability. Whereas the vertical wake is much larger thanthe horizontal wake, the bunch propagation issubject to instability in the horizontal plane. Oursimulations were able to reproduce this behavior;Figure 6 compares experimental data to our simulation. The beam envelope at the instability’s onsetis extracted from the experimental data shown inFigure 6a and plotted with magenta on top of thesimulation results in Figure 6b. A large horizontal

chromaticity is required to stabilize the beam.’3(Chromaticity measures tune dependence onmomentum/energy spread. We define chromaticity

as = w0 ~, where w0 is the revolution angular

frequency of the synchronous particle, ~ =

with ôv and .~- being the tune and momentum relativep

spread, respectively, and ~ the slippage factor thatmeasures the revolution frequency’s dependency onthe particle momentum spread.14)

To understand the instability present in thesystem, we performed simulations with modifiedinteraction terms. We neglected the space chargeterm and focused on the different wake terms inEquations 5, 6, and 7. Contrary to previous speculations,’3 we found that the vertical wake wasn’tresponsible for the instability—it was still presentwhen all wake terms except W~. were set to zero, as

0.004

bare H tune

H tune

AA A 0

A

bare V tune

V tune

AA

AA

0

• ~ exp 2012

• 1v~,, exp 2012

A ~ exp 2010

exp 2010

o ~ simulation

simulation

0.001

0.04

~0.08

—0.12

38 November/December 2014

HCOMBINE TURN BY TURN0864

C•~ 2

cL24680~

the green curve in Figure 7 illustrates. Therefore,the instability was caused by the horizontal waketerm, which couples with the leading particle’s displacement. One reason why instability wasn’t present in the vertical plane, despite the vertical wakebeing larger than the horizontal one, is that instability growth rate is proportional not only to wakestrength but also to the square root of the particles’oscillation amplitude. In the Booster accelerator, the

5001TURN

amplitude of the transverse oscillations in the gradient magnets is much larger in the horizontal plane.

Figure 8 shows simulations where the horizontal dipole wake tail (that is, the wake at distanceslarger than a certain cutoff distance) is set to zero.We found the simulations with a full range wake tobe very close to the one where the wake extendedup to only five bunch lengths. By making the wakerange shorter (on the order of two bunch lengths),

6.25 1

0.05

E0.00

0.05

200 400 600 800 1,000

Turn

Figure 6. Experimental instability data versus our simulation. (a) Beam horizontal instability measured at

= 2irx(0.06,0.025)m 1for the intensity 4 x 1012 p per batch.’3 (b) Beam centroid horizontal

displacement at beam position monitor (BPM) location versus the turn number for different horizontal

chromaticities, ~-~=2~r 0.023m (red), _~&i~=2irx0.046m (blue), ~~=2~x0069m 1 (green), and

= 2irx0.091m ‘(black). The vertical chromaticity is kept constant, ~j-≥’_ = 2irx0.023m 1, and the intensity

is 4.2 x 1012 p per batch. The magenta points represent the beam envelope extracted from the measurement in (a) after

the instability’s onset. The beam shows horizontal instability unless a large horizontal chromaticity ~ = 2~r x 0.091msimilar to the experiment,’3 is considered. /3c

www.computer.org/cise 39

300 400Turn

Figure 7. Simulations with modified wakes. (a) Number of macroparticles. (b) Beam horizontal centroid. Thehorizontal instability is present when the direct space charge and all wake terms expect the horizontal one at thelocation of the focusing magnet are set to zero.

we found a strong suppression of the instability.One conclusion is that the relevant wake rangefor the instability is between one and five bunchlengths.

At a small distance—shorter than one bunchlength—the vertical wake is roughly two timeslarger than the horizontal one. However, at largerdistances in the relevant distance region for instability, the horizontal wake is larger than thevertical one. Figures 4b and 4c show this wakebehavior, which can also be deduced by noticingthat the horizontal impedance at a small frequency(Figure 4a) is larger than the vertical one.

Q ur analysis showed that instability is causedby short-range bunch-bunch interactions viadipole horizontal wake. The reason for instabilityin the horizontal plane is twofold: the large amplitude oscillations in the horizontal plane at the location of the focusing magnets, and at the relevant

200

Wakes and space-chargeWakes onlyW_X wake onlyW X at F magnet only

500

500

NEW FRONTIERS IN LEADERSHIP COMPUTING

110,000

105,000

~ 100,000

95,000

90,000

85,000

80,0000 100

(a)0.10

0.05

-~ 0.00AV

0.05

0.10

(b)

0.005

0.004

0.003

0.002

o.ooiA~ 0.000

—0.001

0.002 10 bucket-length wake5 bucket-length wake

0.003 2 bucket-length wake1 bucket-length wake

0.004~ 200 300 400 500 600

Turn

Figure 8. Simulations with short wakes. All wake terms and space chargeexcept the dipole term W~are set to zero. In the red (blue, green yellow) plotW~ at a distance larger than 10 (5, 2, 1) bunch lengths is set to zero. Therelevant wake range for instability is on the order of a few bunch lengths.

0

600

600100 200 300 400

Turn

40 November/December 2014

distance (between one and five bucket lengths), thehorizontal wake is larger than the vertical one. Weare continuing both the experimental and simulation campaigns to better understand and characterize this effect.!!

AcknowledgmentsThis work was performed at Fermilab, operated by Fermi

Research Alliance, LLC, under contract De-ACO2-

07CH11359 with the US Department of Energy. It was

also supported by the ComPASS project, funded via

the Scientific Discovery through Advanced Computing

program in the DoE Office of High Energy Physics.

We also used resources from the Argonne Leadership

Computing Facility at Argonne National Laboratory,

which is supported by the Office of Science of the DoE

under contract DE-ACO2-06CH11357.

References1. Particle Physics Project Prioritization Panel, “US

Particle Physics: Scientific Opportunities, a Strate

gic Plan for the Next Ten Years,” US Dept. of En

ergy, 2008; http:/lscience.energy.gov/—/medialhep/

pdf/fileslpdfs/p5 report_06022008.pdf.

2. P. Spentzouris et al., “Community Petascale Proj

ect for Accelerator Science and Simulation: Ad

vancing Computational Science for Future Accel

erators and Accelerator Technologies,” I. Physics:

Conf Series, vol. 125, 2008, p. 012005; www.mcs.anl.gov/research/LANS/publications/index.

php?p_pub detail&id=178.

3. Fermilab, The Compass Project: High-PerformanceComputingfor Accelerator Design and Optimization,

2008; https://web.fnal.gov/collaboration/compass/SitePages/Home.aspx.

4. Fermilab, Fermilab Proton Improvement Plan, 2013;http://www-ad.fnal.gov/proton/PIP/PIP_index.html.

5. Fermilab, Project-X, aka Fermilab Proton ImprovementPlan II, Fermilab, 2013; http://projecrx.fnal.gov.

6. V. Lebedev et al., “Proton Source Improvement

Plan,” tech. memo FNAL/Beams-Doc 3781, Fer

milab, 2011; http://beamdocs.fnal.gov/AD-public/DocDB/ShowDocument?docid=3781.

7. L. Michelotti, “C++ Objects for Beam Physics,”Proc. 14th IEEE Particle Accelerator Conf, 1991,

pp. 1561—1563.8. A. Macridin, P. Spentzouris, and J. Amundson,

“Nonperturbative Algorithm for the Resistive Wall

Impedance of General Cross-Section Beam Pipes,”

Physical Rev. ST Accelerator Beams, vol. 16, Dec.

2013, p. 121001.9. A. Macridin et al., “Coupling Impedance and

Wake Functions for Laminated Structures with an

Application to the Fermilab Booster,” Physical Rev.

STAccelerator Beams, vol. 14, 2011, p. 061003.

10. J. Amundson et al., “An Experimentally RobustTechnique for Halo Measurement Using the 1PM

at the Fermilab Booster,” Nuclear Instruments and

Methods, vol. A570, 2007, pp. 1—9.

11. V. Lebedev et al., “Model Calibration and Optics

Correction Using Orbit Response Matrix in the Fer

milab Booster,” Proc. Int’l Particle Accelerator Conf,

2012, Pp. 1251—1253; http://accelconf.web.cern.ch/

accelconf/ipac2Ol2/papers/tuppc040.pdf.

12. A. Chao, S. Heifets, and B. Zotter, “Tune Shifts of

Bunch Trains Due to Resistive Vacuum Chamberswithout Circular Symmetry,” Physical Rev. STAccel

erator Beams, vol. 5, Nov. 2002, p. 111001.

13. Y. Alexahin et al., “Observation of Instabilities ofCoherent Transverse Oscillations in the Fermilab

Booster,” Fermilab Conf Proc., 2012, pp.3129—3131.

14. M. Conte and W.W. MacKay, An Introduction to

the Physics ofParticle Accelerators, World Scientific,

1994.

James Amundson is the Synergia Project technical

lead and head of the Scientific Software InfrastructureDepartment at Fermilab National Accelerator Labora

tory. His research focuses on high-performance parallel

computing and its intersection with computational ac

celerator physics. Amundson has a PhD in theoretical

particle physics from the University of Chicago. He’s amember of the American Physical Society. Contact him

at amundson@fnal.gov.

Alexandru Macridin is a computational physicist in the

Scientific Computing Division at Fermilab National Ac

celerator Laboratory. His research interests include ac

celerator physics, beam dynamics, collective effects inaccelerators, numerical methods, parallel computing,

and many-body physics. Macridin has a PhD in phys

ics from the University of Groningen. Contact him at

macridin@fnal.gov.

Panagiotis Spentzouris is head of the Scientific Computing Division at the Fermi National Accelerator Laboratory. His research interests include computationalaccelerator physics and neutrino physics. Spentzouris

has a PhD in physics from Northwestern University.

He’s a member of the American Physical Society. Con

tact him at spentz@fnal.gov.

~ Selected articles and columnsfrom IEEE Computer

Society publications are also available for free at

http://ComputingNow.computer.org.

www.computer.org/cise 41