ARC: Adaptive Ray-tracing with CUDA, a New Ray Tracing ... · rays, making the task well suited to...

MNRAS 000, 000–000 (0000) Preprint 20 July 2018 Compiled using MNRAS LATEX style file v3.0

ARC: Adaptive Ray-tracing with CUDA, a New RayTracing Code for Parallel GPUs

Blake Hartley1 and Massimo Ricotti11Department of Astronomy, University of Maryland, College Park, MD 20742, USA ?

20 July 2018

ABSTRACTWe present the methodology of a photon-conserving, spatially-adaptive, ray-tracingradiative transfer algorithm, designed to run on multiple parallel Graphic ProcessingUnits (GPUs). Each GPU has thousands computing cores, making them ideally suitedto the task of tracing independent rays. This ray-tracing implementation has speedcompetitive with approximate momentum methods, even with thousands of ionizationsources, without sacrificing accuracy and resolution. Here, we validate our implementa-tion with the selection of tests presented in the ”cosmological radiative transfer codescomparison project,” to demonstrate the correct behavior of the code. We also presenta selection of benchmarks to demonstrate the performance and computational scalingof the code. As expected, our method scales linearly with the number of sources andwith the square of the dimension of the 3D computational grid. Our current imple-mentation is scalable to an arbitrary number of nodes possessing GPUs, but is limitedto a uniform resolution 3D grid. Cosmological simulations of reionization with tensof thousands of radiation sources and intergalactic volumes sampled with 10243 gridpoints take about 30 days on 64 GPUs to reach complete reionization.

Key words: cosmology: theory – cosmology: early Universe – cosmology: dark ages,reionization, first stars

1 INTRODUCTION

The propagation of ionizing and dissociating radiation fromstars and black holes and its effect on the interstellarmedium (ISM) and the intergalactic medium (IGM), is oneof the most fundamental and computationally difficult prob-lems in theoretical astrophysics. A number of schemes havebeen implemented for tackling radiative transfer. As thecomputational power available for astrophysical simulationshas increased over the past few decades, the full seven di-mensional (three spatial, two angular, one frequency, onetime) radiative transfer problem has been solved in earnest.There are multiple popular methods for approaching thisproblem, including:

(i) Moment Methods: The first three moments of theradiation intensity, which are the energy density, flux,and radiation pressure, are tracked by the simulation(Auer & Mihalas 1970; Norman et al. 1998; Stone et al.1992). Simulations of this type have been implementedwith short characteristics (Stone et al. 1992), long charac-teristic (Finlator et al. 2009), utilizing the optically thinvariable Eddington Tensor method (Gnedin & Abel 2001;

? [email protected], [email protected]

Ricotti et al. 2002a,b; Petkova & Springel 2009), and with atwo moment model using a closure relation (Gonzalez et al.2007; Aubert & Teyssier 2008). These methods have the ad-vantage of being fast, with computation times that don’tdepend on the number of radiation sources. However, thesemethods are fundamentally diffusive, so that radiation willin some cases flow around occluding regions in non-physicalways, resulting in incorrect behavior for shadows. In addi-tion, since the radiation is treated as a photon fluid withcharacteristic speed equal to the speed of light, the condi-tion for stability of the integration typically requires takingvery small time step (similarly to the CFL condition). A re-duced speed of light approximation is often adopted in orderto be able to take larger timesteps (Deparis et al. 2018).

(ii) Ray-tracing Methods: Radiation from a point sourceis approximated as a collection of linear rays propagatingaway from the point source. Along the rays the heat-ing/ionization of the gas and extinction of the radiation aretracked through a grid (Abel et al. 1999; Razoumov & Scott1999; Ciardi et al. 2001; Sokasian et al. 2001; Alvarez et al.2006; Mellema et al. 2006; Rijkhorst et al. 2006;Whalen & Norman 2006; Krumholz et al. 2007; Trac & Cen2007; Paardekooper et al. 2010) or collection of parti-cles (Susa 2006; Johnson et al. 2007; Altay et al. 2008;Pawlik & Schaye 2008, 2011; Hasegawa et al. 2009). These

c© 0000 The Authors

arX

iv:1

807.

0709

4v1

[as

tro-

ph.C

O]

18

Jul 2

018

2 Hartley & Ricotti

methods are computationally more expensive, as theamount of computation scales linearly with the number ofsources and the number of grid points or gas particles. Theresults, however, are a better approximation of the truesolution.

(iii) Non-simulation methods, such as the excursion setformalism (inside-out reionization, see Furlanetto et al.2004; Alvarez & Abel 2007), inhomogeneous reionizationmodels (outside-in reionization, see Miralda-Escude et al.2000; Wyithe & Loeb 2003; Choudhury & Ferrara 2005),and semi-analytic inhomogeneous reionization methods(Mitra et al. 2015). These methods require much fewer com-putational resources than simulation methods, allowing forthe exploration of a much larger range of simulation pa-rameters. However, they lack the accuracy of computationalmethods.

Our approach is of the second type, based on thephoton-conserving, spatially-adaptive ray-tracing methodoriginally presented in Abel & Wandelt (2002), but designedfor use on parallel Graphics Processing Units (GPUs). GPUshave slower clock speeds than CPUs of the same generation,but they possess thousands of parallel cores which operateindependently of each other. The cost of calculations foreach individual ray is computationally small and indepen-dent of other rays, which makes GPUs ideally suited for thetask. Despite the large ongoing effort to model the epochof reionization (EOR) by various teams, our proposed ap-proach is complementary to previous and ongoing investiga-tions, and has new and unexplored aspects to it as explainedbelow. We are writing our numerical code with the futuregoal of incorporating it in widely used hydrodynamical cos-mological codes. At the moment, the focus of our effort is onray-tracing radiative transfer, meaning we limit ourselves tosimulations with fixed grid (i.e., the adaptive mesh refine-ment (AMR) grid structure is not implemented at this point)and no coupling to hydrodynamics, thus making this code apurely post-processing method on previously run cosmolog-ical simulations. This approximation is not poor when fo-cusing on simulations of cosmological reionization on scalesmuch larger than galaxy-halo scales, which are the focus ofour test runs and future science investigations.

This paper is organized as follows. In Section 2 we de-scribe the underlying physics and computational implemen-tation used by our code. In Section 3 we present the resultsproduced by our code when running the battery of testslaid out in the radiative transfer codes comparison project(Iliev et al. 2006). We also present a selection of benchmarksdemonstrating the speed of the code in Section 4. Finally, inSection 5 we summarize the methods and results presentedin this paper.

2 METHODOLOGY

Our code utilizes the power of GPU processing to attack thecomputationally intensive problem of adaptive ray-tracingradiative transfer. The tracing of a single ray of photonsis a very simple task, but a ray tracing radiative transfersimulation requires the tracking of a large number of suchrays, making the task well suited to GPU computing. Clas-sically, memory sharing between GPUs is only possible ifthose GPUs are all located on the same node of a computer.

This limits the scale of a purely GPU algorithm, and leadus to add MPI parallelization. MPI parallelization allowsus to share the results of GPU calculations between GPUson different nodes, thus allowing for a highly scalable codeand also overcoming the problems associated with the lim-ited memory of a single GPU. In addition to the relativelystraightforward parallelization of radiation from differentsources, we break up the computational volume and there-fore the ray tracing and the ionization/heating calculationsinto equal portions and distribute them among each node ofthe process using MPI. There, the GPU algorithm performsthe calculations, sharing ray data between nodes using MPIas necessary. The results of the calculations are then con-solidated using MPI and the next time step can be taken.An earlier and simpler version of the code shared copies ofthe whole the grid data among nodes and distributed onlythe work from different ionization sources using MPI. Thisway, there was no need for ray data sharing among nodes.However, this method was severely limited by the availablememory of each GPU. With current GPUs, the maximumresolution of reionization simulations we could run withoutbreaking up the volume was between 2563 and 5123 gridpoints. The current code is indeed faster and not limited bythese memory restrictions.

2.1 Program Design

Each GPU in our code calculates the radiation field pro-duced by sources within the sub-volume assigned to thenode which hosts that GPU. In addition each sun-volumecan share the work with multiple GPUs. The direction ofthe rays is assigned using the HEALPix (Gorski et al. 2005)scheme to distribute rays in all directions. The HEALPixscheme assigns to each ray equal solid angle by dividing thesphere around each sources in equal areas. In order to main-tain constant spatial resolution as the rays travel furtherfrom the source, each subsequent level of the scheme breaksthis area into four sub-areas, allowing the algorithm to eas-ily keep track of the splitting of rays. We initialize an arrayof photon packets for each ray in the initial HEALPix array(typically HEALPix level 1, or 48 photon packets, or rays).Each photon packet traced by our code contains the ID ofthe ionizing source which emitted it, the unique HEALPixPID which determines the direction of the photon’s motion,the distance each photon has traveled, and the optical depthof each frequency bin along the photons path. The posi-tion of the ionizing source is stored in the GPU’s sharedmemory, allowing for a reduction of the required per-photonpacket memory. Once the calculation of the radiation fieldsby the GPUs is complete, we combine the radiation fieldsfrom other sources distributed on different nodes linearlyusing MPI. We then divide and distribute all these fieldsto the processes to evolve the gas fields over a single timestep. The algorithm’s overall structure is described in stepsbelow.

(i) Initialize radiation field arrays to zero at the start ofa new time step.

(ii) Use MPI to divide distribute photon source data andsource array to all nodes of the process.

(iii) Loop through each photon source one at a time usingthe GPU kernel as follows:

MNRAS 000, 000–000 (0000)

ARC: Adaptive Ray-tracing with CUDA 3

(a) Initialize all photon packets to radius zero and zerooptical depth in all frequency bins.

(b) Trace all photon packets at a single HEALPix leveluntil the ray moves to an adjacent sub-volume, terminatesor splits. If the photon moves into an adjacent sub-volume,we store it in a sharing buffer to be sent later. If the photonterminates, set the radius of the packet to zero, signalingthat it should be ignored in future calculations. Otherwise,leave the radius and optical depth at the point of splittingunchanged.

(c) Create a new array of child rays populated by theunterminated rays from the previous step. The radius ofall four child rays is the same as the parent rays, but thenewly populated array have directions based on the nextlevel of the HEALPix scheme.

(d) If any rays are unterminated, return to step (b).(e) Use MPI to share rays between adjacent sub-

volumes, and return to step (a).

(iv) Use MPI to linearly combine radiation fields pro-duced by all of the kernels, thus giving us the overall ra-diation field produced by all the sources.

(v) Apply the ionization/heating calculation to the cur-rent radiation field and gas fields to calculate the changes tothe grid over a single time step.

(vi) Use the execution times of each GPU to redistributethe sources between GPUs handling the same sub-volumeto balance the computational load.

(vii) Return to (i) until the simulation reaches the desiredtime.

Each node and corresponding GPU loops through thiscode to advance the simulation. We can also send the samesub-volume to multiple GPU’s, allowing us to break a setof sources in the same sub-volume between multiple GPUs.When any timing information about the entire simulation,or outputs of the full grid, are required, MPI is used toconsolidate the data from all of the nodes.

In the next subsections we describe in more detail theimplementation of the physics and equations solved by eachmodule of the code.

2.2 Photon Data

Photon packets in our code are polychromatic. As each ray istraced through the grid, we calculate the total optical depthdue to all atomic species (hydrogen and helium) and theirions, for each frequency bin. Each bin is a single frequencywhich represents the frequency averaged ionization cross sec-tion of all present species. We are able to track an arbitrarynumber of frequency bins, subject to memory limitations.

2.3 Grid Data

The basis of our code is a uniform 3-dimensional cubicgrid of N3 cells which store the neutral/ionized fraction,temperature, and photoionization rate of a given set ofatomic/molecular species as a function of time. The cur-rent version of our simulation tracks the neutral fractionsxH i and xHe i (future versions will include tracking of morespecies as required). We assume that helium is either neutralor singly ionized over the course of the simulation. Thus, wecalculate the electron number density as:

ne = nb

[1− YpAH

xH ii +YpAHe

(xHe ii + 2xHe iii)

],

where nb ≡ np + nn is the baryon number density, Yp ≡ρHe/ρb is the Helium fraction, AH = 1 and AHe = 4 are theatomic weights of hydrogen and helium, respectively.

We are able to sub-divide this volume as desired. Thefirst science runs of our code will divide the volume evenlyinto 8 cubic regions, each of which will be handled by asubset of the available GPUs. The current implementationof the code is limited to a fixed grid; however, the structureof the code would allow for a transition into an adaptivemesh refinement scheme (AMR) should it be required in thefuture.

2.4 Photon Transmission

Ionizing sources within our volume are represented by a setof photon packets, or rays, which are initialized at the loca-tion of the source. These rays are traced through the gridof our simulation one step at a time, with each step cor-responding to the distance through the volume it takes toreach the next X, Y, or Z cell boundary within the volume.Each ray is initialized with an ionizing photon productionrate (Sν [s−1] for each frequency bin ν) which correspondsto the luminosity of the source divided between the rays.For each step, we increment the total optical depth alongthe ray τν for each frequency bin by ∆τν , the optical depthof ray within a single simulation cell. We use this opticaldepth to calculate the fraction of ray’s photons absorbed bygas within the cell, and thus calculate the ionization rate ofthe cell for each frequency bin ν per absorbing atom:

sν =Sν(1− e−∆τν )

V nabs, (1)

where nabs is the physical number density of absorbers, andV is the volume of the simulation cell. More specifically, inour code nabs is the neutral number density nSI of a speciesS, where S refers to hydrogen and helium. The flux alongthe ray in each frequency bin is reduced by this amount,so that this method conserves the total number of photons.Each ray in the simulation is traced by a single CUDA core,with the work distributed as evenly as possible by the CUDAkernel. Each GPU contains all of the grid data necessary totrace the rays, and the ionization rate at a given cell in thegrid is altered ”atomically” by each core, eliminating thechance for interference between simultaneous processes.

The directions of the rays are chosen according to theHEALPix scheme (Gorski et al. 2005), which assures thateach ray is assigned an equal solid angle relative to thesource. This means that we can assign a photon emissionrate for each ray by dividing the overall fesc,νSν of the sourceevenly between all rays (where fesc,ν is the mean ionizing ra-diation escape fraction from each source), or we can adopta more sophisticated (and realistic) scheme in which the es-cape fraction is anisotropic. We can, for instance, obtain thewanted escape fraction by only assigning the mean Sν per-ray, to a fraction fesc,ν of the rays, while completely blockingthe radiation escaping from the remaining rays.

Each ray is traced independently by a CUDA core untilit reaches a set distance from the source, at which point the

MNRAS 000, 000–000 (0000)

4 Hartley & Ricotti

Figure 1: Schematic example of the adaptive ray tracingmethod based on HELPix. As rays propagate outwards, theysplit to maintain a constant number of rays per unit areacrossing the surface of the sphere centered on the source. Theleft and right trees illustrate the difference between assumingat least 3.0 and 1.0 rays intersect a given simulation cell forthe left and right ray trees, respectively.

ray is split into four child rays corresponding to the nextlevel of the HEALPix scheme. The distance at which theray is split is a free parameter which allows us to controlhow many rays pass through the average simulation cell ata given distance, giving us control over the spacial accuracyof the ray tracing method. In Figure 1 we plot a schematic ofthe adaptive ray tracing process. The two trees represent theprogress of a single pixel at the lowest HEALPix level tracedout through three branches. The two trees demonstrate thedifference between thresholds for ray splitting, with the lefttree guaranteeing three times as many pixels through a givencell than the right tree.

It is important to emphasize that, as mentioned above,HEALPix method gives us the ability to control to any de-sired accuracy the directions into which the ionizing sourceemits photons, allowing simulations in which ionizing pho-tons leak out of a galaxy anisotropically. Numerical sim-ulations of galaxy formation have shown that ”chimneys”of low optical depth through which most ionizing photonsescape into the intergalactic voids are a more realistic de-scription of the radiation escaping into the IGM than anoverall isotropic attenuation of the emission. We can con-trol these directions on a source by source basis, giving us aversatile tool to study different types of sources in differentcosmic environments. This desirable feature of the code isonly possible because the ray tracing method is non-diffusiveand can capture shadows accurately. The faster ”radiationmoments” methods discussed in the introduction, are prob-ably too diffusive to allow the implementation of anisotropicemission from the sources.

2.5 Geometric Correction

The rays in our code are one-dimensional lines which repre-sent a cone of radiation extending from a point source, orfrom a splitting ray, with a fixed solid angle. The intersec-tion of the ray volume with a given grid cell is a complicatedgeometric shape whose volume is impratical to calculate ex-actly. The ray tracing method approximates this volume asa truncated cone created by the segment of the ray whoselength is the distance between the points where the ray en-ters and exits the grid cell. Thus, when the ray remainscloser to the edge of the grid cell than the radius of the ray(along the length of the ray), a portion of the ray remainsentirely outside of the grid cell. We make a first order cor-rection to this problem following the method presented inWise & Abel (2011). We let LPix be the width of the pixeland let Dedge be the distance from the midpoint of the seg-ment of the ray within the cell to the nearest edge of thecell. We reduce the ionization within the cell by a factor fcdefined by:

fc =

12

+Dedge

LPixDedge < LPix/2,

1 Dedge > LPix/2.(2)

This correction factor is slightly different from the oneused in Wise & Abel (2011) in which (1/2 +Dedge/LPix) issquared. We remove the square from the first expression, sothat fc = 1/2 when Dedge = 0, which physically correspondsto half of the ray lying within the grid cell when the ray trav-els along the edge of the cell. However, this correction factoronly serves to reduce the ionization rate in a given cell, andthus has the effect of reducing the global ionization rate inthe simulation and breaks photon conservation, that is oneof the most desirable properties of the method. We find thatadopting this correction makes a non-negligible reduction tothe volume filling fraction in the simulation, which is morepronounced at the points where the rays split (as may beseen in the radial profiles of Wise & Abel (2011) Figure 5and Figure 6).

We decided to keep the correction factor as it miti-gates small spatial artifacts in azimuthal directions (see Sec-tion 3.6), but we also compensate in order to maintain pho-ton conservation by adding the ionization removed by thecorrection factor to the nearest cell using a secondary cor-rection factor:

f ′c = (1− fc)xabs

x′abs

, (3)

where xabs and x′abs are the absorber fraction in the cell in-tersected by the ray, and in the nearest adjacent cell to theray, respectively. Here we have multiplied by the ratio of thedensity of absorbers between the cells, as the ionization ratein Equation 1 is an overall absorption rate, and needs to becorrected in case the the densities of absorbers vary betweenthe cells. This second correction factor is another slight dif-ference with respect to the method used in Wise & Abel(2011), but we find it very beneficial. The combination ofthese correction factors corrects for geometric artifacts whilemaintaining photon conservation and the correct volume fill-ing factor of the H ii regions (see Section 3.6). This secondarycorrection relies on the regularity of the Cartesian grid, andin the case of non-regular grids or AMR it would need to begeneralized or removed.

MNRAS 000, 000–000 (0000)


2.6 Optically Thin Approximation

In regions of low neutral fraction surrounding active sourcesof ionization, the ray tracing process tracks an optical depthwhich is almost unchanging over the grid cells. This allowsus to implement a procedure for calculating the ionizationwithin a certain volume without tracing rays, and thus startthe ray tracing process at a greater distance and save signif-icant computation time. Our optically thin approximationproceeds as follows:

(i) To each ionizing source within the simulation we as-sign a radius Rτ<0.1, which we initially set to zero.

(ii) As rays are traced from the ionizing source, we setRτ<0.1 to be the minimum of all the ray lengths for whichτν < 0.1 in the softest frequency band.

(iii) When Rτ<0.1 > 0, we assume all cells within a dis-tance of Rτ<0.1 of the ionizing source to be directly exposedto the ionizing source without absorption. We approximatethe ionization rate as:

sν ≈Sνxabsσabs(ν)

4πR2,

which is the approximation of Equation 1 for small τν andreplacing the dilution of spreading rays with a 4πR2 term.

(iv) We initialize rays at a radius of Rτ>0.1. This allowsus to skip ray tracing within the optically thin region.

This approximation is trivial at early times in simula-tion, when Rτ>0.1 is small for most sources, but the amountof calculation is then also small. However, when the averageneutral fraction of the simulation decreases, this approxi-mation becomes more and more efficient, saving significantcomputational resources. Beginning the rays outside of thesethese highly ionized regions decreases the ray tracing com-putation time by a factor of the order of the mean neutralfraction in the simulation box, while the calculation itself isorders of magnitude faster than the ray tracing module.

2.7 Ionization Calculation

Each GPU in our code loops through all sources it is as-signed using the procedure detailed in the previous section.Once all the GPUs have completed their assigned work, theyreturn the radiation array which represents the ionizationrate due to all of the sources assigned to that GPU. Thehost node then sums these arrays using MPI, giving us theoverall ionization rate at each point in the grid due to allsources in the simulation.

Once the ionization rate, Γν , is calculated, we computethe change in the neutral density, nSI of species S of the gasaccording to the differential equation:

nSI = −nSI∑ν

sν − CS(T )nenSI + αS(T )nenSII , (4)

where CS is the collisional ionization rate, αS(T ) is the re-combination rate, ne is the electron number density and nSIIis the ionized number density of species S.

The ray tracing algorithm is computationally the mostexpensive part of the simulation, so we have designed thecode to limit the number of calls to the ray tracer as much aspossible. We solve the non-equilibrium chemistry and energyequations for all species under consideration sub-cyclingbetween the radiative transfer time steps (Anninos et al.

1997). The non-equilibirum chemistry equations are stiff or-dinary differential equations (ODEs). Thus, for individualsteps in the sub-cycle, we tested several integration methods,including predictor-corrector methods, Runga-Kutta meth-ods, semi-implicit methods, backwards difference, etc. Wefound that the backwards difference formula (BDF) gavethe best combination of speed, accuracy, and computationalstability in our tests, in agreement with the results presentedin (Anninos et al. 1997). Effective use of the BDF requireswriting the non-equilibrium chemistry equations in the form:

nSI = D − C · nSI , (5)

where D represents source terms which do not depend onnSI (to the first order) and C represents sink terms whichare linear in nSI (to the first order). The photon conserv-ing method we use (Equation 1) calculates the photo ion-ization rate per absorbing atom within a cell at a givenneutral fraction. In order to use Equation 5, we must as-sume that the photo-ionization (sν) and heating per ab-sorber within the cell, which depend on (1 − e−∆τ )/nSI , iseither independent of the neutral fraction or is ∝ 1/nSI ,so that nSIsν ∼ const. Since ∆τ is proportional to nSI, weare effectively treating single cells as having either ∆τ 1(i.e., 1 − e−∆τ ∝ ∆τ ∝ nSI) or ∆τ 1, over a set ofsub-cycles. We note that where this approximation is lessaccurate, the cells ionize faster than in the accurate case,becoming rapidly optically thin and transitioning into themore accurate regime. We also have the option of treatingthe cells as optically thick in cases of particularly high den-sity. However, in the bulk of the cases we consider, high den-sity cells at the ionization front are pre-ionized by photonswith a long mean-free-path because of a spectrum which hasbeen hardened by absorption along the ray. In summary, theoptically thin approximation is almost always sufficiently ac-curate.

We choose sub-cycle time steps for the ioniza-tion/energy ODEs to limit the fractional change in anyquantity which our equations is tracking, including the en-ergy of the cell. Thus:

dt = min

(εE

|dE/dt| ,εnH i

|dnH i/dt|

).

The choice of ε is free; we find that a choice ofε = 0.1 following the convention of similarly written codes(Anninos et al. (1997), Wise & Abel (2011)) gives a goodbalance of speed and accuracy.

The solution of the ODEs for ionization/heating is alsoparallelized. The computational volume is divided into sub-volumes and distributed between MPI processes, with multi-ple copies distributed to independent processes when thereare a large number of sources. Each process then calls aCUDA kernel to assign these cells to the CUDA cores ofeach GPU. While the processes operate independently, MPIis used to consolidate data as necessary between ray tracingsteps.

Once the calculation in each process is complete, thesub-volumes are recombined in the host process and redis-tributed between all processes. In some cases, the time re-quired to distributed and recombine the grid data may belarger than the time required to actually perform the fullionization calculation; in these cases, we are also able toperform the ionization calculation locally in each process,

MNRAS 000, 000–000 (0000)

6 Hartley & Ricotti

Figure 2: Cooling function Λ(T ) (black solid line) as a func-tion of temperature for a gas of primordial composition(Hui & Gnedin 1997). The solid lines represent the contri-bution to the cooling from different processes for hydrogenand the dashed lines for helium as shown in the legend.

eliminating the need to send any data between the processesduring this step of the simulation.

2.8 Heating and Cooling

The heating of gas within each cell is calculated at the sametime ionization rate is calculated. The energy per unit vol-ume and per unit time added to each cell by a given ray iscalculated based on the ionization rate:

Eν =∑

S

h(ν − νS)sνnSI ,

where νS is the ionization edge frequency of the species be-ing tracked. We then use the available physical quantities(such as temperature, free electron density, ionized/neutralhydrogen/helium density) to calculate the cooling rate for avariety of physical processes, including collisional ionization,recombination, collisional excitation, bremsstrahlung, andadiabatic expansion due to the Hubble flow (Hui & Gnedin1997). In Figure 2, we plot the cooling function we use inour simulations. In order to save memory, the simplest ver-sion of the code only tracks hydrogen and singly ionizedhelium fractions, meaning we assume a soft spectrum of ra-diation (stellar spectrum) in which He iii number densitiesis negligible. In our initial simulations, all of the sourcesare initialized with spectra corresponding to stellar sources,so that temperatures above 2 × 104 K is rare, and we thusexpect our cooling rates to be accurate. We initially omittracking He iii and primordial H2 chemistry to save on com-puter memory; however, we have the ability to include otherrelevant chemistry, for instance when we include sources ofX-rays and Population III stars that emit harder photonsand H2 dissociating radiation (cite).

2.9 Radiative Transfer Time Step

The previous section describes how we dynamically sub-cycle the time integration between radiative transfer calls;

however, the choice of time step between calls of the ra-diative transfer routine is more difficult. The radiation fielddoes not evolve between ray tracing calls, so that the veloc-ity of the I-front is limited by the choice of this time step.This means that if the time step is chosen to be too large,the speed of the I-front is unphysically reduced. However,the I-front still approaches the correct asymptotic solutionat large times regardless of choice of time step.

The ray tracing step is by far the most computationallyexpensive part of the simulation, so we wish to maximizethe time step between ray tracing calls while keeping the re-quired accuracy. In cases where the velocity of the ionizationfront is not important, we can opt to use a constant timestep chosen based on physical considerations. In cases wherehigher accuracy is required, we have implemented two dif-ferent schemes for determining the optimal time step adap-tively:

(i) Minimum neutral fraction change: The simplestmethod for adaptively correcting the time step is regulatethe maximum rate at which the neutral fraction in any givencell changes. For a single cell:

dtn =εnH i

|dnH i/dt|,

where ε is the maximum fractional change in the neutralfraction. We calculate this quantity for every cell within thecomputational volume and calculate the minimum time step.We find that this method constrains the time step to beunnecessarily small in regions with small neutral fraction,where a large fractional change in neutral fraction resultsin a negligible change in the neutral density. We thus addthe constraint to only consider cells with a relatively highoptical depth:

τ = nH iσid > 0.5,

where d is the width of the cell and σi is the cross sectionof the lowest frequency bin. This condition also limits us tocells near the ionization front.

(ii) Minimum intensity change: Our second schemefor adaptively controlling the time step is to regulate themaximum rate at which the photon flux changes in a singlecell. Similar to above, we define the time step for a givencell as:

dtI =εIν

|dIν/dt|,

Where Iν = A exp(−τν). We calculate this expression atruntime by using the following simplification:∣∣∣∣dIνdt

∣∣∣∣ =

∣∣∣∣ ddtA exp(−τν)

∣∣∣∣=

∣∣∣∣A exp(−τν)dτνdt

∣∣∣∣=

∣∣∣∣IνσiddnH i

dt

∣∣∣∣ ,dtI =

ε

σid|dnH i/dt|.

We find that this method gives significantly larger timesteps than the previous methods, with a minimal loss ofaccuracy when the ionization rate is very high.

We note here that for a given choice of time step, smaller

MNRAS 000, 000–000 (0000)


H ii regions are underrepresented compared to larger H ii re-gions. We also note that the choice of time step makes muchless of a difference in regions without radiation (as chem-istry subcycles accurately solve the equations) and whenH ii region overlap becomes prevalent (as most H ii regionsare large and I-front velocities are low).

2.10 Ionizing Background

As the universe expands and the average electron fraction ofthe universe increases, the average density of neutral hydro-gen decreases and the mean free path of photons in the IGMincreases. These photons build up a ionizing backgroundwhich becomes more dominant as more ionizing sources ap-pear. The harder photons of the ionizing spectrum build upa background earlier than the softer photons due to theirlonger mean free path. Individual halos never produce largeenough regions where the gas is fully ionized to completelyreionize the cosmic volume, so the derived average electronfraction underestimates the true electron fraction. We cor-rect for this underestimation by calculating and includingin the photon budget the ionizing background and its ef-fect on the cosmic ionization history. We quantify this effectsolving the equation of radiation transfer in an homogeneousexpanding universe as in Gnedin (2000a); Ricotti & Ostriker(2004), which we briefly summarize here. We begin with thenumber density of ionizing background photons nν at a red-shift z and evolve it to z−∆z. During each code timestep ∆z,we add to the initial background at redshift z (appropriatelyredshifted and absorbed by the neutral IGM) the photonsproduced by ionizing sources within our simulation betweenthe redshifts of z − ∆z0 and z − ∆z including absorptionand redshift effects (source term), where z−∆z0 representsthe redshift at which we begin adding the contribution of thesources to the background radiation. This parameter is used,as explained later, to avoid double counting the emissionfrom low redshift (local) sources, that is already included inthe radiation transfer calculation. Mathematically, we solvethe equation:

nν(z −∆z) = nν(z) exp

[−∫ z−∆z

z

dz′αν′(z′)

]+

∫ z−∆z

z−∆z0

dz′Sν′(z′) exp

[−∫ z−∆z

z′dz′′αν′′(z

′′)

], (6)

where ν′ = ν(1+z′)/(1+z) we have defined a dimensionlessabsorption coefficient and source function:

αν =(1 + z)2

H(z)cnHσν(H i)(1− xe(z)), (7)

Sν =nion〈hν〉gν/hν(1 + z)H(z)

, (8)

where the sources spectra are normalized as∫∞ν0gνdν = 1,

with hν0 = 13.6 eV and 〈hν〉−1 ≡∫∞ν0

(gν/hν)dν. We use thisequation to calculate two quantities which we track through-out the simulation:

(i) The overall background nallν (z−dz) is calculated using

∆z0 = 0, so that we include background and local radiation.This quantity is used as nν(z) in Equation 6 for the nexttime slice.

(ii) The mean local radiation emission nlocν (z− dz) is cal-

culated assuming ∆z0 = H0R0/c, where R0 represents thesize of the simulation box. By combining the local radiationbackground (from stars inside the box but within ∆z) withthe overall background nall

ν from previous slices, we find thebackground radiation from all stars at higher redshifts inthe simulation without double counting the contribution ofsources inside the box.

2.11 MPI/CUDA Parallelization

The N3 uniform grid of our full simulation represents a sim-ple first application of our ray tracing algorithm. In the firstversion of ARC (v1) every GPU requires the full grid data toperform the ray tracing calculations, which limits the possi-ble size of the simulation grid. In the current version (v2),if the grid required for the simulation is larger than can bestored on a single GPU, the code allow the code to break thevolume into smaller regions. When rays reach the edge of asub-volume, they are sent to the adjacent sub-volume intowhich they move, much in the same way as AMR methodsperform the calculation. This means our method is appli-cable in AMR settings, though our current iteration of thecode is limited to Cartesian grids which may be subdivided.Regardless of these considerations, GPUs excel at perform-ing these ray tracing calculations as long as the number ofrays within a single computational volume is large and theCUDA core occupancy is near optimal (> 104).

3 RADIATIVE TRANSFER TESTS

Consistency tests are paramount in demonstrating that anew radiative transfer code produces results that are con-sistent with well established methods or scenarios that haveexact analytical solutions. In this section we present the re-sults produced by our code when running the tests presentedin (Iliev et al. 2006) (RT06). These tests include 0) Trackingchemistry in a single cell 1) expansion of an isothermal H iiregion in pure hydrogen gas 2) expansion of an H ii regionin a pure hydrogen gas with evolving temperature 3) I-fronttrapping and the formation of a shadow 4) multiple sourcesin a cosmic density field. We show that our code performswell when compared to CPU codes that use similar ray-tracing methods. The various codes used for comparison inRT06 show a good deal of variability for several of the tests.The method implemented in our code most closely resemblesC2-Ray (Mellema et al. 2006) and MORAY (Wise & Abel2011). Where possible we compare our results to the pub-licly available RT06 results for C2-Ray. Since at the momentour ray tracing algorithm is used in a post-processing set-ting, it neglects hydrodynamic evolution. We therefore limitourselves to tests which do not require accurate tracking ofthe hydrodynamic properties of the gas.

3.1 Test 0 - Chemistry in a Single Cell

The test presented in RT06 applies a constant radiationfield of plane-parallel radiation to a single cell of the sim-ulation. Our code is limited to tracking point sources ofradiation, so we place a single source of photons at oneside of a 6.6 kpc cube with 1283 cells. The source is given

MNRAS 000, 000–000 (0000)

8 Hartley & Ricotti

6

4

2

0

log 1

0(HI

)

103 104 105 106 107

T (yr)

104

6 × 103

2 × 104

3 × 104

4 × 104

T(K

)

Figure 3: Test 0 - Chemistry in a single cell. In this test, asingle cell is subject to intense radiation for 5 × 105 yr, atwhich point the radiation stops. (Top.) Plot of the neutralfraction (solid line) and ionized fraction (dashed line) as afunction of time. (Bottom.) Plot of the temperature as afunction of time. These results agree well with the resultspresented in RT06, except at the first time step, which isexpected given our use of a fixed time step.

a luminosity 5.2 × 1057 photons s−1 so that the flux atthe cell is 1012 photons s−1 cm−2. The density of thecell is n = 1 cm−3 and it is initially neutral at temper-ature T = 100 K. The radiation is approximated to bea blackbody of temperature 105 K; following Wise & Abel(2011), we use four frequency bins with central energiesEi = (16.74, 24.65, 34.49, 52.06) and relative luminositiesLi/L = (0.277, 0.335, 0.2, 0.188). The radiation is appliedfor 0.5 Myr, after which the radiation is turned off and thecell is tracked for 5 Myr.

In figure 3 we plot the neutral fraction and temperatureof the cell as a function of time. We find that these resultsagree with those presented in RT06. The only discrepancybetween our results and those presented in RT06 is in theelectron fraction for the first time step. This is a result ofthe optically thin approximation for sub-cycles. The first raytracing calculation occurs at xH i = 1, at which point the cellabsorbs more soft photons, so that the chemistry calculatordoesn’t reach the correct equilibrium point. After this step,however, the optically thin approximation is satisfied andthe solution agrees from the second step on.

3.2 Test 1 - Expansion of Isothermal H ii region inPure Hydrogen

The most fundamental test for a radiative transfer code isthe simulation of an expanding H ii region around a constantluminosity source in a constant density medium of pure hy-drogen Assuming the medium is initially neutral and thatthe H ii region has a sharp boundary, we have the well known

analytic solution:

rI(t) = rS

(1− exp

(t

trec

))1/3

, (9)

vI(t) =

(rS

3trec

)exp(t/trec)

(1− exp(t/trec))2/3, (10)

where:

rS =

(3N

4παB(T )n2H

)1/3

,

trec = [αB(T )nH ]−1.

The test domain is a 6.6 kpc cube with a 1283 elementcubic grid. The gas has a fixed density nH = 1.0×10−3 andtemperature T = 104 K and an initial equilibrium ionizationfraction of 1.2×10−3. We place a single ionizing source in thecorner of the simulation box, and assume a monochromaticspectrum with energy hν = 13.6 eV and luminosity of S0 =5 × 1048 photons s−1. The chosen physical parameters givea Stromgren radius RS = 5.4 kpc and a recombination timetrec = 122.4 Myr. We track the evolution of this simulationfor 500 Myr, or roughly four recombination times.

In Figure 4 we plot a slice through the computationalvolume at t=500 Myr (top left). This panel shows that thecode produces a spherically growing region of ionization, asexpected. In the top-right panel of Figure 4 we plot the ra-dially averaged profile of the H i fraction as a function ofradius at four times throughout the simulation. The shapeof this profile agrees well with the results of RT06. In thebottom-left panel of Figure 4 we plot the comparison of oursimulation with the model in Equation 9. We see that ourmodel agrees with the analytic fit within 5% for the major-ity of the simulation. The underestimation of the radius atearly times is a result of using a fixed time step for thesetests; with an adaptive time step, the agreement is muchbetter (see Figure 11). Finally, in the bottom-right panelwe show the histogram of H i fraction for three times in thesimulation (solid lines). The dashed lines represent the sameplots produced using the C2-Ray data from RT06. We seethat our code produces the expected results for this test.

3.3 Test 2 - Expansion of H ii Region in PureHydrogen with Evolving Temperature

The second test is similar to the first test, but with the rein-troduction of heating and cooling processes to the simula-tion. We give the source a blackbody spectrum with T = 105

K, using the same spectrum and luminosity as Test 0. Thegas is given an initial temperature T = 100 K.

In the top and middle left panels of Figure 5 show a slicethrough the neutral fraction at t = 10 Myr and t = 100 Myr,respectively. We see that the boundaries of the H ii region inthese plots is less sharp than those of Fig. 4, as anticipatedwith the harder radiation present in the T = 105 K blackbody spectrum. We plot t = 100 Myr instead of t = 500 Myrfor this test because the edge of the H ii region and heatingreaches the boundary by then, meaning the plots show lessrelevant information. In the top and middle right panels ofthe same figure, we plot a slice through the temperature atthe same times to show how the radiation is able to heat at alarger radius than the ionization front. In Figure 6 we showthe respective radial profiles of the neutral fraction (top-left)

MNRAS 000, 000–000 (0000)


0 1 2 3 4 5 6r (kpc)

10 4

10 3

10 2

10 1

100

HI

t = 10 Myrt = 30 Myrt = 100 Myrt = 500 Myr

0

2

4

6

r(kp

c)

0 100 200 300 400 500T (Myr)

0.90

0.95

1.00

1.05

1.10

r sim

/r ana

lytic

0.0 0.2 0.4 0.6 0.8 1.0HI

10 4

10 3

10 2

10 1

100

Prob

abilit

y di

strib

utio

n

t = 10 Myrt = 100 Myrt = 500 Myr

Figure 4: Test 1 - Expansion of an isothermal H ii region in a gas of pure hydrogen. (Top-left.) Slice plots of the neutralfraction at 500 Myr into the simulation. The size and shape of the H ii region are in good agreement with the results in RT06.(Top-right.) Radially averaged neutral fraction as a function of distance from the source, in kpc. The dashed lines are thesame plots reproduced from RT06 using C2-Ray. (Bottom-left.) Ionization front plotted along with the analytical model (top)and the ratio of these models (bottom). We note that the simulation deviates from the analytic solution at the beginningand end of the simulation. This is a result of the soft 13.6 eV spectrum being the worst case scenario for our optically thinapproximation for non-equilibrium chemistry calculation. (Bottom-right.) Histogram of neutral fractions at t = 10, 100, 500Myr. The dashed lines are the same plots reproduced from RT06 using C2-Ray. We see that our model matches C2-Ray verywell.

and the temperature (top-right). We plot our results (solidlines) against the RT06 results for C2-Ray (dashed lines) forcomparison. In contrast to Figure 4, we see that the neutralfraction increases more gradually, in good agreement withthe results of C2-Ray. We note that our model heats thegas outside of the Stromgren sphere more than the C2-Rayresults from RT06. This is a result of the difference betweenthe blackbody spectrum from Wise & Abel (2011) and theone used in RT06.

In the bottom-left panel of Figure 6 we show the growthof the radius of the Stromgren sphere as a function of time.We see that the radius of the ray tracing model begins lag-ging behind the analytic model while the gas is being heated,and later moves beyond the analytic model once the gas is

heated beyond 104 K. These results are in good agreementwith the models presented in RT06. Finally, the bottom-right panel of Figure 6 shows the histogram of neutral frac-tions (left plot) and temperatures (right plot) at t = 10, 100,and 500 Myr (solid lines) against the C2-Ray results (dashedlines). Again, we see the neutral fractions are in excellentagreement, while our model heats the lower temperaturegas slightly more, due to the adoption of a harder spectrumthat was chosen to match the one adopted in Wise & Abel(2011).

MNRAS 000, 000–000 (0000)

10 Hartley & Ricotti

Figure 5: Test 2 - Expansion of H ii region in a gas of pure hydrogen with evolving temperature. (Top and bottom left.) Sliceplots of the neutral fraction at 10 Myr and 500 Myr into the simulation, respectively. (Top and bottom right.) Slice plots of thetemperature at 10 Myr and 500 Myr into the simulation. The adopted 105 K blackbody spectrum introduces harder photonswhich heat the gas to a much larger radius than the size of the Stromgren sphere, in agreement with the results presented inRT06.

3.4 Test 3 - I-front Trapping and Formation of aShadow

Test 3 in RT06 is designed to test the diffusivity and angularresolution of the radiative transfer code. In this test, a fieldof uniform radiation is projected towards a dense sphereof uniform hydrogen surrounded by a very thin medium.We plot the results of our model in Figure 7. The I-frontpropagates at a constant velocity towards the clump until itreaches the surface, at which point the optically thick clumpbegins slowly absorbing the radiation. The lines of sightwhich pass through the sphere are trapped in the clump,causing a shadow to form behind the clump. The sharpnessof the edge of the shadow is a measure of the diffusivity ofthe method. The edges of the sphere become ionized beforethe rest of the clump, causing the shadow to shrink; thisallows to visually assess the angular resolution of the code.

The test also tracks the rate at which the I-front progressesthrough the clump.

The original test presented in RT06 is contained withina 6.6 kpc box with resolution 1283. The ambient medium hasa density nout = 2× 10−4 cm−3 and temperature Tout,init =8000 K, while the clump has a density nclump = 0.04 cm−3

and temperature Tclump,init = 40 K. The clump has a ra-dius 0.8 kpc and is centered at (xc, yc, zc) = (97, 64, 64) ingrid units. The original test assumes plane-parallel radiationwith flux 106 s−1 cm−3. However, as in Test 0, we followWise & Abel (2011) and replace the plane parallel radiationwith a single point source of radiation opposite the clump,with (xc, yc, zc) = (0, 64, 64). The luminosity of the sourceis set at S0 = 3 × 1051 photons s−1, so that the flux at thecenter of the clump matches the plane parallel flux of theoriginal test. The simulation is evolved for 15 Myr, at which

MNRAS 000, 000–000 (0000)


0 1 2 3 4 5 6r (kpc)

10 4

10 3

10 2

10 1

100

HI


0 1 2 3 4 5 6r (kpc)

102

103

104

105

T(K

)


0

2

4

6

r(kp

c)

0 100 200 300 400 500T (Myr)

0.8

0.9

1.0

1.1

1.2

r sim

/r ana

lytic

0.00 0.25 0.50 0.75 1.00HI

10 4

10 3

10 2

10 1

100

Prob

abilit

ydi

strib

utio

n

t = 10 Myrt = 100 Myrt = 500 Myr

102 103 104

T (K)

10 4

10 3

10 2

10 1

100

Figure 6: Test 2 - Expansion of H ii region in a gas of pure hydrogen with evolving temperature. Radially averaged neutralfraction (top-left) and temperature (top-right) as a function of distance from the source, in kpc. The shaded thickness ofthe solid neutral lines represents the variance of the radial average. The dashed lines in both plots represent the same plotsreproduced from RT06 using C2-Ray. The slight disagreements are a result of a different assumption on the spectrum of thesource (monocromatic vs blackbody spectrum) between our code and the C2-Ray run in RT06. (Bottom-left.) Ionization frontplotted along with the analytical isothermal-model (top) and the ratio of these models (bottom). This model deviates morethan in test 1, a result of the non-isothermal nature of the simulation. (Bottom-right.) Histograms of the neutral fraction andtemperature within the simulation, respectively. The dashed lines represent the same results calculated from C2-Ray datafrom RT06. We find that our code produces slightly higher temperatures than C2-Ray, a result of the difference assumptionon the source spectrum between our code and the C2-Ray run used in RT06.

point RT06 find that the I-front is just past the center ofthe clump.

The top and middle left panels in Figure 7 show theneutral fraction in a slice through the center of the volume att = 1 Myr and t = 15 Myr, respectively. The correspondingpanels in the top and middle right show the temperaturefor the same slice and times. We notice immediately thatthe shadows in our simulation are opening with distance,a result of the relatively small distance of the point sourcefrom the clump, as opposed to the plane parallel radiationin the original test. We also see that the diffuse gas outsidethe clump is immediately ionized and photo-heated to thepoint of becoming optically thin. We also see that edge of

the shadow is very sharp, with the neutral fraction goingfrom ∼ 1 to ∼ 10−4 in the space of ∼ 1− 2 cells.

The bottom panels in Figure 7 show the neutral frac-tion (bottom-left) and temperature (bottom-right) withinthe clump along the ray passing trough the center of theclump. The clump is centered at r/Lbox = 0.75 and extendsbetween r/Lbox ∼ 0.6 and r/Lbox ∼ 0.90. We see that thephoto-heated gas outside the clump reaches values above3× 104 K, while the photo-heated gas inside the clump hasa temperature below 2 × 104 K, a result of the increasedcooling at higher gas density. We see that the neutral frac-tion rises and the temperature falls as we move through theI-front. The neutral fraction reaches 50% at r/Lbox ∼ 0.8, aresult consistent with RT06. For comparison we also plot the

MNRAS 000, 000–000 (0000)


0.60 0.65 0.70 0.75 0.80 0.85 0.90r/Lbox

10 4

10 3

10 2

10 1

100

HI

t = 10 Myrt = 30 Myrt = 150 Myr

0.60 0.65 0.70 0.75 0.80 0.85 0.90r/Lbox

101

102

103

104

105

T(K

)

t = 10 Myrt = 30 Myrt = 150 Myrt = 150 Myrt = 150 Myrt = 150 Myr

Figure 7: Test 3 - I-front trapping and formation of a shadow. (Top and middle left.) Slice plots of the neutral fraction at 1 Myrand 15 Myr into the simulation, respectively. (Top and middle right.) Slice plots of the temperature at 1 Myr and 15 Myrinto the simulation. The shape of our shadow region expands with distance, a result of the point-like nature of the source ofionizing photons. Our results agree well with Wise & Abel (2011), a simulation which uses the same point-like configuration.Neutral fraction (bottom-left) and temperature (bottom-right) along a ray going through the center of the dense clump. Thedashed lines represent the same plots calculated using C2-Ray using data from RT06. We interpret the discrepancy in theright figure as the result of using a point source, as in our simulation, rather than a plan-parallel front. The diverging raysheat more effectively closer to the source (low r) and less effectively further from the source (high r).

MNRAS 000, 000–000 (0000)


0.0

0.2

0.4

0.6

0.8

1.0

clum

p

0 2 4 6 8 10 12 14 16t (Myr)

0

2500

5000

7500

10000

12500

15000

Tclu

mp

Figure 8: Test 3 - I-front trapping and formation of a shadow.Average neutral fraction (top) and temperature (bottom)within the dense clump as a function of time during the sim-ulation. The dashed lines represent the same results plottedusing C2-Ray data from RT06. We see that our ionizationand heating are lower; we interpret this discrepancy as theresult of adopting a point source of radiation rather than aplane-parallel front.

results of C2-Ray as similarly colored dashed lines. While theneutral fractions agree well, there is more of a discrepancyin the temperatures. We believe this is a result of the raydivergence present with a point source that is absent in thecase of plane parallel radiation. The diverging rays overheatthe gas behind the front and underheat the gas in front ofthe front, an effect we see in our plot.

Finally, in Figure 8 we plot the average neutral fractionand temperature inside the clump for our model (solid lines)against the results from C2-Ray (dashed lines). While wesee that our model both heats and ionizes the clump lesseffectively than C2-Ray, we again believe this is a resultof the diverging nature of our radiation, which exposes asmaller portion of the sphere to the full radiation, at leastinitially. Despite this discrepancy, our results are still wellwithin the spread of models shown in RT06.

3.5 Test 4 - Multiple Sources in a Cosmic DensityField

The final test in RT06 is a simple simulation of a cosmolog-ical density field. The simulation volume is cube with side0.5 h−1 cMpc at redshift z = 9 and resolution 1283 (hereh = 0.7). The source of ionization is 16 point sources cen-tered within the 16 most massive halos, and emit fγ = 250ionizing photons per baryon with the same T = 105 K black-body spectrum used in tests 2 and 3. The sources are as-sumed to live for ts = 3 Myr, which is longer than the lengthof the simulation, so that they remain on for the entire sim-ulation. The luminosity of each source is thus:

Nγ = fγMΩb

ΩmmHts.

Here M is the mass of the halo, Ωb = 0.043, and

Ωm = 0.27. It is assumed that radiation leaving the boxis lost, and the volume is tracked for a total period of 0.4Myr. The density grid and halo position/luminosities arecurrently available from the Cosmological Radiative Trans-fer Comparison Project website. In Figure 9 we show a se-lection of plots from this simulation. The top four figures areslices through the center of the simulation to allow for visualcomparison between our results and the RT06 results. TheML and MR plots show slices through the neutral fractiongrid at the z = zsim/2 plane of the simulation at t = 0.05Myr and t = 0.2 Myr, respectively. The LL and LR plotsshow slices through the temperature grid at the z = zsim/2plane of the simulation at t = 0.05 Myr and t = 0.2 Myr,respectively. The results are in good agreement, althoughthe difference in colormap between our plots and those ofRT06 may make visual comparison difficult. Our plots mayalso be visually compared to those of Wise & Abel (2011),which use a similar colormap and thus closely resemble ourresults.

The bottom-left panel in Figure 9 shows histogram ofneutral fraction (left) and temperature (right) at times of 50,200, and 400 kyr. These results are in good agreement withthe codes presented in RT06. The bottom-right panel of Fig-ure 9 shows the volume averaged (χv, solid black line) andmass averaged (χm, dashed red line) ionized fraction withinthe volume as a function of time. We see that the massaveraged ionized fraction is larger at early times, in agree-ment with the expectation of inside-out reionization withinthe RT06 simulation (Gnedin 2000b; Miralda-Escude et al.2000; Sokasian et al. 2004). These results are again in goodagreement with those presented in RT06.

3.6 Methodology Test: Spherical CorrectionFactor

The spherical correction factor defined in Section 2.5 repre-sents an attempt to approximate the true overlap betweenthe cubic grid cells and the conic ray intersecting at ar-bitrary angles. The true correction factor depends on toomany factors to calculate exactly, so any simple approxima-tion will necessarily be somewhat flawed. In Figure 10 weplot a collection of slice plots from Test 1 using differentchoices for the power used in the correction factor formula.The top and bottom rows of this figure represent the sameslices with different choices of ray splitting rate (Φ = 1 andΦ = 3, respectively), while each column represents a differ-ent choice for the correction factor. The first column showsthe case with no correction, and the artifacts are clearly vis-ible. The second column shows the linear correction factor,and the second plot includes the secondary correction (whichmoves the excess radiation from a given cell to the adjacentcell which the ray is closest to). Finally, the last columnshows the squared correction, as described in Wise & Abel(2011). We see that the linear power correction with the sec-ondary correction results is the closest approximation of atrue sphere. The inclusion of this secondary correction alsoserves the purpose of compensating for the lost radiation dueto the primary correction factor. In Figure 11 we plot thecomparison of the model radius to the analytic Stromgremradius for the same four models in Figure 10. We see thatthe linear and square models reduce the radius of the modelby factors of 5% and 10%, respectively. The correction fac-

MNRAS 000, 000–000 (0000)


10 2 10 1 100

HI

10 3

10 2

10 1

100

Prob

abilit

ydi

strib

utio

n

t = 50 kyrt = 200 kyrt = 400 kyr

102 103 104 105

T (K)

10 3

10 2

10 1

100

0 50 100 150 200 250 300 350 400t (kyr)

0.0

0.2

0.4

0.6

0.8

1.0

x

Figure 9: Test 4 - Multiple sources in a cosmic density field. (Top and middle left.) Slice plots of the neutral fraction at 0.05Myr and 0.2 Myr into the simulation, respectively. (Top and middle right.) Slice plots of the temperature at 0.05 Myr and0.2 Myr into the simulation, respectively. These results are almost identical to those presented in Wise & Abel (2011), whichuses a similar adaptive ray tracing scheme to ours. (Bottom left.) Histograms of neutral fraction (left) and temperature (right)at fixed times in the simulation. (Bottom right) Mass averaged (dashed line) and volume averaged (solid line) ionized fractionas a function of time throughout the simulation box. These results are in good agreement with the results presented in RT06.

MNRAS 000, 000–000 (0000)


Φ = 1. 0

Φ = 3. 0

No correction Primary Secondary Square

Figure 10: This figure shows a selection of correction methods for two different ray splitting rates. The parameter Φ representsthe number of rays per cell area, so that the bottom row have three times the splitting rate of those in the top row. The firstcolumn represents the method without correction factors. The second column illustrates the case with a primary correction,or the application of Equation 2 to the rate of ionization in every cell. The third column represents the case with primary andsecondary correction, or the application of Equation 3 to the cell which is closets to the cell face has the greatest overlap. Thefourth column represents the case with only the primary correction with fc squared, as presented in Wise & Abel (2011). Thelow Φ = 1 plots were included to emphasize the difference between these configurations; clearly a higher value of Φ improvesthe accuracy for any configuration, but also increases the computational work for the simulation.

0 100 200 300 400 500T (Myr)

0.80

0.85

0.90

0.95

1.00

1.05

1.10

r sim/r

anal

ytic

Uncorrected

Linear

Corrected

Squared

Figure 11: Same as the bottom panel in Fig. 4 (left) butusing different spherical correction factors. We see that thefc and f2

c corrections shrink the size of the ionized regionby roughly 5% and 10%, respectively, while including a sec-ondary correction to the fc correction restores the size ofthe region to the uncorrected size.

tor, however, restores the accuracy of the model radius quitewell. We thus chose to use in our simulations the secondarycorrection in combination to the fc factor as it preservesphoton conservation and removes azimuthal artifacts.

4 BENCHMARKS

The way in which our code processes individual rays withGPUs is inherently parallel (CUDA kernel). For a singleGPU, the number of cores over which this work is distributedis fixed by the specifications of the individual GPU. We thuschoose to benchmark our code by measuring its performanceon a simple test problem varying the number of source ofradiation and the dimension of the computational grid. Ourtest problem is based on a test in Wise & Abel (2011). Weplace the ionizing source(s) at the center of a cubic 643 gridof size 15 kpc. The medium is pure hydrogen of density10−3 cm−3, and the ionizing source(s) have a monochro-matic 17 eV spectrum with luminosity 5× 1048 photons/s.The simulation is run with the radiation-base adaptive timestep (see Section ) for 250 Myr. For each simulations, weplot the time it takes for the entire ray tracing algorithmto run (Total Rad) as well as the time it takes for the algo-rithm to run on a single sub-volume (we divide the volumein 8 sub-volumes).

Dimension Scaling: We measure how our code per-forms with varying grid size by performing the test simula-tion on grids by running the same simulation with grid res-olutions of 643, 1283, 2563, 5123, 10243. The red line showsthe time it takes for the full radiative transfer calculation,while the blue line shows the line shows the time it takes fora single sub-volume to complete its ray tracing calculation(the blue dashed line is the time it takes for the CUDA ker-nel to execute in isolation; this time is measured locally onthe GPU, while the rest of the times are measured on the

MNRAS 000, 000–000 (0000)


64 128 256 512 1024Ndim

10 2

10 1

100

101

102

t(s)

Total Rad StepLocal Rad StepKernelRad OverheadFull Ion StepN3

1 10 100 1000Nsources

10-3

10-2

10-1

100

101

t(s

)

Total Rad Step

Local Rad Step

Kernel

Rad Overhead

Full Ion StepN 1

Figure 12: Benchmarks for a Stromgren sphere in a cubic grid. (Left.) The time taken to trace rays from a single sourcethrough the entire grid for grid sizes 643 through 10243. The time taken scales between N2 and N3. This is expected for thisproblem, as the number of rays scales with N2 and the number of steps along each ray scales with N . (Right.) The time takenfor a single radiative transfer step at grid size 643 for a number of sources between n = 1 and n = 1000. For small sourcecounts the per-step overhead can be seen, and as the number of sources increase, the time taken scales linearly, as expected.

host node). The difference between these lines is the timeit takes for the rays to be shared between the nodes andfor the rays to propagate through the rest of the volume.We plot the difference between the total radiation calcula-tion and the kernel execution times with the green line, as ameasure of the MPI overhead during the radiative transfercalculation. Finally, we include the time taken for the ion-ization calculation with the cyan line. We see that the MPIoverhead varies little with dimension, while the overall exe-cution time scales with grid dimension as a power-law withslope between N2

dim and N3dim. The problem should theoret-

ically scale with N3dim: two powers of Ndim for the number of

rays, and one power of Ndim for the number of steps alonga ray of the same physical length. However, our GPU codedoesn’t perform as well when the number of rays is small,as the occupancy of the GPU processors is lower and somecomputational power is wasted.

Source Scaling: We measure how our code performswith a varying number of sources by measuring how long thecode takes to perform a single step of the calculation withN sources at the same location, with the luminosity of eachreduced by a factor of N , so that the final result should bethe same. In the left plot of Figure 12 we plot the resultingtimes for 1, 10, 100, and 1000 sources. We see that the code’soverall time scales linearly with the number of sources, asanticipated.

Cosmic Simulation Benchmark: We also include abenchmark based on a full cosmic simulation run using ourcode on a 1283 grid divided evenly into 8 sub volumes, each643. In Figure 13 we plot the time taken for a complete raytracing step divided by the total number of sources (blue)and by the number of sources in the most populated sub-volume (green). In such cosmic simulations, the bottleneckedis the sub-volume which takes the longest time to process,which is why we plot the time divided by the number ofsources in the most populated volume. However, the sincethe GPU’s process the rays independently, the time divided

by the total number of sources is a better proxy for thetime taken per step per source, as long as the sources arerelatively evenly distributed between the sub volumes. Inthis simulation, when xe ∼ 0.2, the simulation is able tofully process rays from ∼ 104 sources on a 2563 grid in ∼ 20seconds.

5 SUMMARY

We have described our implementation of the spatially adap-tive ray tracing radiative transfer code ARC which is de-signed to take advantage of the extremely parallel process-ing power of the GPUs available in today’s supercomputers.Our algorithm is based on the well known method presentedin Abel & Wandelt (2002). We have presented the method-ology of our code, as well as a novel approach to a correctionmethod for ray tracing in a Cartesian grid. Our code is ableto split a computational volume into sub-volumes, each ofwhich is contained on an independent GPU linked by MPI,allowing our code to tackle very large problems avoiding thelimitations related to the available memory on the GPU. Weverified the accuracy of our method by performing a selec-tion of tests presented in Iliev et al. (2006). Finally, we dis-cussed a selection of benchmarks to demonstrate the speedand scaling of our code.

We believe that the unique characteristics of GPUsmake them ideal for the computational problem of ray trac-ing. ARC already takes advantage relatively recent innova-tion such as CUDA aware MPI (only becoming available in2013), but the optimization and speed up of our code is anongoing process that will take advantage of new capabilitiesof GPUs as they become available. The speed, number ofcores and the memory of GPUs has been growing rapidlyover the years allowing us to tackle problems which werecomputationally unfeasible only a few years ago.

The first application of ARC will be to simulate the

MNRAS 000, 000–000 (0000)


0 100 200 300 400 500tH Myr

10 1

t(s)

Ndim = 128

tRad/NtottRad/Nsub

Figure 13: Execution times (in seconds) for a 1282 simulationof cosmic reionization plotted as a function of simulationtime (on 8 GPUs). The plot shows the time taken by a fullradiative transfer calculation divided by the total numberof sources (blue line) and divided by the maximum numberof sources in any sub-volume (green line). The eight sub-volumes are processed in parallel, so as long as the sourcesare well distributed within the volume, the blue line shouldrepresent the mean time to execute the radiative transferstep per source at this resolution. We note that the timesare unusually large at the beginning of the simulation, as theconstant overhead dominates when the number of radiationsources is very small (i.e., zero or one source)

reionization epoch using pre-computed dark matter simu-lations in a (10 cMpc)3 volume, with sufficient resolutionto capture minihalos with masses > 106 M. These simu-lations can resolve the sites of formation of Population IIIstars, which we will be able to model adopting the model in(Ricotti 2016), and have a sufficiently large volume to cap-ture the formation of galaxies observed in the Hubble ultra-deep fields. We will take advantage of the non-diffusive na-ture of the ray-tracing method to simulate realistic emissionof ionizing radiation from the minihalos, including the shortduration of the bursts of radiation and anisotropic emissionexpected if the dominant star formation mode is in com-pact star clusters (Ricotti 2002; Katz & Ricotti 2013, 2014;Ricotti et al. 2016). Based on our previous analytical study(Hartley & Ricotti 2016), we believe that properly account-ing for these effects will have a major impact on both thetopology of reionization and the budget of ionizing photonsnecessary to reionize by redshift z ∼ 6.2.

ACKNOWLEDGMENTS

MR thank the National Science Foundation for sup-port under the Theoretical and Computational Astro-physics Network (TCAN) grant AST1333514 and CDI grantCMMI1125285. Thanks to the anonymous referee.

REFERENCES

Abel T., Norman M. L., Madau P., 1999, Astrophysical Journal

, 523, 66

Abel T., Wandelt B. D., 2002, MNRAS , 330, L53

Altay G., Croft R. A. C., Pelupessy I., 2008, MNRAS , 386, 1931

Alvarez M. A., Abel T., 2007, MNRAS , 380, L30

Alvarez M. A., Bromm V., Shapiro P. R., 2006, AstrophysicalJournal , 639, 621

Anninos P., Zhang Y., Abel T., Norman M. L., 1997, New As-

tronomy, 2, 209

Aubert D., Teyssier R., 2008, MNRAS , 387, 295

Auer L. H., Mihalas D., 1970, MNRAS , 149, 65

Choudhury T. R., Ferrara A., 2005, MNRAS , 361, 577

Ciardi B., Ferrara A., Marri S., Raimondo G., 2001, MNRAS ,324, 381

Deparis N., Aubert D., Ocvirk P., Chardin J., Lewis J., 2018,

ArXiv e-prints

Finlator K., Ozel F., Dave R., 2009, MNRAS , 393, 1090

Furlanetto S. R., Zaldarriaga M., Hernquist L., 2004, Astrophys-ical Journal , 613, 1

Gnedin N. Y., 2000a, Astrophysical Journal , 535, 530

Gnedin N. Y., 2000b, Astrophysical Journal , 542, 535

Gnedin N. Y., Abel T., 2001, New Astronomy, 6, 437

Gonzalez M., Audit E., Huynh P., 2007, Astronomy & Astro-

physics, 464, 429

Gorski K. M., Hivon E., Banday A. J., Wandelt B. D., Hansen

F. K., Reinecke M., Bartelmann M., 2005, Astrophysical Jour-

nal , 622, 759

Hartley B., Ricotti M., 2016, MNRAS , 462, 1164

Hasegawa K., Umemura M., Susa H., 2009, MNRAS , 395, 1280

Hui L., Gnedin N. Y., 1997, MNRAS , 292, 27

Iliev I. T. et al., 2006, MNRAS , 371, 1057

Johnson J. L., Greif T. H., Bromm V., 2007, Astrophysical Jour-

nal , 665, 85

Katz H., Ricotti M., 2013, MNRAS , 432, 3250

Katz H., Ricotti M., 2014, MNRAS , 444, 2377

Krumholz M. R., Stone J. M., Gardiner T. A., 2007, AstrophysicalJournal , 671, 518

Mellema G., Iliev I. T., Alvarez M. A., Shapiro P. R., 2006, New

Astronomy, 11, 374

Miralda-Escude J., Haehnelt M., Rees M. J., 2000, AstrophysicalJournal , 530, 1

Mitra S., Choudhury T. R., Ferrara A., 2015, MNRAS , 454, L76

Norman M. L., Paschos P., Abel T., 1998, MmSAI , 69, 455

Paardekooper J.-P., Kruip C. J. H., Icke V., 2010, Astronomy &

Astrophysics, 515, A79

Pawlik A. H., Schaye J., 2008, MNRAS , 389, 651

Pawlik A. H., Schaye J., 2011, MNRAS , 412, 1943

Petkova M., Springel V., 2009, MNRAS , 396, 1383

Razoumov A. O., Scott D., 1999, MNRAS , 309, 287

Ricotti M., 2002, MNRAS , 336, L33

Ricotti M., 2016, MNRAS , 462, 601

Ricotti M., Gnedin N. Y., Shull J. M., 2002a, Astrophysical Jour-

nal , 575, 33

Ricotti M., Gnedin N. Y., Shull J. M., 2002b, Astrophysical Jour-

nal , 575, 49

Ricotti M., Ostriker J. P., 2004, MNRAS , 352, 547

Ricotti M., Parry O. H., Gnedin N. Y., 2016, Astrophysical Jour-

nal , 831, 204

Rijkhorst E.-J., Plewa T., Dubey A., Mellema G., 2006, Astron-omy & Astrophysics, 452, 907

Sokasian A., Abel T., Hernquist L. E., 2001, New Astronomy, 6,

359

Sokasian A., Yoshida N., Abel T., Hernquist L., Springel V., 2004,MNRAS , 350, 47

Stone J. M., Mihalas D., Norman M. L., 1992, ApJS , 80, 819

Susa H., 2006, PASJ , 58, 445

Trac H., Cen R., 2007, Astrophysical Journal , 671, 1

Whalen D., Norman M. L., 2006, ApJS , 162, 281

Wise J. H., Abel T., 2011, MNRAS , 414, 3458

MNRAS 000, 000–000 (0000)


Wyithe J. S. B., Loeb A., 2003, Astrophysical Journal , 586, 693

MNRAS 000, 000–000 (0000)

ARC: Adaptive Ray-tracing with CUDA, a New Ray Tracing ... · rays, making the task well suited to...

Documents

Transcript of ARC: Adaptive Ray-tracing with CUDA, a New Ray Tracing ... · rays, making the task well suited to...