Fred a new GPU-based fast-MC code

and its applications in proton beam therapy

MCMA 2017 - Napoli

A. Schiavi

Fast paRticle thErapy Dose evaluator

Collaboration• A. Schiavi, V. Patera, M. Senzacqua, Univ. La Sapienza

Roma /INFN (Italy)

• - G. Battistoni, S. Pioli - INFN (Italy)

• - I. Rinaldi, N. Krah - CNRS/IN2P3 and Lyon 1 University (France)

• A. Rucinski, J. Gajewski - PAN, Krakow (Poland)

FRED fast-MC platform• MC for protons in voxel geometry

• Tabulated total stopping power in water (PSTAR-NIST), energy straggling (Gaussian and Landau-Vavilov regimes)

• MCS models: single-,double-,triple-gaussian, 2 gauss+Rutherford

• Nuclear interactions: elastic and inelastic; fragmentation; local deposition of heavy ions; tracking of secondary protons and deuterons

• HU to density conversion (Schneider) and stoppow calibration

• MC-TPS: dose optimization using DDO (Lomax)

• RBE models = fixed 1.1, LETd-based (Wedenberg, Carabe, Wilkens, Chen), table-based (LEM1, MKMPIDE)

Fred - no nucl

Fred

Fluka

200 MeV protons in liquid waterDose map of a pencil beam

Water model: energy deposition

100 MeV 150 MeV 200 MeV 250 MeV

nucl. off

elasticnucl. only

elasticnucl. only

nucl. on

GeantFLUKAFred

22

20

18

16

14

1224 25 26 27

A. Schiavi et al, PMB 62 (2017) 7482–7504

Longitudinal profile and lateral tails

c)

d)

a)

Dose map

FLUKA

b)

QA SOBPz x

y

head-onside-on

Water tank CT

3 cm cube at 15 cm depth

QA SOBP: dose profiles and measurements

E = 226.61 MeV/u at 20 cm depth

Field size factor

• standard codes

• expensive (€€€€€)

• maintenance (staff)

• low budget (€€€€)

• redundancy

• in-house maintenance

+

Hardware for rapid MC recalculation

Parallel execution model in Fred

Fred front-end

ExtranodeMPI

IntranodeMulti-GPUOpenCl

IntranodePOSIX

Multi-threads

Queues and timeline

Execution timeline for 8 queues on 4 GPUs using OpenCL.

Host-to-device transfers (green), kernel execution (red), and device-to-host (blue) transfers

Hardware and Performance

full-MC * * FLUKA or Geant4C

PU

benchmark: 150 MeV protons in a water phantom on a 1 mm3 dose scoring grid

Hardware and Performance

full-MC * * FLUKA or Geant4C

PU

GP

U

• Patient-specific QA protocol at CNAO

• Patient-specific HU-RSP calibration

• Commissioning of CCB proton center in Krakow

• Dose monitoring using secondary protons

Applications to proton therapy

see S. Muraro talk this afternoon (ID 67)

Fast-MC recalculation of patient verification plans at CNAO

Fast-MC recalculation of patient verification plans at CNAO

18

Patient verification planTPS

FRED

TPS

FRED

Gy

tps

fred

head-on side-on

fred

tps

99.6% @ 2mm/2% 96.7% @ 1mm/1%

ɣ-index pass rates

Patient-specific and treatment-specific HU-RSP

calibration

Fred

Fred commissioning @ CCB Krakow Proton Beam Therapy Centre

FRED central cross-section

0 50 100 150 200 250

Z [mm]

-30

-20

-10

0

10

20

30

X [m

m]

0 50 100 150 200 250

Z [mm]

0

0.2

0.4

0.6

0.8

1

Dose

norm

. [-

]

Bragg Peak comparison

TPS Beam Model

(Rmax

=156.59mm, R 2

bort=1.000)

FRED MC sim.

(Rmax

=156.59mm, R 2

bort=1.000)

0 50 100 150 200 250

Z [mm]

2

3

4

5

6

7

Spot si

ze [m

m]

Spot primary sigma at depth

TPS σmean

Prim

FRED σmean

Prim

0 50 100 150 200 250

Z [mm]

5

10

15

20

25

Spot si

ze [m

m]

Spot secondary sigma at depth

TPS σmean

Sec

FRED σmean

Sec

-50 -40 -30 -20 -10 0 10 20 30 40 50

X [mm]

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dose

norm

. [-

]

σXTPS=5.61mm (R 2=0.9999)

σXFRED=5.65mm (R 2=0.9999)

TPS Eclipse profile

FRED MC sim. profile

Signle Gauss fit to TPS

Signle Gauss fit to FRED

-50 -40 -30 -20 -10 0 10 20 30 40 50

X [mm]

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

Log d

ose

norm

. [-

]

Prim. σXTPS=5.74mm

Sec. σXTPS=16.50mm

(R2=0.9994)

Prim. σXFRED=5.55mm

Sec. σXFRED=11.06mm

(R2=0.9943)

TPS Eclipse profile

FRED MC sim. profile

Double Gauss fit to TPS

Double Gauss fit to FRED

-50 -40 -30 -20 -10 0 10 20 30 40 50

Y [mm]

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dose

norm

. [-

]

σYTPS=5.19mm (R 2=0.9999)

σYFRED=5.54mm (R 2=0.9999)

TPS Eclipse profile

FRED MC sim. profile

Signle Gauss fit to TPS

Signle Gauss fit to FRED

-50 -40 -30 -20 -10 0 10 20 30 40 50

Y [mm]

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

Log d

ose

norm

. [-

]

Prim. σYTPS=5.38mm

Sec. σYTPS=16.54mm

(R2=0.9993)

Prim. σYFRED=5.47mm

Sec. σYFRED=11.08mm

(R2=0.9950)

TPS Eclipse profile

FRED MC sim. profile

Double Gauss fit to TPS

Double Gauss fit to FRED

Fred code is currently being commissioned at CCB as a quality assurance tool.Preliminary results show good agreement of single beam dose distributions calculated with Eclipse and Fred, indicating an accurate implementation of CCB beam model in the Fred MC-TPS code. Dose distributions for a complete plan can be obtained in about one minute using Fred on GPU. Robustness studies of treatment plan strategy can be conducted on the HPC cluster Prometheus.

Top performance on Prometheus

• 72 Nodes with 24 CPUs and 2 Tesla K40d GPUs

• Up to 144 GPUs in parallel + 1728 CPUs

perfect linear scaling

up to 0.3 billion primary/s

2.4 mln/s per GPU

This research was supported in part by PL-Grid Infrastructure.

• clinical validation of fast-recalculation tool

• applications to clinical routine

• extensions to include other ions (Carbon, Helium) and secondary particles (alphas, delta-rays and neutrons)

• dose monitoring using charged secondary particles

Future developmentsand perspectives

24

Patient recalculation plan

tps

fred

gamma-index 97% @ 2mm/2% gamma-index 92% @ 1mm/1%

recalculation at 1% = 700 million primary protons

simulation time = 72 s

Hardware: 4x GPU NVIDIA GTX 10801x CPU Intel i7-5930K @ 3,50 GHz with 12 cores

20 mln primary/s

Water-cooled 4 GPU workstation

4x NVIDIA Titan-Xp40 mln primary/s

budget: 10 kEuro

compare with newNVIDIA DGX-1(8x Tesla P100)

expected performance:80 mln primary/s

budget: 125 kEuro

Case study: 3D raytracing for legacy F77 hydrocode

From 1 to 2 Mray/s(equivalent to 800 MPI processes)

Raytracing step well belowhydrodynamic step

Duty-cycle 30%

Duty-cycle 100%

4 water-cooled GPU

2 air-cooled GPU