CyberShake Study 15.3 Science Readiness Review

Study 15.3 Scientific Goals• Calculate a 1 Hz map of Southern California

• Produce meaningful 2 second results for the UGMS• RotD50 and RotD100 at 2, 3, 4, 5, 7.5, 10 seconds• Contour maps

• Compare 0.5 Hz and 1 Hz hazard maps

• Use Graves & Pitarka (2014) rupture generator with regular spaced hypocenters

• 336 sites (10 km mesh + points of interest + “gap” sites)• Run 14 UGMS sites first

• Produce 1 Hz seismograms which could be combined with BBP high-frequency seismograms

Proposed Study sites (336)

Green sites are the 50 new “gap” sites

Study 15.3 Data Products

• CVM-S4.26 Los Angeles-area hazard maps• RotD100 2, 3, 4, 5, 7.5, 10 sec• RotD50 2, 3, 4, 5, 7.5, 10 sec• Geometric mean 2, 3, 5, 10 sec

• Hazard curves for 286 sites, at 2s, 3s, 5s, 10s

• 336 sets of 2-component SGTs

• Seismograms for all ruptures (~160M)

• Peak amplitudes in DB for 2s, 3s, 5s, 10s• RotD100, RotD50 and geometric mean SA

Study 15.3 Notables

• First 1 Hz hazard maps

• First study with RotD50 and RotD100 calculated

• First study to use OLCF Titan

• First study with Graves & Pitarka (2014) rupture generator with uniformly spaced hypocenters

• First study with 200 m rupture grid point spacing

• First study with source filtered at a different frequency than the simulation frequency

Study 15.3 Parameters

• 1.0 Hz deterministic• 100 m spacing• dt=0.005 sec• nt=40000 timesteps

• CVM-S 4.26• Vs min = 500 m/s

• UCERF 2

• Graves & Pitarka (2014) rupture variations• 200 m rupture grid point spacing

• Source filtered at 2.0 Hz

Rupture Generator Differences• When rupture geometry was changed from

1000m to 200m resolution, hazard curves changed dramatically

TEST site: blue = 200 m, black = 1000 m

0.5Hz UCERF23 sec SACVM-S4

Rupture Generator• We determined that the change in hazard

curves was due to hypocenter undersampling

M6.55, Puente Hills

Rupture Generator changes

• Previous number of realizations related to fault length

• # of realizations = max(10, C * Area/10.0)• C = 0.5

• Each realization is unique slip + hypocenter location

• Supports either random or uniform hypocenter distribution

Rupture Generator v3.3.1• Use of new G&P rupture generator (v3.3.1)

brought 1000m and 200m curves into agreement

TEST site, black = 200 m, magenta = 1000 m

0.5Hz UCERF23 sec SACVM-S4

Random vs Uniform Hypocenters

• Variation counts• G&P 2010: 423k• Uniform: 485k• Random: 542k

• Uniform easierto interpolate

0.5Hz UCERF23 sec SACVM-S4.26

WNGC site: black=random, magenta=uniform

Source Filtering, 0.5 Hz simulation• Changed frequency of 4th order lowpass

Butterworth filtering of SGT source from 0.5 to 1 Hz

WNGC site: Blue = filtered at 1 Hz; Black = filtered at 0.5 Hz

PseudoAA content by source filter

WNGC site, blue = filtered at 0.5 Hz, green = filtered at 1 HzM8.05, Elsinore

Source Filtering, 1 Hz simulation

Blue = filtered at 2 Hz; Black = filtered at 1 Hz

• Changed frequency of 4th order lowpass Butterworth filtering of SGT source from 1 Hz to 2 Hz

PseudoAA content by source filter

WNGC site, blue = filtered at 1 Hz, green = filtered at 2 HzM8.05, Elsinore

Fourier content by source filter

WNGC siteblue = filtered at 1 Hzgreen = filtered at 2 HzM8.05, Elsinore

Computational Requirements

• Per site: ~3720 node-hrs • SGTs: depends on execution site (~50%)

• Titan = 2110 node-hrs / 63,300 SUs• Blue Waters = 1760 node-hrs / 30,200 SUs• More expensive for Titan because of padding in pilot jobs and

different node-hrs -> SU conversion• PP: 1880 node-hrs / 60,200 SUs (~50%)

• Computational time:• Titan (SGTs): 355K node-hours / 10.7M SUs• Blue Waters: 928K node-hours

• SGTs: 275K GPU node-hrs, 21K CPU node-hrs• PP: 632K CPU node-hrs

• Titan has 104M SUs remaining

• Blue Waters has 5.3M node-hrs remaining

Storage Requirements

• Titan• Purged: 526 TB (for SGTs and temp data)

• Blue Waters• Delayed purge: 506 TB (for Titan SGTs)• Purged: 526 TB SGTs + 9 TB data products

• SCEC• Archived: 9.1 TB (seismograms, PSA, RotD)• Database: 268 GB (Geom @ 4 periods, RotD @ 6)• Temporary: 608 GB (workflow logs)• Shared SCEC disks have 171 TB free

Estimated Duration

• Limiting factors:• XK node queue time

• 800 XK nodes is 19% of Blue Waters• Titan -> Blue Waters

• If throughput is very high, transfer could be bottleneck• USC HPC downtime for ~1 week in April

• Estimated completion is 12 weeks (11 running + 1 downtime)• Based on same node availability as Study 14.2

• Planning to request reservation on Blue Waters

• Planning to request high priority on Titan

Personnel Support• Scientists

• Tom Jordan, Kim Olsen, Rob Graves

• Technical Lead• Scott Callaghan

• Job Submission / Run Monitoring• Scott Callaghan, David Gill, Phil Maechling

• NCSA Support• Omar Padron, Tim Bouvet

• Titan Support• Val Anantharaj

• USC Support• John Yu, John Mehringer

• Workflow Support• Karan Vahi, Gideon Juve

Science To-dos Pending

• Confirm SGTs from Titan give same result as SGTs from Blue Waters• Calculate two duplicate SGT sites on Blue Waters

and Titan and confirm results match

• Forward simulations versus reciprocity?• Run forward simulations to confirm reciprocity and

forward calculations match• Requires converting SRF to AWP-ODC source input

format

Risks

• Queue times on Blue Waters for XK nodes • Will try to dynamically assign SGT jobs to resources

• Unforeseen complications with Titan pilot jobs• Small tests have worked OK, but issues at scale?

• Congestion protection events (network overloaded)• If triggered consistently, will need to limit number of

post-processing workflows

Action Items

• Confirm 200s is long enough for SGT simulation

• Insert ERF 36 hypocenters into DB

• Decide whether or not to run forward simulation

• Determine why 2 Hz filtered source isn’t showing expected differences in seismograms or hazard curves

• Select additional sites to help fill in gaps and discontinuities in the hazard map

Thanks for your time!