06 Session3 G8ExascaleWorkshop INGENIOUS

7/28/2019 06 Session3 G8ExascaleWorkshop INGENIOUS

http://slidepdf.com/reader/full/06-session3-g8exascaleworkshop-ingenious 1/38

INGENIOUS:Using next generation computers and algorithms for

modeling the dynamics of large biomolecular

systems

Makoto Taiji

Computational Biology Research Core

RIKEN Quantitative Biology Center

Processor Research Team

RIKEN Advanced Institute for Computational Sciences

[email protected]



Our future targets

Cilia of mouse embryo

Bacterial FlagellumFluid dynamic mechanism responsible for breaking left-

right symmetry of the Human Body: The Nodal Flow ,

N. Hirokawa, Y. Okada, Y. Tanaka,

Annual Review of Fluid Mechanics 41, 53-72 (2009).

https://www.youtube.com/watch?v=3y_P67KwuvU

https://www.youtube.com/watch?v=vxiwhfgzL0Q



Challenges in Molecular Dynamics

simulations of biomolecules

S. O. Nielsen, et al, J. Phys. (Condens. Matter.), 15 (2004) R481

30,000 year・ ExaFLOPS

Strong

Scaling

Weak Scaling

K computer

Anton/MDG4 Target Region

1021

J energy(~3x1018J

is spent in Japan

in each year)

Multiscale approachis essential



Organization

n Molecular Fluctuations (Aston Group)

n Molecular Fluctuations ↔ Fluid Dynamics

(Moscow Group)

n Multiscale Fluid Dynamics (Univ. London/

Cambridge Group)

n HPC (RIKEN)



Mercedes-Benzwaterasabridgetothemacroscale

Implementa8oninmolecular

dynamics

ArtursScukinsandDmitryNerukh



WhyMBwater?

• Arela8velysimple,2Dmodelthathasallfeaturesandpeculiari8esofrealwater

• Being2DscalesmuchbeFerwithsize:allowstoreachthespa8alsizesofhydrodynamics

• Welldevelopedtheore8cally(startedbyBen-Naiminearlyseven8es)

• Computa8onallywellstudiedbyMonteCarlo,butnoinves8ga8onsbyMolecularDynamics• WellsuitedfroourpurposeofdevelopinghybridMolecularDynamics–luidDynamicsapproach



MercedesBenzpoten8al:,

whereisLennard-Jonespoten8al,

,

isorienta8ondependantpoten8al,isaGaussianfunc8on.



Wehavederivedtheformulasforcalcula8ng

thermodynamicsfromMDtrajectories

• Temperature:

• Pressure:

• Heatcapacity:

• Heatexpansioncoefficient:

• Compressibility:

whereisa8meaverage,Visanarea,N-numberofmolecules,T-temperature,K–kine8c

energy,–density.



Results



Structure

TheRDqualita8velydiffers

fromLennard–JonesRD

butcoincideswiththeresults

obtainedusingMonteCarlo



Conclusions

• The2DMercedes-Benzmodelmimicsrealwaterbehaviour.

• Capturesminimumofpressure(volume),nega8veexpansioncoefficient,minimumof

compressibilityandhighheatcapacity.

• RDqualita8velydiffersfromLennard-JonesRD.



Towardsaccuratemodelingacrossdifferentscales:high-

resolu6onmethodsforFluctua6ngHydrodynamics

equa6onsV.Y.Glotov,V.M.Goloviznin,A.V.Danilin

( )

( ) ( )

22

2

0

40

3

40

3

u

t x

u P u u s

t x x x

E P u q u s E u T ut x x x x x

ρ ρ

ρ ρ η

ρ ρ η κ

∂ ∂+ =

∂ ∂

∂ +∂ ∂ ∂+ − ⋅ ⋅ − =

∂ ∂ ∂ ∂

∂+

∂+

⋅∂ ∂ ∂ ∂⎛ ⎞+ − ⋅ ⋅ + ⋅ − =⎜ ⎟∂ ∂ ∂ ∂ ∂ ∂⎝ ⎠

2

;2

v

u

E c T ρ ρ ρ = +

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2

8, , ;

3

2, , ;

k T s x t s x t x x t t

k T q x t q x t x x t t

η δ δ

σ

κ δ δ

σ

⋅ ⋅ ⋅ʹ ʹ ʹ ʹ= ⋅ − ⋅ −

⋅

⋅ ⋅ ⋅ʹ ʹ ʹ ʹ= ⋅ − ⋅ −

( )

( )

( )

( )

( )

( )

2

12 2 2 2

2

22 2 2 2

11 1;

1 1

11 1;

1 1

1ln

v

su P u P u c G

t t x xc s c s

su P u P u c G

t t x xc s c s

P s

t c T t γ

γ

ρ ρ ρ γ ρ ρ γ

γ

ρ ρ ρ γ ρ ρ γ

ρ ρ

⎛ ⎞ ⎛ ⎞⎧ ⎫−∂ ∂ ∂ ∂⎪ ⎪⎜ ⎟ ⎜ ⎟+ ⋅ + + − ⋅ + ⋅ =⎨ ⎬⎜ ⎟ ⎜ ⎟∂ ∂ ∂ ∂⋅ − − ⋅ − −⎪ ⎪⎩ ⎭⎝ ⎠ ⎝ ⎠

⎛ ⎞ ⎛ ⎞⎧ ⎫−∂ ∂ ∂ ∂⎪ ⎪⎜ ⎟ ⎜ ⎟− ⋅ + − − ⋅ − ⋅ =⎨ ⎬⎜ ⎟ ⎜ ⎟∂ ∂ ∂ ∂⋅ − − ⋅ − −⎪ ⎪⎩ ⎭⎝ ⎠ ⎝ ⎠

⎛ ⎞ ⎛∂ ∂− ⋅⎜ ⎟ ⎜∂ ∂⎝ ⎠ ⎝

3

1ln ;

v

P su G

x c T xγ

ρ ρ

⎡ ⎤ ⎡ ⎤⎞ ⎛ ⎞ ⎛ ⎞∂ ∂+ ⋅ − ⋅ =⎢ ⎥ ⎢ ⎥⎟ ⎜ ⎟ ⎜ ⎟∂ ∂⎠ ⎝ ⎠ ⎝ ⎠⎣ ⎦ ⎣ ⎦

( )

2

1

c

s

ρ

γ

⋅

<

−

One dimensional case

Characteristic form of LL-NS equations

Condition for hyperbolicity



Stochas6cfluxes

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2

8, , ;

32

, , ;

k T s x t s x t x x t t

k T q x t q x t x x t t

η δ δ

σ κ

δ δ σ

⋅ ⋅ ⋅ ⎫ʹ ʹ ʹ ʹ= ⋅ − ⋅ − ⎪⋅ ⎪ ⇒⎬

⋅ ⋅ ⋅ ⎪ʹ ʹ ʹ ʹ= ⋅ − ⋅ −⎪⎭

( ) ( )

( ) ( )2

8, 0,1 ;

3

2, 0,1 ;

h

h

k T s x t Gauss

x t

k T q x t Gauss

x t

η

σ

κ

σ

⋅ ⋅ ⋅= ⋅

⋅ ⋅ Δ ⋅ Δ

⋅ ⋅ ⋅= ⋅

⋅ Δ ⋅ Δ

Stochastic fluxes approximation

• For high value of stochastic forcing (large s and q fluxes) the

solution of the LL Navier-Stokes equations is very challenging



•

Iserlis 1986• Roe 1998

• Samarskii and Goloviznin 1998

• Goloviznin and Karabasov 1998

• Karabasov, Hynes and Goloviznin 2001

• Tran and Scheurer 2002

• Kim 2004

• Goloviznin 2005

• Karabasov and Goloviznin 2007, 2009

c 0t x

∂ϕ ∂ϕ+ ⋅ =

∂ ∂

• Explicit, second-order in space and time

• Non-dissipative and low-dispersive

• Very compact stencil• Conservation form

• Staggered variables: one-cell stencil in space and time

• Nonlinear flux correction based on maximum principle

• Nonlinear flux reconstruction based on the minimum

solution variation

• Highly scalable method and has already been

successfully used in unsteady convection-dominatedflow modelling

x

t

Our choice: Compact Accurately Boundary Adjustinghigh-REsolution Technique (CABARET)



Comparisonofseveralcomputa8onalschemesfortheBellproblem

MacCormackscheme 2.61 -.4%

Piecewiseparabolicmethod 2.5 -9.4%

Third-orderRunge-KuFa 2.7 0.9%

CABARET 2.75 -3.2%

MolecularSimula8on 2.7 -2.1%

Variance in conserved quantities at equilibrium

MacCormackscheme 2.01 -14.3%


Third-orderRunge-KuFa 2.34 -1.3%

CABARET 2.31 -1.7%

MolecularSimula8on 2.35 0%

MacCormackscheme 13.31 -0.3%


Third-orderRunge-KuFa 13.65 2.3%

CABARET 13.1 -1.2%

MolecularSimula8on 13.21 -1%

2δρ Exact value:

2

J δ Exact value:

82.35 10

−

×

2 E δ Exact value:

13.34

102.84 10×



§ Towards micro- and nano-scales• Temperature uctuations important, large density and velocity uctuations

• Acoustics: ultra-sound / Biological applications: Coupling with MD

§ Interesting phenomena concurrently occur at small andlarger scale, both in time and space• Numerically difficult to deal efficiently with large time/space diff erences

• A multi-space-time algorithm is demonstrated

AMULTI-SPACE-TIMEALGORITHMFORCONCURRENTLARGE/SMALL

SCALEFLUIDDYNAMICSSIMULATIONS

AntonMarkesteijnandSergeyKarabasov



Mul6Space-Timealgorithm-Overview

§ Fluctuating Hydrodynamics (Landau&Lifshitz)• Mimic microscopic behaviour at macroscopic scales

• Dissipative uxes treated as stochastic variables• Thermodynamics: “Fluctuation-Dissipation theorem”

§ “Scale Function”: A (pre)dened “meshless” zoom value• The value of this function is increased where small time and space

phenomena are dominant• The scale function also determines the actual comp. grid

§ Equation transformations both in space and time• Transformations are dependent on scale function

• Transformed (Computational Domain) / Untransformed (Physical Domain) § Special time marching (local and global time)

• Local time step controlled by scale function

• Cells only updated when necessary (local<global time)

• (CFL curse): increased efficiency, decreased error

l f l



SomeExamplesofScaleFunc6ons

§ 1D Scale diff erence of 5• Both mesh size and local time scaled

• Computational domain simple Cartesian mesh

§ 2D Mesh (radial 25 to 1 (200x200 mesh)

l l 6 d d i



2DExample:Fluctua6ngHydrodynamics

§ Scale diff erence of 100, on a 200x200 mesh

§ Probe in centre of domain• Measure density transient

• Acoustic signal recovered from noise‒ Time ensemble

• Variables are Maxwellian

Fl 6 H d d i MD



Fluctua6ngHydrodynamicsvsMD

§ Density uctuations and the speed of sound• Domain 250x40, Scale Function 1 to 25 to 1 in

plateaus• Smallest volume 0.6x0.6x0.6 nm3 (liquid water)

• Speed of sound obtained by t (~1510 m/s)

• Continuum results compared to MD results



Scaling challenges in MD

n 〜50,000 FLOP/particle/step

n Typical system size : N=105

n 5 GFLOP/step

n 5TFLOPS eff ective performance

1msec/step = 170nsec/day

Rather Easy

n 5PFLOPS eff ective performance

1μsec/step = 200μsec/day???

Di fficult, but important



Scaling of MD on K Computer

Strong scaling〜50 atoms/core

~3M atoms/Pflops

22

1,674,828 atoms

Since K Computer is still under development,

the result shown here istentative.



GRAPE: special-purpose computer

for classical particle simulations

n GRAvity PipEn Originaly proposed by Prof. Chikada, NAOJ

n Special-purpose accelerator

▷ Astrophysical N -body simulations

▷ Molecular Dynamics Simulations

J. Makino & M. Taiji, Scientific Simulations with Special-Purpose Computers,John Wiley & Sons, 1997.

Host

Computer GRAPE

Most of Calculation → GRAPE Others → Host computer

Particle Data

Results

Problem in Heterogeneous System GRAPE/



Problem in Heterogeneous System - GRAPE/

GPUsn In small system

▷ Good acceleration, High performance/costn In massively-parallel system

▷ Scaling is often limited by host-host network,

host-accelerator interface

HostComputer

Accelerator

HostComputer

Accelerator

Host Network

High-Latency

Low-Bandwidth

HostComputer

Accelerator

General-purpose core

Embeddedmemories

Accelerator

General-purpose core

Embeddedmemories

Low-Latency

High-BandwidthLow-Latency

System-on-Chip

Low-BandwidthHigh-LatencyNetwork

Typical Accelerator System SoC-based System



Anton

n D. E. Shaw Research

n Special-purpose pipeline

+ General-purpose CPU core

+ Specialized network

n Anton showed the importance of

the optimization in communicationsystem

R. O. Dror et al., Proc. Supercomputing 2009, in USB memory.



MDGRAPE-4

n Special-purpose computer for MD simulation

n Test platform for special-purpose machines

n Target performance

▷ 20μsec/step for 100K atom system

▷ 8.6μsec/day (2fsec/step)

n Target application : GROMACS

n Completion: ~2013

n Enhancement from MDGRAPE-3

▷ 130nm→ 40nm process

▷ Integration of Network / CPU



MDGRAPE-4 System

48 Optical Fibers

12 lane6Gbps

Optical

12 lane6Gbps

Electric = 7.2GB/s

(after 8B10Bencoding)

Node(2U Box)

Total 64 Nodes(4x4x4)=4 pedestals

MDGRAPE-4SoC

Total 512 chips

(8x8x8)



MDGRAPE-4 System-on-Chip

n 40 nm (Hitachi HDL4S), ~ 230mm2

n 64 force calculation pipelines

@ 0.8GHz

~ 2.5 TFLOPS equivalentn 64 general-purpose processors

Tensilica Extensa LX4

@0.6GHz

n 72 lane SERDES @6GHz

n 65W

S C Bl k Di



SoCBlockDiagram

Network Unit FPGA IF

Global Memory

x y z 6Gbps x 12 x 6100MHz x 128

Bus Arbiter /DMAC

IMem

DMemCore

IMem

DMemCore

8 Pipelines

8 Pipelines

8 Pipelines

Bus Arbiter /DMAC

IMem

DMemCore

IMem

DMemCore

IMem

DMemCore

InstructionMemory (1)

InstructionMemory (2)

InstructionMemory (CGP)

Pipeline Blocks GP BlocksControl GP

MessageQueue



Embedded Global Memories in SoC

n ~1.8MBn 4 Block

n For Each Block

▷ 128bit X 2 for General-

purpose core▷ 192bit X 2 for Pipeline

▷ 64 bit X 6 for Network

▷ 256bit X 2 for Inter-block

GM4 Block 460KB

GM4 Block

460KB

GM4 Block 460KB

GM4 Block

460KB

Network

2 PipelineBlocks

2 GP Blocks



General-Purpose Core

n Tensilica LX @ 0.6 GHz

n 32bit integer / 32bit Floating

n 4KB I-cache / 4KB D-cache

n 8KB Local Memory

▷ DMA or PIF access

n 8KB Local Instruction Memory

▷ DMA read from 512KB Instruction memory

Core

4KB

Dcache

8KB

D-ram

4KBIcache

8KB

I-ram

Core

Core Core

Core Core

Core Core

Core Core

DMAC

QueueIF

Integer

Floa:ngQueue

PIF

Inst-

ruc:on

DMAC

GP Block

Barrier

Global Memory

Control Processor

InstructionMemory



Software evaluation platform for

MDGRAPE-4: RTL model

n RTL-based simulator on Candence Ncsimn Cycle accurate

n Slow (>10ms/cycle)




MDGRAPE-4:

n Under construction (4Q 2012)n Tensilica XTMP based multicore processor

simulator (non-free)

n Includes behavior models of

▷ Network

▷ Special-purpose pipeline

▷ Memories (latency can be considered)

n Two-levels▷ Precise memory models for instruction

▷ Innite memory for instruction




MDGRAPE-4 (2)

n Programming language▷ C▷ C++ (without malloc)

n Direct control of network unitsn No operating system, with simple monitor

l l f



Evaluation platform

based on MDGRAPE-4 simulator

n Extend MDGRAPE-4 simulatorn Change Balance▷ More resources for general-purpose cores

n General-purpose cores▷ Shared on-chip memory for 8-16 cores

▷ off -chip memory

▷ synchronization mechanism

n Special-purpose pipelines

n Network interface



Special-

purposeblock

General-

purposecores

Local/Cache

memories

On-chip

Network

Special-

purposeblock

General-

purposecores

Local/Cache

memories

On-chip

Network

Off-chip

Network

Off-chip

Memory

Special-

purposeblock

General-

purposecores

Local/Cache

memories

On-chip

Network

Special-

purposeblock

General-

purposecores

Local/Cache

memories

On-chip

Network

Off-chip

Network

Off-chip

Memory



Toward Exascale

For Molecular Dynamicsn Single-chip system

▷ >1/30 of the MDGRAPE-4 system can be embedded with11nm process

▷ Local MD + Multiscale▷ Still network is necessary inside SoC

n For further strong scaling for MD

▷ # of operations / step / 20Katom ~ 109

▷ # of arithmetic units in system ~ 106 /PopsExascale means “Flash” (one-path) calculation

▷ More specialization is required



Meetings & Visits

n Past▷ Nov 2011 @ Cambridge

▷ UK PI(Dr. Nerukh)’s visit to RIKEN for a month in Dec 2011

▷ Sep 2012 @ Kobe

▷ UK Researcher (Mr. Skukins)→ RIKEN (Sep-Nov 2012)n Future related events

▷ Dec 2012: UK-Japan bilateral workshop at British embassyin Tokyo (supported by British embassy Japan)

▷ Jul 2013: Royal Society Kavli Seminar in UK “Multiscale systems: linking quantum chemistry, moleculardynamics, and microuidic hydrodynamics”

▷ Project workshops in UK or/and Japan (2013)

06 Session3 G8ExascaleWorkshop INGENIOUS

Documents

Transcript of 06 Session3 G8ExascaleWorkshop INGENIOUS