Development of Large-scale Quantum Mechanical Molecular Dynamics Simulation: Divide-and-conquer Density Functional Tight-binding Approach

Yoshifumi Nishimura,1 Aditya Wibawa Sakti,2

and Hiromi Nakai1–4

1RISE, Waseda Univ., 2Waseda Univ.,3JST-CREST, 4ESICB, Kyoto Univ.

1

The 1st International Workshop on Advanced Methods for Nano Materials Design (2017/07/14, KINTEX, Gyeonggi-do, Korea)

2

Chemical reaction simulation of large systems

Bond formation/cleavage, Electron transfer

QM-MD Classical MD

WFT DFT DFTB MM

Cluster collision[1] Electrolyte decomposition[2]

Carbon nanotube growth[3] Virus[4]

< 100 < 1000 < 1000 ~10000000

[1] H. Nakai, Y. Yamauchi, A. Matsuda, Y. Okada, K. Takeuchi, J. Mol. Struct. (THEOCHEM) 592, 61 (2002).[2] K. Ushirogata, K. Sodeyama, Y. Okuno, Y. Tateyama, J. Am. Chem. Soc. 135, 11967 (2013).[3] A. J. Page, Y. Ohta, S. Irle, K. Morokuma, Acc. Chem. Res. 43, 1375 (2010).[4] Y. Andoh, N. Yoshii, A. Yamada, K. Fujimoto, H. Kojima, K. Mizutani, A. Nakagawa, A. Nomoto, S. Okazaki, J. Chem. Phys. 141, 165101 (2014).

• Quantum mechanics (QM)

• Molecular dynamics (MD)

• Linear-scaling O(N) More than 104 atoms, Long simulation time

Dynamical behavior

WFT: Wave function theory

DFT: Density functional theory

DFTB: Density-Functional Tight-Binding

MM: Molecular Mechanics

Semi-empirical model derived from density functional theory (DFT)[1,2]

Density-Functional Tight-Binding (DFTB) method

3[1] M. Elstner, G. Seifert, Phil. Trans. R. Soc. A 372, 20120483 (2014).[2] M. Gaus, Q. Cui, M. Elstner, WIREs Comput. Mol. Sci. 4, 49 (2014).

• Dμν: Density matrix

• ΔqA: Induced Mullikencharge on atom A

atom2

atomatomrep

AO0

DFTB3

1

2

1

AB

BAAB

AB

BAAB

BA

AB qqqqVHDE

DFTB1 DFTB2 DFTB3

① ② ③ ④

Evaluate using parameters determined by DFT calculations

①: Charge independent term Precomputed Hamiltonian and overlap matrices Two-center approximation

②: Short-ranged two-body repulsive term Core-core repulsion DFT double-counting contribution

③, ④: Charge dependent term Self-consistent determination of ΔqA (SCC) Monopole approximation

No integral evaluation at runtime

Proper description of chemical bonds in MD with small cost

Key ingredient for reasonable accuracy and transferability

Efficiency of DFTB: Pros and cons

4[1] M. Elstner, G. Seifert, Phil. Trans. R. Soc. A 372, 20120483 (2014).

Molecule N(atoms) DFTB2 RI-PBE/6-31G(d) B3LYP/6-31G(d)

C60 60 1 1112 9398

(Ala)20 212 12 3418 27605

(H2O)48 144 3 769 3466

(H2O)123 369 15 5488 30822

Single-point calculation time for various molecules [s][1]

Hundreds of times faster than DFT for hundreds of atoms system

0 2000 4000 6000 8000 10000

Number of water molecules

0000

0500

1000

1500

2000

Tim

e [s

]DFTB2

O(N3.1)Cubic scaling with respect to system size is problematic for truly large systems

Combination with linear-scaling technique

Divide-and-Conquer (DC) method[1,2]

5[1] M. Kobayashi, H. Nakai, in Linear-Scaling Techniques in Computational Chemistry and Physics, (2011), pp. 97–127.[2] W. Yang, T.-S. Lee, J. Chem. Phys. 103, 5674 (1995).

…

Divide

Buffer region

Subsystem

Subsystem fragmentation without overlap

Localization region

…Conquer

SCF

Calculate• Total density matrix• Total energy• Other properties

Set common Fermi level (Conserve total number of electrons)

…

…

…

Solve subsystems’ equations• Reduce diagonalization cost: O(N)• Obtain subsystems’ orbitals

Fragment approach to accelerate QM calculation of large systems

• Automatized fragmentation of subsystems at each time step

Applicable to dynamical bond formation and cleavage

• No predefinition of electron & spin numbers of subsystems

Applicable to electron and spin delocalized systems

Advantages of DC method in QM-MD simulations

6

1tt 2tt

?

DC methodConventional fragmentation method

21S 21S212 S01 S 03 S ?2 S?1 S ?3 S

Performance of DC-DFTB

7

Energy and gradient calculation of isolated water system[1]

rb

0 2000 4000 6000 8000 10000

Number of water molecules

0000

0500

1000

1500

2000

Tim

e [s

]

DFTB2

DC-DFTB2 (rb = 5.0 Å)

DC-DFTB2 (rb = 5.5 Å)

DC-DFTB2 (rb = 6.0 Å)

Linear-scaling computational cost of DC-DFTB[1] H. Nishizawa, Y. Nishimura, M. Kobayashi, S. Irle, H. Nakai, J. Comput. Chem. 37, 1983 (2016).

O(N3.1)

O(N1.1) DC-DFTB2 (rb = 5.0 Å)

DC-DFTB2 (rb = 5.5 Å)

DC-DFTB2 (rb = 6.0 Å)

0 1000 2000 3000 4000 5000

Number of water molecules

Accuracy control with buffer size

0.0

0.5

1.0

1.5

2.0

EDC

-DFT

B−

EDFT

B[m

har

tree

]

2.5

• Subsystem: 1 H2O

• Buffer：Sphere region with radius of rb Å

• 1 node of Intel Xeon (8 cores)

• DC-DFTB-K program

8

Numerical assessment of DC-DFTB-MD

DC-DFTB-MD reproduces DFTB-MD

Total energy fluctuation

18.64 Å

Timing of 1 MD step[1]

Using only single node still takes time: e.g. ~75 sec/step for (H2O)1051

0 200 400 600 800 1000Number of water molecules

0

200

400

600

Tim

e [s

]

■DFTB-MD●DC-DFTB-MD

[1] H. Nakai, A. W. Sakti, Y. Nishimura, J. Phys. Chem. B 120, 217 (2016).

Ener

gy [

mH

artr

ee]

0.0

0.1

0.2

−0.1

−0.2

0.0 0.1 0.2 0.3 0.4 0.5Time [ps]

DFTB-MDDC-DFTB-MD

• Cubic water box: (H2O)216

• Subsystem: 1 H2O

• Buffer radius: 6 Å

• Δt = 0.2 fs

• NVE ensemble

• Cubic water box • Intel Xeon E5-2637 v3 (3.50 GHz)

O(N2.5)O(N1.2)

Massively parallel implementation

①

0

SHc ii

SCC

Hybrid parallelization of DC-DFTB

…

…

Common Fermi level εF

BufferSubsystem α

Vrep, H0,α, Sα, γα, Γα

MPI comm.

Hα

MPI comm.

Dα

qα

MPI comm.

q

① Independent calculation for each α

• MPI: Assign subsystems to processes

• OpenMP parallelization of ②, ③, ⑤, ⑥

oldmax qq

9

②

④

⑤

⑥

② Evaluation of components

③ Update Hamiltonian Diagonalization: O(N)

[1] H. Nishizawa, Y. Nishimura, M. Kobayashi, S. Irle, H. Nakai, J. Comput. Chem. 37, 1983 (2016).

④ Determination of Fermi level

• Interpolation based algorithm[1]

ε

n(ε)

εF

ne

)MO(

)(2)(

p

pp wfn

⑤ Calculate density matrix )MO(

*

F2

i

iii ccfD

A L

A SDq

)(

⑥ Calculate Mulliken charges

⑦ Calculate total Mulliken charges

SCC convergence check

③

⑦

Proton diffusion

10[1] C. J. T. de Grotthuss, Ann. Chim. 58, 54 (1806).

• Fundamental phenomenon Acid catalyst Enzymatic reaction Fuel cell design

• Proton transfer (Grotthuss shuttling) mechanism was proposed more than 200 years ago[2]

Dv: Vehicular diffusion in Eigen (H3O+) form

DG: Grotthuss diffusion via Zundel (H5O2

+) form

Forward shuttling

Backward shuttling

Dp: Overall proton

diffusion

Environmental science Porous materials design ...

Proton diffusion in bulk water

11[1] H. Nakai, A. W. Sakti, Y. Nishimura, J. Phys. Chem. B 120, 217 (2016).[2] Z. Luz, S. Meiboom, J. Am. Chem. Soc. 86, 4768 (1964).

h(t

)

thtthth

30

20

10

000 4 8 12

300 K320 K360 K400 K

Time [ps]

523H2O + 1H+, T = 300 K

T [K] This work[1] Experiment[2]

300 0.69 0.67

320 0.88 0.86

360 1.25 1.31

400 1.84 1.83

Proton transfer rate [ps−1]dt

tdhr

)(p

Forward shuttling

Backward shuttling

1

1

0

th

No shuttling

Forward

Backward

12

Proton diffusion in bulk water

[1] N. Agmon, Chem. Phys. Lett. 244, 456 (1995). [2] C. M. Maupin, B. Aradi, G. A. Voth, J. Phys. Chem. B 114, 6922 (2010). [3] P. Goyal, M. Elstner, Q. Cui, J. Phys. Chem. B 115, 6790 (2011). [4] H. Nakai, A. W. Sakti, Y. Nishimura, J. Phys. Chem. B 120, 217 (2016). [5] R. Mills, J. Phys. Chem. 77, 685 (1973). [6] S. Meiboom, J. Chem. Phys. 34, 375 (1961).[7] N. K. Roberts, H. L. Northey, J. Chem. Soc. Faraday Trans. 70, 253 (1974).

T = 300 K CPMD[2] DFTB3-diag[3] This work[4] Experiment

N(H+)/N(H2O)a 1/128 1/128 1/523

Dv [Å2/ps] 0.10b 0.38b 0.19b 0.23[5]

DG [Å2/ps] 0.23b 0.28b 0.72b 0.70[6]

Dp [Å2/ps] 0.33b 0.66b 0.91b 0.94[7]

DG/Dv 2.30b 0.74b 3.79b 3.04[6]

Dp/Dv 3.30b 1.74b 4.77b 4.09[6]

• Dp: Proton diffusion

• Dv: Vehicular diffusion

• DG: Grotthuss diffusion

• l: Proton hopping length (2.5 Å)[1]p

22

OO

Gvp66

)0()(lim r

l

t

rtrDDD

t

aNumber of species in the simulated system bDefined as Dp − Dv

Proton diffusion in bulk water

13

[kJ/mol] This work[1] Experiment[2]

ΔEv 9.69

ΔEG 9.38 10.04

ΔEp 9.47

RT

EDD

0lnln

Vehicular diffusion

Grotthuss diffusion

Proton diffusion

Grotthuss diffusion (Experiment)

2.4

1000/T [K−1]

2.6 2.8 3.0 3.2 3.4 3.6

1

ln(D

)

0

−1

−2

[1] H. Nakai, A. W. Sakti, Y. Nishimura, J. Phys. Chem. B 120, 217 (2016).[2] Z. Luz, S. Meiboom, J. Am. Chem. Soc. 86, 4768 (1964).

• ΔEv: Water diffusion barrier for protonated system

• ΔEG: Grotthuss diffusion barrier• ΔEp : Proton diffusion barrier

ΔEv

ΔEG

ΔEp

Arrhenius plot for diffusion constant

14

Application to CO2 capture and separation process

[1] P. Tans, NOAA/ESRL, (www.esrl.noaa.gov/gmd/ccgg/trends/, data accessed October 12, 2016).[2] GISTEMP Team 2016, GISS Surface Temperature Analysis (GISTEMP), (data.giss.nasa.gov/gistemp/, data accessed October 12, 2016).[3] International Energy Agency, Energy Technology Perspectives 2016, (www.iea.org/etp2016, data accessed October 12, 2016).

• CO2: Main source of greenhouse gases causing global warming

• From 6 ˚C scenario (6DS, current trends) to 2˚C scenario (2DS)

1960 1980 2000 2020

320

340

360

380

400

Year

Ato

mo

sph

eri

cC

O2

[pp

m]

−0.2

0.0

0.2

0.4

0.6

0.8

Glo

bal

tem

per

atu

re a

no

mal

y [˚

C]

CO2 at Mauna Loa

Temperature

CO2 and temperature[1,2]

Carbon capture and storage (CCS) is one of the critical components for future reduction of CO2 emissions

Technologies against CO2 emissions[3]

1990 2010 2030 2050Year

00

20

40

60

CO

2em

issi

on

[G

tCO

2]

6DS

2DS

End-use energy

Renewables

CCS

Fossil fuel

Nuclear

Power generation efficiency

15

Carbon capture and storage (CCS)

[1] B. Smit, J. R. Reimer, C. M. Oldenburg, I. C. Bourg, Introduction to Carbon Capture and Sequestration, Imperial College Press, London, 2014.[2] K. Teranishi, A. Ishikawa, H. Nakai, J. Comput. Chem. Jpn. 15, A15 (2016).

• Capture CO2 from sources before releasing

• Store in geological formation through pipeline transportation

Chemical absorption is a well-known technology with industrial applicability and suitable for retrofit

Process flow of CCS[1] Main approaches in CO2 capture[2]

16

CO2 Chemical absorption

[1] K. Teranishi, A. Ishikawa, H. Nakai, J. Comput. Chem. Jpn. 15, A15 (2016).

CO2 is scrubbed with amine solution before releasing from sources

Process flow of chemical absorption[1]

Design of high performance and low cost amine solution is required

17

Reaction of amine-CO2 system[1]

[1] K. Teranishi, A. Ishikawa, H. Nakai, J. Comput. Chem. Jpn. 15, A15 (2016).

Reactant Product

Product

18

Simulation of CO2 regeneration process[1]

[1] H. Nakai, Y. Nishimura, T. Kaiho, T. Kubota, H. Sato, Chem. Phys. Lett. 647, 127 (2016).

Absorption process

Regeneration process

T = 393.15 K

5 10 15 20 25

Time [ps]

180

160

140

120

OC

O a

ngl

e [d

egre

e]

100

19

Simulation of CO2 regeneration process[1]

Collision between protonated amine (cation) and carbamate (anion)

14.8 ps8.0 ps

−+

+

−

180

160

140

120

OC

O a

ngl

e [d

egre

e]

2.5

2.0

1.5

1.0

0.5

Inte

rmo

lecu

lar

dis

tan

ce [

Å]

5 10 15 20 25Time [ps]

100

Reactive collision

Non-reactive collision17.7 ps 22.0 ps

Proton transfer

CO2

AmineAmine

Carbamate

Protonated amine

Carbamate

Protonated amine

[1] H. Nakai, Y. Nishimura, T. Kaiho, T. Kubota, H. Sato, Chem. Phys. Lett. 647, 127 (2016).

Summary and Acknowledgement

20

QM-MD Classical MDWFT DFT DFTB DC-DFTB MM

Cluster collisionElectrolyte

decompositionCarbon nanotube

growthCO2 chemical

absorptionVirus

< 100 < 1000 < 1000 ~100000 ~10000000

• Summary: Chemical reaction dynamics with DC-DFTB-MD DC: Linear-scaling computation DFTB: Small pre-factor with acceptable accuracy K: Single node to massively parallel calculation by hybrid parallelization Applications to aqueous solution systems

• Acknowledgement FLAGSHIP2020, MEXT within the priority study 5

HPCI system research project for using the K computer (Project ID: hp160215)