Massively Parallel Simulations of Particulate ... · 10/5/2018 · Massively Parallel Simulations...

Indo-German Symposium

IIT Delhi, India

Massively Parallel Simulations of Particulate Electrokinetic Micro-fluid Flows

Dominik Bartuschat, Ulrich Rüde

Chair for System Simulation, FAU Erlangen-Nürnberg

October 5, 2018

Advanced Measurements and Multiscale

CFD Simulations for Intensification of

Multiphase Flow Processes

“ASimulations for Intensification of Multiphase Flow Processes”

FAU-ErlangenIIT Delhi

Dominik Bartuschat | Chair for System Simulation | Massively Parallel Simulations of Particulate Electrokinetic Micro-fluid Flows October 5, 2018 2

Interactions of large numbers of charged particles in fluid flows,influenced by external electric fields

• Medical applications • Optimization of Lab-on-a-Chip systems

• Sorting of different cells

• Trapping cells and viruses

• Deposition of charged aerosol particles in respiratory tract

• Industrial applications • Filtering particulates from (exhaust) gases • Particle deposition in cooling systems

• …

Motivation

© Kang and Li „Electrokinetic motion of particles and cells in microchannels“ Microfluidics and Nanofluidics


• Widely applicable Lattice Boltzmann framework

• Suited for various flow applications

• Large-scale, distributed memory parallelization: • Domain partitioned into Cartesian grid of blocks

• Blocks can be assigned to different processes

• Ghost layers to exchange cell data between blocks

waLBerla

Simulation Domain

Block

Ghost Layer

Cell /Unknown


Hydrodynamic Interactions

Electro-

quasi-statics

Fluiddynamics

Rigid body dynamics

hydrodynamic force

object motion

ion motion

force on ions

elec

trostat

ic fo

rce

char

ge d

ensity

Dominik Bartuschat | Chair for System Simulation | Massively Parallel Simulations of Particulate Electrokinetic Micro-fluid Flows October 5, 2018

Domain discretized into cubic cells

Discrete velocities and associated distribution functions per cell

5

The Lattice Boltzmann Method

D3Q19 modelIllustration by Klaus Iglberger

fq(~xi +~cqdt, tn+dt)− fq(~xi , tn) = Ωq (fq(~xi , tn))fq(~xi +~cqdt, tn+dt)− fq(~xi , tn) = Ωq (fq(~xi , tn))

fq

Discrete lattice Boltzmann equation Describes probabilities fq that fluid molecules move

with given discrete velocities

Molecule collisions represented by collision operator Ωq

~cq


The equation is solved in two steps: • Stream step:

• Collide step:

6

The Lattice Boltzmann Method

fq(~xi , tn) = fq(~xi , tn)−1

τ

fq(~xi , tn)− f

eqq (~xi , tn)

| z

Ωq with single relaxation time

fq(~xi +~eq, tn+dt) = fq(~xi , tn)

Fluid viscosity νf determined by τ , fluid velocity~u, and density ρf


Fluid-Particle Interaction

Illustration by Jan Götz

* N. Nguyen, A. Ladd. „Lubrication corrections for lattice-Boltzmann simulations of particle suspensions“ Phys. Rev. E (2002), doi:10.1103/PhysRevE.66.046708

Hydrodynamic forces computed by momentum exchange method*

Four-way coupling


• Tumbling motion of elongated particles in Stokes flow

• Spherocylinders in periodic domain, aspect ratio

• Comparison against slender body formulation [*]

Tumbling Spherocylinders

* D. Bartuschat, E. Fischermeier, K. Gustavsson, U. Rüde. „Two Computational Models for Simulating the Tumbling Motion of Elongated Particles in Fluids“, Comput. Fluids (2016), doi:10.1016/j.compfluid.2015.12.010

Typical runs on SuperMUC:

on 8192 cores

for 15 to 48 hrs

(70 000 to 605 000 time steps)

1/ε =L

R= 12


Flow field for spherocylinders of 1/ε =12

9

Two Tumbling Spherocylinders

Time steps: 600 000 Runtime on LiMa: 16 h, 768 cores

Domain size: [576dx]3 dx = 4.98µm, R = 4dx

Fluid: Water (20 C) ρP = 1492kg/m3


Charged Particles in Fluid Flow

Electro-

quasi-statics

Fluiddynamics

Rigid body dynamics

hydrodynamic force

object motion

ion motion

force on ions

elec

trostat

ic fo

rce

char

ge d

ensity


• Electric potential described by Poisson equation with particle‘s charge density on RHS:

• Discretized by finite volumes on lattice

• Solved with cell-centered multigrid (MG) solverimplemented in waLBerla

• Supersampling for computing overlap degree

to set RHS more accurately

• Electrostatic force on particle:

Electrostatic Force on Particles

−∆Φ(~x) =ρe (~x)

εe

~FC =−qparticle ·∇Φ(~x)

Illustration by Jan Götz

ρe


6-Way Coupling Approach

hydrodynam. force

object motion

Lubricationcorrection

electrostat. force

velocity BCs

object distance

LBM

correction force

charge distribution

Newtonian mechanicscollision response

treat BCsstream-collide step

Finite volumes

MGiterat.

treat BCsV-cycle

D. Bartuschat, U. Rüde. „Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows“, J. Comput. Sci. (2015), doi:10.1016/j.jocs.2015.02.006


Analytical solution for homogeneously charged particle in infinite domain:

Validation of Electric Potential

0 50 100 150 200 2500

0.1

0.2

0.3

·10−3

xL

Φ/V

Analytical solution

Numerical solution

Sphere surface

~r

qe

Rx

Domain: [256 dx]3

Dirichlet BCs: exact solution

MG: 5 V(2,2)-cycles

Residual threshold 2⋅10-9

Φ(~r) =1

4πεe

qe

|~r |if |~r |≥ R

Radius: R = 600µm

Charge: qe = 8000e

Supersampling: factor 3


Charged Particles in Fluid Flow

•Computed on 144 cores (12 nodes) of RRZE - LiMa

•210 000 time steps

•15.7h runtime

•643 unknowns per core

Channel: 2.56mm7.68mm1.92mm Particles: R = 80µm, qe =±40 000e Charged plates: Φ =±76.8V

dx = 10µm,τ = 1.7,dt = 40µs Water (20 C), Inflow 1 mm/s, Outflow 0 Pa else: no-slip & insulating BCs


Parallel Scaling Experiments

Weak scaling: Constant size per core:

1283 cells

9.4% moving obstacle cells

Size doubled (in all dimensions) Cell-centered multigrid

V(3,3) with 7 levels

6 to 116 CG coarse-grid iterations

Convergence rate: 0.07

2x4x2 cores per node

Experiments on SuperMUC: 18 thin islands with 512 compute nodes, each:

16 cores (2 Xeon chips) @2.5 GHz

32 GB DDR3 RAM

Ranked #6 in TOP500 during experiments


Weak Scaling for 240 Time Steps

0 250 500 750 1000 1250 1500 1750 2000Number of nodes

0

10

20

30

40

50

60

70

80

90

10

3 M

FL

UP

S (

LB

M)

LBM Perform.20

40

60

80

100

120

10

3 M

LU

PS

(M

G)

MG Perform.

1 2 4 8 16 32 64 128

256

512

1024

2048

0

100

200

300

400

Number of nodes

To

tal

run

tim

es/

s

LBM

Map

Lubr

HydrFpe

MG

SetRHS

PtCm

ElectF

32 768 cores 7.1M particles

Parallel efficiency

@ 2048 nodes:

Overall 83 %

LBM 91 %

MG - 1 V(3,3) 64 %

MG performance restricted by coarsest-grid solving

D. Bartuschat, U. Rüde. „Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows“, J. Comput. Sci. (2015), doi:10.1016/j.jocs.2015.02.006


Electrophoresis

Electro-

quasi-statics

Fluiddynamics

Rigid body dynamics

hydrodynamic force

object motion

ion motion

force on ions

elec

trostat

ic fo

rce

char

ge d

ensity

Equilibrium description of electrical double layer → 7.5-way coupling


Electrical double layer potential

Electrostatic force on double layer charge

Lattice Boltzmann equation with body force term

18

Electrical Double Layer Effect

~fb = ρe (ψ) ·~Eext

fq(~xi +~cqdt, t+dt)− fq(~xi , t) = Ωq +dtFq(~fb)

Symmetric Poisson-Boltzmann equation (PBE)

−∆ψ =−

2z en∞

εesinh

z eψ

kB T

−∆ψ =−κ2 ψ, κ = λ−1

D

Debye-Hückel approximation (DHA) for |ζ |< 25mV


Analytical solution for sphere with uniform surface charge in infinite domain (DHA):

19

Validation of Electric Potential

Domain: [128 dx]2 x 256 dx

SOR solver Dirichlet BCs: exact solution

Residual reduction 2⋅10-7

ψ(~r) = ζR

|~r|e−κ(|~r|−R)

if |~r|≥ R

r

θ

ψ = ζ

R

λD

x

Radius: R = 120nm

Charge: qs = −124e

Fluid: Water (20 C)

κ R 0.890 20 40 60 80 100 120

−10

−8

−6

−4

−2

0

·10−3

xL

ψ/V

Analytical

solution

Numerical

solution


Analytical solution for sphere with uniform surface charge in infinite domain (Stokes flow):

• Validation in large, finite domains (SuperMUC, up to 8192 cores):

• Relative error in electric effects below 1%

20

Electrophoretic Velocity Validation

RL 4 6 8 9 12

κR 0.27 0.40 0.53 0.60 0.80

Retardation / % -21 -28 -34 -36 -43

dev. from UEP/% -2.56 -2.48 -3.59 -3.51 -3.91

dev. from UStokes/% -3.04 -2.65 -2.98 -2.89 -3.00

difference /% 0.5 0.2 -0.6 -0.6 -0.9

dx = 5nm

ζ = 10mV

Eext = 99 · 10 6 V/m

Fluid: Water (20 C)

λD = 75nm

Rep,d = 0.018 ...0.057

~UEP =2εeζ

3µf

0

B

B

B

@

1+1

2

1+ 2.5

κR(1+2e−κR)

3

1

C

C

C

A

~Eext

Eext

ψ = ζ

R

λD

~UEP


Electrophoresis of charged particle along channel axis • Result after 30 000 time steps

• Flow field, double layer potential, and ion charge distribution

Electrophoresis in Micro-Channel

U∗

EP = 0.21m

s

Channel: 1.28µm2.56µm1.28µm Particle: R = 120nm, qe = −124e ζ = −10mV, Ey =4.7 · 10 6 V/m

Fluid: Water (20 C), c∞ = 5µmol/l dx = 10nm,τ = 6.5,dt = 0.2ns BCs: Periodic in axial direction,else insulating & no-slip BCs


Weak Scaling for 240 Time Steps

65 536 cores 4.2M particles

Parallel efficiency

@ 4096 nodes:

Overall 87 %

LBM 94 %

SOR 82 %

D. Bartuschat, U. Rüde. „A scalable multiphysics algorithm for massively parallel direct numerical simulations of electrophoretic motion“, J. Comput. Sci. (2018), doi:10.1016/j.jocs.2018.05.011

100 500 1000 1500 2000 2500 3000 3500 40000

40

80

120

160

200

240

Number of nodes

10

3M

FL

UP

S(L

BM

)

20

40

60

80

100

120

10

3M

FL

UP

S(S

OR

)

LBM Perform.

SOR Perform.

RL=6

1443 cells per core

1.95% moving obstacle cells

1 2 4 8 16 32 64 128

256

512

1024

2048

4096

0

200

400

600

800

Number of nodes

To

tal

run

tim

es/s

LBM

MOmap

HydrForcepe

LBMForce

EDLField

SOR

PotBCmap

Thank you for your attention!

S. Bogner, U. Rüde. „ Simulation of floating bodies with lattice Boltzmann“ (2012), doi:10.1016/j.camwa.2012.09.012

Massively Parallel Simulations of Particulate ... · 10/5/2018 · Massively Parallel Simulations...

Documents

Transcript of Massively Parallel Simulations of Particulate ... · 10/5/2018 · Massively Parallel Simulations...