Chacon Plenary
Transcript of Chacon Plenary
-
8/2/2019 Chacon Plenary
1/38
S c a l a b l e i m p l i c i t a l g o r i t h m s f o r s t i h y p e r b o l i c P D E s y s t e m s
L . C h a c n
O a k R i d g e N a t i o n a l L a b o r a t o r y
J . N . S h a d i d , E . C y r , P . T . L i n , R . T u m i n a r o , R . P a w l o w s k i
S a n d i a N a t i o n a l L a b o r a t o r i e s
D O E A p p l i e d M a t h e m a t i c s P r o g r a m M e e t i n g
O c t o b e r 1 7 - 1 9 , 2 0 1 1
W a s h i n g t o n , D C
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
2/38
O u t l i n e
M o t i v a t i o n : t h e t y r a n n y o f s c a l e s
B l o c k - f a c t o r i z a t i o n p r e c o n d i t i o n i n g o f h y p e r b o l i c P D E s
C o m p r e s s i b l e r e s i s t i v e M H D
C o m p r e s s i b l e e x t e n d e d M H D
I n c o m p r e s s i b l e N a v i e r - S t o k e s a n d M H D ( i n n i t e s o u n d - s p e e d l i m i t )
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
3/38
T h e t y r a n n y o f s c a l e s
( 2 0 0 6 N S F S B E S r e p o r t )
F i g u r e 1 : T i m e s c a l e s i n f u s i o n p l a s m a s ( F S P r e p o r t )
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
4/38
A l g o r i t h m i c c h a l l e n g e s i n t e m p o r a l s c a l e - b r i d g i n g
P D E s y s t e m s o f i n t e r e s t t y p i c a l l y h a v e m i x e d c h a r a c t e r , w i t h h y p e r b o l i c a n d p a r a b o l i c c o m p o n e n t s .
J H y p e r b o l i c s t i n e s s ( l i n e a r a n d d i s p e r s i v e w a v e s ) : (J) t f ast ttCF L 1
J P a r a b o l i c s t i n e s s ( d i u s i o n ) : (J) t Dx2
1
I n s o m e a p p l i c a t i o n s , f a s t h y p e r b o l i c m o d e s c a r r y a l o t o f e n e r g y ( e . g . , s h o c k s , f a s t a d v e c t i o n o f
s o l u t i o n s t r u c t u r e s ) , a n d t h e m o d e l e r m u s t f o l l o w t h e m .
I n o t h e r s , h o w e v e r , f a s t t i m e s c a l e s a r e p a r a s i t i c
, a n d c a r r y v e r y l i t t l e e n e r g y .
J T h e s e a r e t h e o n e s t h a t a r e u s u a l l y t a r g e t e d f o r s c a l e - b r i d g i n g .
B r i d g i n g t h e t i m e - s c a l e d i s p a r i t y r e q u i r e s a c o m b i n a t i o n o f a p p r o a c h e s :
J A n a l y t i c a l e l i m i n a t i o n ( e . g . , r e d u c e d m o d e l s ) .
J W e l l - p o s e d n u m e r i c a l d i s c r e t i z a t i o n ( e . g . , a s y m p t o t i c p r e s e r v i n g m e t h o d s )
J S o m e l e v e l o f i m p l i c i t n e s s i n t h e t e m p o r a l f o r m u l a t i o n ( f o r s t a b i l i t y ; a c c u r a c y r e q u i r e s c a r e ) .
K e y a l g o r i t h m i c r e q u i r e m e n t : S C A L A B I L I T Y
CPU O
N
np
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
5/38
A l g o r i t h m i c s c a l a b i l i t y v s . p a r a l l e l s c a l a b i l i t y
" T h e t y r a n n y o f s c a l e s w i l l n o t b e s i m p l y d e f e a t e d b y b u i l d i n g b i g g e r a n d f a s t e r c o m p u t e r s "
( N S F S B E S 2 0 0 6 r e p o r t , p . 3 0 )
O p t i m a l a l g o r i t h m : CPU N/np .
CPU N1+
n1p
; N =
L
d 0, algorithmic scalability 0, parallel scalability
M u c h e m p h a s i s h a s b e e n p l a c e d o n p a r a l l e l s c a l a b i l i t y ( ) .
H o w e v e r , p a r a l l e l ( w e a k ) s c a l a b i l i t y i s l i m i t e d b y t h e l a c k o f a l g o r i t h m i c s c a l a b i l i t y :
J N np CPU n+p r e q u i r e s = = 0!
E x p l i c i t I m p l i c i t ( d i r e c t ) I m p l i c i t ( K r y l o v i t e r a t i v e ) I m p l i c i t ( m u l t i l e v e l )
= 1/d = 2 2/d > 1 (varies) 0
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
6/38
H o w d o m u l t i l e v e l ( m u l t i g r i d ) m e t h o d s w o r k ?
M G e m p l o y s a d i v i d e - a n d - c o n q u e r a p p r o a c h t o a t t a c k e r r o r c o m p o n e n t s i n t h e s o l u t i o n .
J O s c i l l a t o r y c o m p o n e n t s o f t h e e r r o r a r e E A S Y t o d e a l w i t h ( i f a S M O O T H E R e x i s t s )
J S m o o t h c o m p o n e n t s a r e D I F F I C U L T .
I d e a : c o a r s e n g r i d t o m a k e " s m o o t h " c o m p o n e n t s a p p e a r o s c i l l a t o r y , a n d p r o c e e d r e c u r s i v e l y
S M O O T H E R i s m a k e o r b r e a k o f M G !
S m o o t h e r s a r e h a r d t o n d f o r h y p e r b o l i c s y s t e m s , b u t f a i r l y e a s y f o r p a r a b o l i c o n e s :
C a n o n e m a k e h y p e r b o l i c P D E s M G - f r i e n d l y ?
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
7/38
I m p l i c i t d i s c r e t i z a t i o n o f h y p e r b o l i c P D E s : a c a s e s t u d y
tu =
1
xv , tv =
1
xu ; =
k
i s a m e a s u r e o f h y p e r b o l i c s t i n e s s . D i s c r e t i z e i m p l i c i t l y i n t i m e :
un+1 = un +1
xv
n+1 , vn+1 = vn +1
xu
n+1.
V e r y i l l c o n d i t i o n e d a s
0! H o w e v e r , i f o n e c o m b i n e s e q u a t i o n s :
I
t
22x
un+1 = un +
t
xv
n
E q u a t i o n i s n o w w e l l - p o s e d w h e n 0 ( i . e . , i t i s a s y m p t o t i c - p r e s e r v i n g ) !
J L i m i t s y s t e m i s e l l i p t i c / p a r a b o l i c ( M G - f r i e n d l y ! )
JT e m p o r a l l y u n r e s o l v e d
h y p e r b o l i c t i m e s c a l e s h a v e b e e n p a r a b o l i z e d .
N o f u r t h e r m a n i p u l a t i o n o f P D E t h a n i m p l i c i t d i e r e n c i n g ( n o t e r m s a d d e d t o P D E ) !
T h i s f a c t c a n b e e x p l o i t e d t o d e v i s e o p t i m a l s o l u t i o n a l g o r i t h m s ( b l o c k f a c t o r i z a t i o n ) !
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
8/38
B l o c k - f a c t o r i z a t i o n o f h y p e r b o l i c P D E s
un+1 = un +
t xvn+1 , vn+1 = vn +
t xun+1
C o u p l i n g s t r u c t u r e : I t x
t x I
un+1
vn+1
=
un
vn
2
2 b l o c k c a n b e f o r m a l l y i n v e r t e d v i a b l o c k f a c t o r i z a t i o n :
D1
1U
1
L D2
=
I 1UD
12
0 I
D1
12
UD12 L 0
0 D2
I 0
1
D12 L I
O n l y i n v e r s e o f D1 UD12 L ( S c h u r c o m p l e m e n t ) i s r e q u i r e d !
D11
2UD12 L = I
t
22x
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
9/38
N o n l i n e a r h y p e r b o l i c P D E s :
J F N K a n d b l o c k f a c t o r i z a t i o n p r e c o n d i t i o n i n g
O b j e c t i v e : s o l v e n o n l i n e a r s y s t e m G(xn+1) =0 e c i e n t l y ( s c a l a b l y ) .
C o n v e r g e n o n l i n e a r c o u p l i n g s u s i n g N e w t o n - R a p h s o n m e t h o d : G
x
k
xk = G(xk) .
J a c o b i a n - f r e e i m p l e m e n t a t i o n :Gx
k
y = Jky = lim0
G(xk +
y)
G(xk)
K r y l o v m e t h o d o f c h o i c e : G M R E S ( n o n s y m m e t r i c s y s t e m s ) .
R i g h t p r e c o n d i t i o n i n g : s o l v e e q u i v a l e n t J a c o b i a n s y s t e m f o r y = Pkx :
JkP
1
k Pkx
y=
Gk
A p p r o x i m a t i o n s i n p r e c o n d i t i o n e r d o n o t a e c t a c c u r a c y o f c o n v e r g e d s o l u t i o n ; o n l y e c i e n c y !
B l o c k - f a c t o r i z a t i o n + M G w i l l b e o u r p r e c o n d i t i o n i n g s t r a t e g y .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
10/38
I m p l i c i t r e s i s t i v e M H D s o l v e r
L . C h a c o n , P h y s . P l a s m a s
( 2 0 0 8 )
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
11/38
R e s i s t i v e M H D m o d e l e q u a t i o n s
t+ (v) = 0,
B
t+ E = 0,
(v)
t + vv BB v +I (p + B2
2 ) = 0,T
t+v T + ( 1)T v = 0,
P l a s m a i s a s s u m e d p o l y t r o p i c p n .
R e s i s t i v e O h m ' s l a w :
E = v B + B
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
12/38
R e s i s t i v e M H D J a c o b i a n b l o c k s t r u c t u r e
T h e l i n e a r i z e d r e s i s t i v e M H D m o d e l h a s t h e f o l l o w i n g c o u p l i n g s :
= L(, v)
T = LT(T, v)
B = LB(B, v)
v = Lv(v, B, , T)
T h e r e f o r e , t h e J a c o b i a n o f t h e r e s i s t i v e M H D m o d e l h a s t h e f o l l o w i n g c o u p l i n g s t r u c t u r e :
Jx =
D 0 0 Uv
0 DT 0 UvT
0 0 DB UvB
Lv LTv LBv Dv
T
B
v
D i a g o n a l b l o c k s c o n t a i n a d v e c t i o n - d i u s i o n c o n t r i b u t i o n s , a n d a r e e a s y t o i n v e r t u s i n g M G
t e c h n i q u e s . O d i a g o n a l b l o c k s L a n d U c o n t a i n a l l h y p e r b o l i c c o u p l i n g s .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
13/38
B l o c k f a c t o r i z a t i o n o f r e s i s t i v e M H D
W e c o n s i d e r t h e b l o c k s t r u c t u r e :
Jx =
M U
L Dv
y
v
; y =
T
B
; M =
D 0 0
0 DT 0
0 0 DB
M i s e a s y t o i n v e r t ( a d v e c t i o n - d i u s i o n , n o t v e r y s t i , M G - f r i e n d l y ) .
S c h u r c o m p l e m e n t a n a l y s i s o f 2 x 2 b l o c k J y i e l d s :
M U
L Dv
1=
I 0
LM1 I
M1 0
0 P1Schur
I M1U
0 I
,
PSchur = Dv LM1U .
E X A C T J a c o b i a n i n v e r s e o n l y r e q u i r e s M1 a n d P1Schur .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
14/38
P h y s i c s - b a s e d p r e c o n d i t i o n e r ( P B P )
3 - s t e p E X A C T i n v e r s i o n a l g o r i t h m :
Predictor : y = M1Gy
Velocity update : v = P1Schur[Gv Ly], PSchur = Dv LM
1U
Corrector : y =
y
M
1
Uv
M G t r e a t m e n t o f PSchur i s i m p r a c t i c a l d u e t o M1
.
W e c o n s i d e r h e r e t h e s m a l l - f l o w l i m i t : v vA M1 t I ( c h e a p )
W e h a v e e x t e n d e d t h e f o r m u l a t i o n t o a r b i t r a r y - o w s , v vA b a s e d o n c o m m u t a t i o n i d e a s 1 ( m o r e
e x p e n s i v e , b u t m o r e r o b u s t
2) .
1E l m a n , S I S C 2 7 , 1 6 5 1 ( 2 0 0 6 )
2L . C h a c n , J . P h y s i c s : C o n f . S e r i e s , 1 2 5 , 0 1 2 0 4 1 ( 2 0 0 8 )
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
15/38
P B P : S m a l l - o w l i m i t
S m a l l o w a p p r o x i m a t i o n : M1 t I i n s t e p s 2 & 3 o f S c h u r a l g o r i t h m :
y = M1 Gy
v P1SI [Gv Ly] ; PSI = Dv tLU
y y tUv
w h e r e :
PSI = nI/t + (v0 I + I v0
n2I)
+ t2W(B0, p0)
W(B0, p0) = B0 [I B0]j0 [I B0][I p0 + p0 I]
O p e r a t o r W(B0, p0) i s i d e a l M H D e n e r g y o p e r a t o r , w h i c h h a s r e a l e i g e n v a l u e s !
PSI i s p a r a b o l i c , a n d h e n c e b l o c k d i a g o n a l l y d o m i n a n t b y c o n s t r u c t i o n !
W e e m p l o y m u l t i g r i d m e t h o d s ( M G ) t o a p p r o x i m a t e l y i n v e r t PSI a n d M : 1 V ( 4 , 4 ) c y c l e
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
16/38
P B P : 2 D s e r i a l p e r f o r m a n c e ( t e a r i n g m o d e )
G r i d c o n v e r g e n c e s t u d y ( t = 1.0 A )
NG M R E S /
t CPUex p/CPU
t/
tCF L3 2 x 3 2 1 4 2 . 4 3 1 5 9
6 4 x 6 4 1 1 . 8 5 . 8 3 2 2
1 2 8 x 1 2 8 1 1 . 2 1 3 . 3 6 6 7
2 5 6 x 2 5 6 1 1 . 4 2 8 . 5 1 4 2 9
CPU O(N) OPTIMAL SCALING!
t c o n v e r g e n c e s t u d y ( 1 2 8 x 1 2 8 )
t G M R E S / t CPUex p/CPU t/tCF L
0 . 5 8.0 8 . 0 3 8 0
0 . 7 5
9.51 0 . 0 5 7 0
1 . 0 11.2 1 2 . 7 7 6 0
1 . 5 14.6 1 4 . 6 1 1 4 0
CPU O(t0.6) FAVORABLE SCALING!
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
17/38
P B P : 3 D s e r i a l p e r f o r m a n c e ( i s l a n d c o a l e s c e n c e )
1 0 t i m e s t e p s , t = 0.1, V ( 3 , 3 ) c y c l e s , m g t o l = 1 e - 2
G r i d G M R E S / t C P U
163 5 . 5 8 1
323 7 . 9 1 1 7 6
643 7 . 0 1 1 1 3 5
L u i s C h a c n , c h a c o n l @ o r n l . g o v
http://home/lch/Documents/artwork/pixie3d/movies/3dic-movie -
8/2/2019 Chacon Plenary
18/38
P B P : 3 D p a r a l l e l p e r f o r m a n c e ( i s l a n d c o a l e s c e n c e )
( W e a k s c a l i n g , 163 p o i n t s p e r p r o c e s s o r , C r a y X T 4 ) t = 0.1 tCF L
K e y t o p a r a l l e l p e r f o r m a n c e :
J M a t r i x - l i g h t m u l t i g r i d , w h e r e o n l y d i a g o n a l s a r e s t o r e d ; r e s i d u a l s a r e c a l c u l a t e d m a t r i x - f r e e .
J O p e r a t o r c o a r s e n i n g v i a r e d i s c r e t i z a t i o n : a v o i d s f o r m i n g / c o m m u n i c a t i n g a m a t r i x .
C u r r e n t l i m i t a t i o n s : w e d o n o t f e a t u r e a c o a r s e - s o l v e b e y o n d t h e p r o c e s s o r s k e l e t o n g r i d .
J T h i s e v e n t u a l l y d e g r a d e s a l g o r i t h m i c s c a l a b i l i t y ( o n l y s h o w s a t > 1 0 0 0 - p r o c e s s o r l e v e l ) .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
19/38
I m p l i c i t e x t e n d e d M H D s o l v e r
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
20/38
E x t e n d e d ( t w o - u i d , H a l l ) M H D m o d e l e q u a t i o n s
t+ (v) = 0,
Bt
+ E = 0,
(v)
t+
vv BB +
+
I (p +
B2
2)
= 0,
Tet
+v Te + ( 1)Te v = ( 1)
Q q
(1 + ),
=
i +
e ;
e = eve ; ve = v di
j
; v = v
di1 +
j
; =
TiTe
OhmsLaw :
E = v B + j + di (
j Bpe e ) electron EOM
E =
v
B +
j + di[t
v +
v
v +
1
(pi +
i )] ion EOM
N o t e t h a t E O M i - E O M e = E O M . A d m i t s a n e n e r g y p r i n c i p l e .
T h i s m o d e l s u p p o r t s f a s t d i s p e r s i v e w a v e s k2 .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
21/38
E x t e n d e d M H D J a c o b i a n b l o c k s t r u c t u r e : e l e c t r o n E O M
( s t a n d a r d c h o i c e )
E = v B + j + di
(j Bpe e )
L i n e a r i z e d i n d u c t i o n e q u a t i o n B = E h a s t h e f o l l o w i n g c o u p l i n g s :
B = LB(B, v, , T)
J a c o b i a n c o u p l i n g s t r u c t u r e :
Jx =
D 0 0 Uv
LTB DT UBT UvT
LB LTB DB UvB
Lv LTv LBv Dv
T
B
v
W e h a v e a d d e d o - d i a g o n a l c o u p l i n g s t o b l o c k M .
S t i e s t b l o c k i s DB b r e a k s a p p r o x i m a t i o n s i n b l o c k - f a c t o r i z a t i o n a p p r o a c h . U N S U I T A B L E !
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
22/38
E x t e n d e d M H D J a c o b i a n b l o c k s t r u c t u r e : i o n E O M
E v B + j + di[tv +v v +1
(pi +
i [v])]
H a l l c o u p l i n g i s m a i n l y v i a tv .
J a c o b i a n c o u p l i n g s t r u c t u r e b e c o m e s :
Jx
D 0 0 Uv
0 DT 0 UvT
0 0 DB URvB + U
HvB
Lv LTv LBv Dv
T
B
v
W e c a n t h e r e f o r e r e u s e A L L r e s i s t i v e M H D P C f r a m e w o r k !
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
23/38
E x t e n d e d M H D p r e c o n d i t i o n e r
U s e s a m e b l o c k f a c t o r i z a t i o n a p p r o a c h .
M b l o c k c o n t a i n s i o n t i m e s c a l e s o n l y M1 t I i s a v e r y g o o d a p p r o x i m a t i o n
A d d i t i o n a l b l o c k UHvB :
PSIv = nv/t + (v0 v + v v0 +
)
+ t2W(B0, p0)v
W(B0, p0) = B0 [I B0 dit
I ]j0 [I B0][I p0 + p0 I]
A d d i t i o n a l t e r m b r i n g s i n d i s p e r s i v e w a v e s k2 !
W e c a n s h o w a n a l y t i c a l l y t h a t a d d i t i o n a l t e r m ( y e l l o w ) i s a m e n a b l e t o s i m p l e d a m p e d J a c o b i
s m o o t h i n g !
W e c a n u s e c l a s s i c a l M G !
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
24/38
O n t h e i s s u e o f d i s s i p a t i o n i n e x t e n d e d M H D
D i s p e r s i v e w a v e s
k2
r e q u i r e h i g h e r o r d e r d i s s i p a t i o n .
R e s i s t i v i t y i s u n a b l e t o p r o v i d e a d i s s i p a t i o n s c a l e .
D i s s i p a t i o n s c a l e d e n e d b y e l e c t r o n v i s c o s i t y , e :
e e
2( v) e4v
V i s c o s i t y c o e c i e n t c a n b e d e t e r m i n e d t o p r o v i d e a d e q u a t e d i s s i p a t i o n o f d i s p e r s i v e w a v e s
vAdik2 : ek
4 e > CdivAk,max
k3max
I n t h e p r e c o n d i t i o n e r , w e d e a l w i t h e b y c o n s i d e r i n g 2 s e c o n d - o r d e r s y s t e m s , a n d s o l v i n g
t h e m c o u p l e d w i t h i n M G .
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
25/38
E x t e n d e d M H D p e r f o r m a n c e r e s u l t s
( 2 D t e a r i n g m o d e )
di = 0.05, e = 2.5 106
1 0 0 t i m e s t e p s , t = 1.0 , 1 V ( 4 , 4 ) M G c y c l e
G r i d G M R E S / t CPUex p/CPU t/tex p t/tCF L
3 2 x 3 2 2 2 . 3 0 . 7 4 1 3 5 1 1 0
6 4 x 6 4 1 5 . 4 1 0 . 9 1 5 8 2 3 8 4
1 2 8 x 1 2 8 1 0 . 6 2 1 4 2 3 8 0 9 1 4 3 6
2 5 6 x 2 5 6 1 3 . 1 3 0 9 7 3 7 0 3 7 0 5 6 6 0
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
26/38
2 D n o n l i n e a r v e r i c a t i o n : G E M c h a l l e n g e
I o n H a l l v s . e l e c t r o n H a l l
90
100
110
120
130
140
150
160
0 5 10 15 20 25 30 35 40 45
Kinetic energy
Ion Halle Hall
0
1
2
3
4
5
6
7
8
9
0 5 10 15 20 25 30 35 40 45
Magnetic energy
Ion Halle Hall
6570
75
80
85
90
95
100
105
110
115
120
0 5 10 15 20 25 30 35 40 45
Thermal energy
Ion Halle Hall
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
27/38
I n c o m p r e s s i b l e N a v i e r - S t o k e s s o l v e r
C y r , S h a d i d , T u m i n a r o , J C P 2 0 1 1
E l m a n , H o w l e , S h a d i d , S h u t t l e w o r t h , T u m i n a r o , J C P 2 0 0 8
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
28/38
Block preconditioning:CFD example
Consider discretized Navier-Stokes equations
Properties of block factorization1. Important coupling in Schur-complement
2. Better targets for AMG leveraging scalability
Properties of approximate Schur-complement1. Nearly replicates physical coupling
2. Invertible operators good for AMG
Fully Coupled JacobianFully Coupled Jacobian PreconditionerPreconditioner
Required operators:
Multigrid
PCD, LSC,
SIMPLEC
Block FactorizationBlock Factorization
Coupling in Schur-complement
-
8/2/2019 Chacon Plenary
29/38
Discrete N-S Exact LDU Factorization Approx. LDU
Brief Overview of Block Preconditioning Methods for Navier-Stokes:
(A Taxonomy based on Approximate Block Factorizations, JCP 2008)
Now use AMG type methods on sub-problems.
Momentum transient convection-diffusion:
Pressure Poisson type:
Precond. Type References
Pres. Proj;
1st TermNeumann Series
Chorin(1967);Temam (1969);
Perot (1993): Quateroni et.al. (2000) as solvers
SIMPLEC Patankar et. al. (1980) assolvers; Pernice and Tocci
(2001) as smoothers/MG
PressureConvection /Diffusion
Kay, Loghin, Wathan,Silvester, Elman (1999 -
2006); Elman, Howle, S.,Shuttleworth, Tuminaro(2003,2008)
-
8/2/2019 Chacon Plenary
30/38
Fully coupled AlgebraicAggC: Aggressive Coarsening Multigrid
DD: Additive Schwarz Domain Decomposition
Block PreconditionersPCD & LSC: Commuting Schur complement
SIMPLEC: Physics-based Schur complement
CFD Weak Scaling: Steady Backward Facing Step
* Paper accepted: E. C. Cyr, J. N. Shadid, R. S. Tuminaro, Stabilization and Scalable Block Preconditioning for
the Navier-Stokes Equations, Accepted by J. Comp. Phys., 2011.
Take home: Block preconditioners competitive with fully
coupled multigrid for CFD
Take home: Block preconditioners competitive with fully
coupled multigrid for CFD
W k S li f NK S l ith F ll
-
8/2/2019 Chacon Plenary
31/38
Weak Scaling of NK Solver with Fully-
coupled AMG and Approx. Block
Factorization Preconditioners
Quad-core Nehalemswith Infini-band
SNL Red Sky
Transient Kelvin-Helmholtz instability
(Re = 5 x 103 shear layer, constant CFL = 2.5)
-
8/2/2019 Chacon Plenary
32/38
I n c o m p r e s s i b l e M H D s o l v e r
L u i s C h a c n , c h a c o n l @ o r n l . g o v
-
8/2/2019 Chacon Plenary
33/38
Incompressible MHD: 2D Vector Potential
Formulation
Magnetohydrodynamics (MHD) equations couple fluid flow to
Maxwells equations
u
t +u u
2u
+p +
1
0BB
+
1
20 B
2I
= f
u = 0
Az
t+ u Az
02Az = E
0
z
where B = A, A = (0, 0, Az)
Discretized using a stabilized finite element formulation
-
8/2/2019 Chacon Plenary
34/38
F B
TZ
B C 0Y 0 D
= I
BF1 I
Y F1
Y F1B
TS1 I
F B
TZ
S
BF1Z
P
where
S = CBF1BT
P = D
Y F1(I+ BTS1BF1)Z
Cis zero for mixed interpolation FE and staggered FV methods, nonzero for stabilized FE Indefinite system hard to solve with incomplete factorizations without pivotingBlock factorization of 3x3 system leads to nested Schur complementsUse an operator splitting approximation to factor
Reduces to 2 2x2 systems for Navier-Stokes and magnetics-velocity blocks;C need not be non-zero or invertible (C-1doesnt need to exist!)
Block LU Factorization
-
8/2/2019 Chacon Plenary
35/38
Fully coupled AlgebraicAggC: Aggressive Coarsening Multigrid
DD: Additive Schwarz Domain
Decomposition
Block PreconditionersSplit: New Operator split preconditioner
SIMPLEC: Extreme diagonal approximations
Take home: AggC and Split preconditioner scale algorithmically
1. SIMPLE preconditioner performance suffers with increased CFL
2. Run times are for unoptimized code3. AggC not applicable to mixed discretizations, block factorization is
Take home: AggC and Split preconditioner scale algorithmically
1. SIMPLE preconditioner performance suffers with increased CFL
2. Run times are for unoptimized code3. AggC not applicable to mixed discretizations, block factorization is
Transient Hydro-Magnetic
Kelvin-Helmholtz Problem(Re = 700, S = 700)
-
8/2/2019 Chacon Plenary
36/38
Take home: Split preconditioner scales algorithmicallyTake home: Split preconditioner scales algorithmically
Fully coupled Algebraic
AggC: Aggressive Coarsening Multigrid
DD: Additive Schwarz Domain Decomposition
Block Preconditioners
Split: New Operator split preconditioner
SIMPLEC: Extreme diagonal approximations
Driven Magnetic Reconnection: Magnetic Island
Coalescence Half domain symmetry on [0,1]x[-1,1]
with S = 10e+4
I iti l W k S li P f f AMG V l L d hi Cl M hi
-
8/2/2019 Chacon Plenary
37/38
Initial Weak Scaling Performance of AMG V-cycle on Leadership Class Machines
Cray XE6 and BG/P Weak Scaling
(Transport-reaction: Drift-diffusion simulations)
Steady-state drift-diffusion BJT
TFQMR time per iteration
Cray XE6 2.4GHz 8-core Magny-Cours
(Paul Lin)
Sub-domain smoothers: Impact of data
locality of smoother?
BG/P: ILU(2); overlap = 1
BG/P: ILU(0); overlap = 0
[Better scaling and faster time to
solution than ILU(2),ov=1]
Cray XE6: ILU(2); overlap = 1> 2200x
-
8/2/2019 Chacon Plenary
38/38
S u m m a r y a n d C o n c l u s i o n s
S t i h y p e r b o l i c P D E s d e s c r i b e m a n y a p p l i c a t i o n s o f i n t e r e s t t o D O E .
I n a p p l i c a t i o n s w h e r e f a s t t i m e s c a l e s a r e p a r a s i t i c , a n i m p l i c i t t r e a t m e n t i s p o s s i b l e t o b r i d g e
t i m e - s c a l e d i s p a r i t y .
A f u l l y i m p l i c i t s o l u t i o n m a y o n l y r e a l i z e i t s e c i e n c y p o t e n t i a l i f a s u i t a b l e s c a l a b l e a l g o r i t h m i c
r o u t e i s a v a i l a b l e .
H e r e , w e h a v e i d e n t i e d s t i - w a v e b l o c k - p r e c o n d i t i o n i n g ( a k a p h y s i c s - b a s e d p r e c o n d i t i o n i n g ) i n
t h e c o n t e x t o f J F N K m e t h o d s a s a s u i t a b l e a l g o r i t h m i c p a t h w a y .
J A n i m p o r t a n t p r o p e r t y i s t h a t i t r e n d e r s t h e n u m e r i c a l s y s t e m s u i t a b l e f o r m u l t i l e v e l p r e c o n d i -
t i o n i n g .
W e h a v e d e m o n s t r a t e d t h e e e c t i v e n e s s o f t h e a p p r o a c h i n i n c o m p r e s s i b l e N a v i e r - S t o k e s , i n c o m -
p r e s s i b l e M H D , a n d c o m p r e s s i b l e r e s i s t i v e M H D a n d e x t e n d e d M H D .
J I n a l l t h e s e a p p l i c a t i o n s , t h e a p p r o a c h i s r o b u s t a n d s c a l a b l e , b o t h a l g o r i t h m i c a l l y a n d i n
p a r a l l e l .
L u i s C h a c n , c h a c o n l @ o r n l . g o v