Metaprogramming Patterns for Physics Simulationand Verification
Roy H. Stogner
The University of Texas at Austin
April 16, 2012
Roy H. Stogner Metaprogramming April 16, 2012 1 / 52
Computational Modeling and Verification
PECOS ResearchVerification, Validation, Uncertainty Quantification• Atmospheric entry vehicle, two key quantities of interest:
I Rate of recession of TPS (throughout peak heating regime)I Local peak heat flux to after-body
• Validation and uncertainty quantification for these QoI
Roy H. Stogner Metaprogramming April 16, 2012 2 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes
• New subphysicsmodels
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
PECOS Development
Experimental Data
QUESO
FIN-SlibMeshBoost
Turbulence
SHOCKINGMUTATION
GSL
Chemistry/Shocktube
Plug FlowABLATION1D
GSL
Ablation
Calibration/Validation
FIN-S
Inve
rsio
n
Forw
ard
QUESO/Dakota
Multiphysics Analysis
MASA
QoI
GRVYExternal Libraries
LDVFlow
Measurements
NASA EAST Shocktube
Measurements
Molecular Beam (O2)Heated Flow Titration (N2)
New Codes• New subphysics
models
• New developers
• New QoIpostprocessing
• New numerics
• New bugs
Roy H. Stogner Metaprogramming April 16, 2012 3 / 52
Computational Modeling and Verification
Method of Manufactured Solutions
Verification Process• “Manufacture” a closed form exact solution u, e.g. parameterized trig
functions:(Roy, 2002)
u(x, y, z, t) = u0 + ux fs
(auxπxL
)+ uy fs
(auyπyL
)+
+ uz fs
(auzπzL
)+ ut fs
(autπt
L
)• Insert u into model equation system F , evaluate source terms
• Use simulation code to solve modified physics F (u) ≡ F (u)− F (u)• Verify that solution uh → u at expected rate
Roy H. Stogner Metaprogramming April 16, 2012 4 / 52
Computational Modeling and Verification
Maple MMS: 3D Navier-Stokes Energy Term
Qe =−apxπpx
L
γ
γ − 1sin
�apxπx
L
� �u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
+apyπpy
L
γ
γ − 1cos
�apyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��+
−apzπpz
L
γ
γ − 1sin
�apzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��+
+aρxπρx2L
cos�aρxπx
L
� �u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
−aρyπρy2L
sin�aρyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
+aρzπρz2L
cos�aρzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
����u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2+
+�w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2�
+
+auxπux
2Lcos
�auxπx
L
����w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
��2+
�v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
��2+
+3�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��2��
ρ0 + ρx sin�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��+
+�p0 + px cos
�apxπx
L
�+ py sin
�apyπy
L
�+ pz cos
�apzπz
L
�� 2γ
(γ − 1)
�
+
−auyπuy
Lsin
�auyπy
L
� �v0 + vx cos
�avxπx
L
�+ vy sin
�avyπy
L
�+ vz sin
�avzπz
L
�� �ρ0 + ρx sin
�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��·
·�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
−auzπuz
Lsin
�auzπz
L
� �w0 + wx sin
�awxπx
L
�+ wy sin
�awyπy
L
�+ wz cos
�awzπz
L
�� �ρ0 + ρx sin
�aρxπx
L
�+ ρy cos
�aρyπy
L
�+ ρz sin
�aρzπz
L
��·
·�u0 + ux sin
�auxπx
L
�+ uy cos
�auyπy
L
�+ uz cos
�auzπz
L
��+
Roy H. Stogner Metaprogramming April 16, 2012 5 / 52
Computational Modeling and Verification
And more 3D Navier-Stokes Energy Term
Roy H. Stogner Metaprogramming April 16, 2012 6 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification
Combinatorial Explosion
Explosion in source term size• Exponential complexity increase with more sophisticated models
I Basic Sutherland viscosity starts at a million characters
• Large CAS RAM requirements (128 GB insufficient for Sutherland)
• segfaults, incorrect output
Hierarchic MMS?• Manual simplification of sub-terms, one-by-one
• Recombination into final forcing function
• Still labor-intensive
Automatic Differentiation?• Adding complex dependencies to MASA?
Roy H. Stogner Metaprogramming April 16, 2012 7 / 52
Computational Modeling and Verification Generic Programming
Generic Programming
• Or more informally, “Duck Typing”: templating around unspecifieddata types that expected to be overloaded to act like mathematicalobjects, creating new code and new data types at compile time.
• We do this all the time for float/double/long double.
• So we can duct tape our templated typename to any class thatbehaves like we expect math to behave
Roy H. Stogner Metaprogramming April 16, 2012 8 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Generic Programming
Duck Typing
If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.
Unless it’s a dragon doing a duckimpression...
We ran into a couple of those.
Roy H. Stogner Metaprogramming April 16, 2012 9 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Multi-precision
Example
template <typename T, typename S>
struct ShadowNumber {.........T _val;
S _shadow;
};
• Simultaneous calculation with multiple floating point representations,to estimate rounding error:
ExampleShadowNumber<float, double> shadowed_float;
ShadowNumber<double, long double> shadowed_double;
Roy H. Stogner Metaprogramming April 16, 2012 10 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
template <unsigned int size, typename T>
struct NumberArray {.........
template <typename T2>
NumberArray<size,T>& operator+= (const NumberArray<size,T2>& a) {
for (unsigned int i = 0; i != size; ++i)
_data[i] += a[i]; return *this;
}.........T _data[size];
};
• Compile-time fixed array sizes - no heap memory
• Fixed length loops can unroll with no branching
• Mixed data subtypes
Roy H. Stogner Metaprogramming April 16, 2012 11 / 52
Computational Modeling and Verification Example Data Types
Overloaded Functions, Operators
Example
#define NumberArray_std_unary(funcname) \
template <unsigned int size, typename T> \
inline \
NumberArray<size, T> \
funcname (const NumberArray<size, T>& a) \
{ \
NumberArray<size, T> returnval; \
\
for (unsigned int i = 0; i != size; ++i) \
returnval[i] = funcname(a[i]); \
\
return returnval; \
}
NumberArray_std_unary(exp)
NumberArray_std_unary(sin)
NumberArray_std_unary(cos)
• Templates shouldreplace writing NTNf
repetitive codes usingunsafe macros
• Templated function andstruct names can’t betemplate arguments.
• We still need macros toavoid Nf repetitivetemplate functions.
• This gives usmultidimensional nth
derivatives in 300 linesof C++
Roy H. Stogner Metaprogramming April 16, 2012 12 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f);
// Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v);
// Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f);
// Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Example Data Types
Operator-Overloaded Vectors
Example
NumericArray<3,double> v;
v[1] = 2;
NumericArray<3,float> f;
f[2] = 1;
v += sin(f); // Great!
f += sin(v); // Tolerable!
v = v + sin(f); // Undefined?
What is the return type supposed to be for
template <unsigned int size, typename T1, typename T2>
operator+(const NumberArray<size,T1>& a,
const NumberArray<size,T2>& b)
std:: problem too - complex<float> + double won’t compile
Roy H. Stogner Metaprogramming April 16, 2012 13 / 52
Computational Modeling and Verification Computing new Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
And then we can specialize
Exampletemplate <>
struct CompareTypes<float,double> {
typedef double
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 14 / 52
Computational Modeling and Verification Computing new Data Types
Return Type Selection, Template Specialization
Let’s give it a template:
template <typename S, typename T>
struct CompareTypes {
// typedef something supertype;
};
Or, for templated classes, partially specialize
Exampletemplate <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,typename CompareTypes<T,T2>::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 14 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,double
>::supertype) ==
typeid(NumberArray<3,double>))
how_much_we_rock = 10;
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A NumberArray of something, combined with anything else we haven’tseen before
Example
template <unsigned int size, typename T, typename T2>
struct CompareTypes<NumberArray<size,T>,T2> {
typedef NumberArray<size,
typename CompareTypes<T,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,NumberArray<float> >,double
>::supertype) ==
typeid(NumberArray<3,NumberArray<double> >))
how_much_we_rock = 11; // That extra push over the cliff
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with anything else we haven’tseen before
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
ShadowNumber<float,double>,double
>::supertype) ==
typeid(ShadowNumber<double,double>))
how_much_we_rock = 7; // C is still a passing grade
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with a NumberArray
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(NumberArray<3,ShadowNumber<float,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Here Be Dragons
A ShadowNumber of something, combined with a NumberArray
Example
template <typename T, typename S, typename T2>
struct CompareTypes<ShadowNumber<T,S>,T2> {
typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,
typename CompareTypes<S,T2>::supertype>
supertype;
};
if(typeid(CompareTypes<
NumberArray<3,float>,ShadowNumber<float,double>
>::supertype) ==
typeid(ShadowNumber<NumberArray<3,float>,NumberArray<3,double> >))
how_much_we_rock = -1; // This didn’t even compile
Roy H. Stogner Metaprogramming April 16, 2012 15 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Computing new Data Types
Return Types: Killing Dragons• Mathematically, these are isomorphic:
I (V ⊗W )n
I V n ⊗Wn
• Your compiler doesn’t understandisomorphisms:
I {float, double}[size]I {float[size]}, {double[size]}
• Need partial template specialization,SFINAE to disambiguate
Example
template <unsigned int size, typename T, typename T2, typename S2>
struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {
typedef NumberArray<size,
typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>
supertype;
};
Roy H. Stogner Metaprogramming April 16, 2012 16 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ − 1
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ − 1 ...no, wait, that’s C
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
“Dual Numbers” - [Clifford 1873],[Study 1903]
• Add a new element ε to R, close the field:{m∑i=0
aiεi : ai ∈ R,m <∞
}
• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}
• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)
• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),
f(a+ bε) = f(a) + bf ′(a)ε
• This time the isomorphism practically codes itself!
Roy H. Stogner Metaprogramming April 16, 2012 17 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
Operator-Overloaded Forward Differentiation
Example
typedef double RawType
// typedef ShadowNumber<double, long double> RawType;
typedef DualNumber<RawType,RawType> FirstDerivType;
const FirstDerivType x(pi/6,1); // Initializing independent var
const FirstDerivType sinx = sin(x); // Caching just like normal
const FirstDerivType y = sinx*sinx; // Smart pow would work too
const double raw_y = raw_value(y); // No implicit down-conversions
double deriv = raw_value(y.derivatives());
assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code
Roy H. Stogner Metaprogramming April 16, 2012 18 / 52
Computational Modeling and Verification Automatic Differentiation
“Hyper-dual Numbers” - [Fike 2009]
• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0
H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}
• (a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2
• Extending gives: for any function f(x),f(a+bε1+bε2+cε1ε2) = f(a)+bf ′(a)(ε1+ε2)+(b2f ′′(a)+cf ′(a))ε1ε2
• adding εiεj = 0 to quotient would give vector gradients instead...
Roy H. Stogner Metaprogramming April 16, 2012 19 / 52
Computational Modeling and Verification Automatic Differentiation
Hyper-dual Numbers via Recursive Mathematics
• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε
• Applied directly to R we get first derivatives• Applied recursively:
I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f
′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f
′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2
• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!
Roy H. Stogner Metaprogramming April 16, 2012 20 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
Hessian Differentiation via Recursive Templates
Example
typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;
typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;
NumberArray<3, FAD2> xyz;
xyz[0] =
FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());
xyz[1] =
FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());
xyz[2] =
FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());
const FAD2 u = some_function(xyz);
RawType laplacian_u = 0;
for (unsigned int i=0; i != 3; ++i)
laplacian_u += u.derivatives()[i].derivatives()[i];
Roy H. Stogner Metaprogramming April 16, 2012 21 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Manufactured Solution
// Arbitrary manufactured solution
U.template get<0>() = u_0 + u_x * sin(a_ux * PI * x / L) +
u_y * cos(a_uy * PI * y / L);
// Why not U[0] and U[1]? Explanation to come...
U.template get<1>() = v_0 + v_x * cos(a_vx * PI * x / L) +
v_y * sin(a_vy * PI * y / L);
ADScalar RHO = rho_0 + rho_x * sin(a_rhox * PI * x / L) +
rho_y * cos(a_rhoy * PI * y / L);
ADScalar P = p_0 + p_x * cos(a_px * PI * x / L) +
p_y * sin(a_py * PI * y / L);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Euler
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U)) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U));
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
MASA PDE Examples
Navier-Stokes
// Gas state
ADScalar T = P / RHO / R;
ADScalar E = 1. / (Gamma-1.) * P / RHO;
ADScalar ET = E + .5 * U.dot(U);
// Constitutive laws
Tensor GradU = gradient(U);
Tensor Tau = mu * (GradU + transpose(GradU) -
2./3. * divergence(U) * RawArray::identity());
FullArray q = -k * T.derivatives();
// Conservation equation residuals
Scalar Q_rho = raw_value(divergence(RHO*U));
RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U) - Tau) +
P.derivatives());
Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U + q - Tau.dot(U)));
Roy H. Stogner Metaprogramming April 16, 2012 22 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Automatic upconversion is dangerous!
Example
double problem(FirstDerivType x, SecondDerivType y) {
// This is desirable:
x *= 2.0;
// But now the compiler doesn’t see a problem with this!
return (x+y).derivatives().derivatives();
}
• Code with 2nd and higher derivatives still requires care
• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”
Roy H. Stogner Metaprogramming April 16, 2012 23 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!
I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)
I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:
I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data types
I Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizations
I Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Computational Modeling and Verification Automatic Differentiation
Lessons Learned
• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)
n∑i=0
(m+ i− 1)!
(m− 1)!i!
• For efficient higher derivatives: partial specialization of recursiveDualNumber
• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations
Roy H. Stogner Metaprogramming April 16, 2012 24 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type ContainerExample
template <typename HeadType,
typename TailContainer=NullContainer,
typename Comparison=ValueLessThan>
struct Container
{
typedef HeadType head_type; // car and cdr would scare
typedef TailContainer tail_set; // off the Fortran folks
typedef Comparison comparison;
// These may be empty types or may have data
HeadType head;
TailContainer tail;.........};
struct NullContainer
{.........};
Roy H. Stogner Metaprogramming April 16, 2012 25 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Data Structures with Templates: Type Container
Example
Container<IntType<1> > no_data_one_static_int;
Container<IntType<1>,
Container<IntType<4> > no_data_two_static_ints;
Container<IntType<1,float>,
Container<IntType<4,float> > float_data_two_static_ints;
Container<IntType<1,double>,
Container<IntType<4>,NumberArray<3,double> > heterogenous_data;
Roy H. Stogner Metaprogramming April 16, 2012 26 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting into a NullContainer is easy:
• If we’re inserting null, we’re still left with a NullContainer
• If not, we have a one-element container with the inserted element.
Examplestruct NullContainer
{.........template <typename ValueType, typename NewComparison>
struct Insert
{
typedef
typename IfElse<
(TypesEqual<ValueType,NullContainer>::value),
NullContainer,
Container<ValueType, NullContainer, NewComparison>
>::type type;
};.........};
C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?
Roy H. Stogner Metaprogramming April 16, 2012 27 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations: Insert
Inserting in nonempty sorted sets is trickier:
• If we’re inserting null, we return our same set
• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail
• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail
• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”
• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior
Roy H. Stogner Metaprogramming April 16, 2012 28 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Set Operations
• Union, Intersection, Difference with an empty container is trivial.
• With non-empty containers, Insert and Contains methods makerecursive definitions easy:
Example
template <typename Set2>
struct Union {
typedef typename
tail_set::Sorted::template Union<
typename Set2::Sorted::template Insert<
head_type
>::type
>::type type;
};
Roy H. Stogner Metaprogramming April 16, 2012 29 / 52
Numeric Functional Metaprogramming Functional Data Structures
Runtime Indexed ForEach
• Applying a standard functor to every set element
• Recursive compile-time metaprogram unrolls into a simple linearrun-time program
• f receives a non-compile-time-const v
Example
struct RuntimeForEach
{
template <typename Functor>
void operator()(const Functor &f) {
// f might want a reference, so we pass in a non-static copy
const typename head_type::value_type v = head_type::value;
f(v);
typename tail_set::RuntimeForEach()(f);
}
};
Roy H. Stogner Metaprogramming April 16, 2012 30 / 52
Numeric Functional Metaprogramming Functional Data Structures
Compile-time Indexed ForEach
• Applying a “template functor” to every set element
• Recursive compile-time metaprogram unrolls into a simple linearrun-time program
• f receives a compile-time-const HeadType
Example
struct ForEach
{
template <typename Functor>
void operator()(const Functor &f) {
f.operator()<HeadType>(v);
typename tail_set::ForEach()(f);
}
};
Roy H. Stogner Metaprogramming April 16, 2012 31 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• e.g. the vector [0 0 1 0 2 0] will have {3, 5} or a superset as index set
• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........
T& raw_at(unsigned int i)
{ return _data[i]; }
.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices
• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........
T& operator[](index_value_type i)
{ return _data[IndexSet::runtime_index_of(i)]; }
.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices• Const-index O (1) access via template arguments
Example
template <typename T, typename IndexSet>
class SparseNumberArray {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;.........template <unsigned int i>
typename entry_type<i>::type& get() {
return _data[IndexSet::template IndexOf<i>::index];
}.........private:
T _data[size];
};
Roy H. Stogner Metaprogramming April 16, 2012 32 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:
• Multiplication by a constant: for loop over all data, multiply datum
Other operations require simple set operations:
• Element-wise addition takes a Union of index sets
• Element-wise multiplication takes an Intersection of index sets
• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.
Others require repeated set operations:
• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:
• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.
• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.
Roy H. Stogner Metaprogramming April 16, 2012 33 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·]
= [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Vectors
Consider recursion with dual numbers, for automatic differentiation
• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y
• DualNumber x and y are given unit vector gradients
• Their sums, products, etc. then all have optimally sparse gradients.
We now have excellent sparse vectors-of-T!
[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]
[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]
• From 6 FLOPs down to 0
• But not good enough yet
Roy H. Stogner Metaprogramming April 16, 2012 34 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians
0 0 0 00 0 2 00 0 0 00 0 0 0
+
0 0 0 00 0 0 00 0 0 01 0 0 0
=
0 0 0 00 0 2 00 0 0 01 0 0 0
· · · ·· · 2 ·· · · ·· · · ·
+
· · · ·· · · ·· · · ·1 · · ·
=
· · · ·0 · 2 ·· · · ·1 · 0 ·
• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.
I Triangular matrices become full?I Diagonal matrices become full??
Roy H. Stogner Metaprogramming April 16, 2012 35 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices
• Index-dependent data type makes variable-index component accessimpossible!
Example
template <typename IndexSet>
class SparseNumberStruct {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;
template <unsigned int i>
typename entry_type<i>::type& get() {
return _data.template data<UnsignedIntType<i> >(); }.........private:
IndexSet _data;
};
Roy H. Stogner Metaprogramming April 16, 2012 36 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices• Index-dependent data type makes variable-index component access
impossible!
Example
template <typename IndexSet>
class SparseNumberStruct {
public:
typedef IndexSet index_set;
static const unsigned int size = IndexSet::size;
template <unsigned int i>
typename entry_type<i>::type& get() {
return _data.template data<UnsignedIntType<i> >(); }.........private:
IndexSet _data;
};
Roy H. Stogner Metaprogramming April 16, 2012 36 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Operator-Overloaded Sparse Structure Operations
No operations are easily defined.
• C for loops don’t work over heterogenous structures.
• Templated functors must be defined for every operation.
Multiplication by a constant, using a functor and subfunctor:
Exampletemplate <typename T2, typename IndexSet>
inline
typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype
operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)
{
typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;
typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype
returnval;
typename IndexSet::ForEach()
(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>
(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));
return returnval;
}
Roy H. Stogner Metaprogramming April 16, 2012 37 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation
• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types
• CompareTypes or similar classes (or C++11 decltype) providesproper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides
proper return types
Example
struct MultipliesSubfunctor {
template <typename T1, typename T2>
struct Return {
typedef typename
SymmetricMultipliesType<T1,T2>::supertype type;
};
template <typename T1, typename T2>
typename Return<T1,T2>::type
operator()(const T1& x, const T2& y) const { return x * y; }
};
Roy H. Stogner Metaprogramming April 16, 2012 38 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]
{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]
Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Sparse Data Structures
Sparse Matrix Operations
Example
A ≡[a bT
c D
]{L,U} ≡ {LUL(A), LUU (A)}
L =
[a 0T
c LUL(D− ca−1bT)
]U =
[1 a−1bT
0 LUU (D− ca−1bT)
]Then apply to sparse arrays of arrays:
• One compile-time pass to allocate output structure
• One run-time pass to do calculations
• Asymptotic complexity drops beautifully
• But now we have asymptotic code generation to worry about!
Roy H. Stogner Metaprogramming April 16, 2012 39 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations• A templated functor is almost a full physics equation!
I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible
I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?
Example
#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \
struct physicsname { \
static const char* name() { return #physicsname; } \
typedef UIntSetConstructor<input1enum>::type inputset; \
typedef UIntSetConstructor<outputenum>::type outputset; \
\
template <typename inputtype> \
auto operator() (const inputtype& input1enum##_NAME) \
-> decltype(code) { \
return code; \
} \
}
Roy H. Stogner Metaprogramming April 16, 2012 40 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations• A templated functor is almost a full physics equation!
I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible
I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?
Example
#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \
struct physicsname { \
static const char* name() { return #physicsname; } \
typedef UIntSetConstructor<input1enum>::type inputset; \
typedef UIntSetConstructor<outputenum>::type outputset; \
\
template <typename inputtype> \
auto operator() (const inputtype& input1enum##_NAME) \
-> decltype(code) { \
return code; \
} \
}
Roy H. Stogner Metaprogramming April 16, 2012 40 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equations
Example
DeclareUnaryPhysics(DensityFromSpeciesDensities,
DENSITIES_VAR, DENSITY_VAR,
rhoi.sum());
DeclareBinaryPhysics(MomentumFromVelocity,
DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,
U * rho);
DeclareBinaryPhysics(VelocityFromMomentum,
DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,
rhoU / rho);
DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,
SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,
SPECIFIC_ENERGY_VAR,
e + UdotU / 2);
• Functors get reused in every Physics set, for all input argument types
• Minimizing how much code must be entrusted to physics people
Roy H. Stogner Metaprogramming April 16, 2012 41 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets
• Equations are now represented by data types
• Data types can be stuck into metaprogramming containers!
Example
typedef VectorConstructor<
DensityFromSpeciesDensities,
VelocityFromMomentum,
MomentumFromVelocity,
SpeedSquaredFromVelocity,
SpecificEnergyFromConserved,
SpecificInternalEnergyFromSpecificEnergy,
SpecificEnergyFromSpecificInternalEnergy,
LinearTranslationalRotationalEnergyFromTemperature,
SpecificInternalEnergyFromOnlyTransationalRotationalEnergy
>::type AllPhysics;
Roy H. Stogner Metaprogramming April 16, 2012 42 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Variable Sets
• Variables are now represented by enumerated values
• Enums can be stuck into metaprogramming containers!
Example
typedef UIntSetConstructor<
DENSITIES_VAR,
VELOCITY_VAR,
TEMPERATURE_VAR
>::type PrimitiveVars;
typedef UIntSetConstructor<
DENSITIES_VAR,
MOMENTUM_VAR,
ENERGY_VAR
>::type ConservedVars;
Roy H. Stogner Metaprogramming April 16, 2012 43 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
• Consider each variable as a rednode
• Consider each equation as ablack node
I Each output is a directededge from equation to variable
I Each input is a directed edgefrom variable to equation
Then our physics now looks like agraph!
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
Roy H. Stogner Metaprogramming April 16, 2012 44 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
We can use metaprogramming toanswer questions about the graph.
• Given a set of input variables:I Which non-input variables in
the graph are nowdetermined?
I What are the dependencies ofeach variable?
• And a set of output variables:I Which equations must be
solved to evaluate them?I In which order?
Roy H. Stogner Metaprogramming April 16, 2012 45 / 52
Numeric Functional Metaprogramming Physics as a Graph Problem
Equation Sets as Bicolored Graphs
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
We can use metaprogramming toanswer questions about the graph.
• Given a set of input variables:I Which non-input variables in
the graph are nowdetermined?
I What are the dependencies ofeach variable?
• And a set of output variables:I Which equations must be
solved to evaluate them?I In which order?
Roy H. Stogner Metaprogramming April 16, 2012 45 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
SolveList
T
ei
ρi
(ρe)i
ρe
ρ
~u
~u · ~u
ρ~u · ~u
ρE
45
6
12
3
7
Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.
1 Find all the dependencies ofyour output variables
I Unreachable output variablesbecome a compile-time error
I Unnecessary non-inputnon-output variables are leftout
2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list
Roy H. Stogner Metaprogramming April 16, 2012 46 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables
3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types
2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equation
I Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Evaluations on the Graph
State Updating
With user-specified types on all variables, SolveList evaluation is easy:
1 Create a Container type holding the specified variable types
2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList
I Pass input variables from state, assign output variables to state foreach equation
With user-specified types on input variables only, we first need ametaprogramming pass:
1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList
I Pass input variable types to each equationI Insert output variable types into the State
3 Do a ForEach loop at runtime as before
Roy H. Stogner Metaprogramming April 16, 2012 47 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
Differentiation looks already solved!
1 Collect your Physics
2 Choose your input variables
3 Evaluate the SolveList
4 Use DualNumber input variable types
5 Take derivatives() of output variables
6 Cheer
Roy H. Stogner Metaprogramming April 16, 2012 48 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Equation Derivatives
But how well does that solve differentiation?• Forward differentiation issues:
1 Needlessly expensive for many-input-variable, few-output variablegraphs
• or subgraphs!
• Adjoint differentiation issues:1 Needlessly expensive for many-output-variable, few-input variable
graphs• or subgraphs!
2 Typically requires additional temporary data• particularly with operator overloading...
• Optimal Jacobian Accumulation issues:1 NP-complete!
• don’t want to stress-test compilers...
• Heuristic: “Eh, Good Enough” Jacobian Accumulation
Roy H. Stogner Metaprogramming April 16, 2012 49 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives
• Ni inputs, No outputs gives Ni ·No partial derivatives
• Small equations are efficiently evaluated with sparse forward AD
• Equation nodes are out of the picture
• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj
An intermediate variable M can be “condensed away”:
• For all outgoing edges to output variable O:
• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)
I (starting from D(I,O) ≡ 0 when creating a new edge)
Keep condensing until all remaining edges are directly from user inputs touser outputs
Roy H. Stogner Metaprogramming April 16, 2012 50 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• In what order do we condense these equations?I N -factorial possibilitiesI Some possibilities are cheaper than othersI NP -complete optimization problem
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Forward differentiation:I Evaluates equations inputs-to-outputsI Effectively does Ma,Mb,Mc,Md,Me,Mf
I 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Reverse differentiation:I Evaluates equations outputs-to-inputsI Effectively does Mf ,Me,Md,Mc,Mb,Ma
I Still 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md
I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Chain Rule Application: Condensing the Graph
Ia
Ib
Ic
Ma
Mb
Mc Md
Me
Mf
Oa
Ob
Oc
A
B
C D
E
F
G
H
I
• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md
I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!
Eh, good enough
Roy H. Stogner Metaprogramming April 16, 2012 51 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Any Questions?
Lots of My Questions:• How much code for non-MASA
apps can be replaced bystandard libraries?
I Boost MTLI SacadoI Phalanx
• Can physics functors bedecomposed automatically?
I Precompilation script?
• Can we “roll up” sparsestructure algorithms?
Roy H. Stogner Metaprogramming April 16, 2012 52 / 52
Numeric Functional Metaprogramming Differentiation on the Graph
Any Questions?
Lots of My Questions:• How much code for non-MASA
apps can be replaced bystandard libraries?
I Boost MTLI SacadoI Phalanx
• Can physics functors bedecomposed automatically?
I Precompilation script?
• Can we “roll up” sparsestructure algorithms?
Roy H. Stogner Metaprogramming April 16, 2012 52 / 52
Top Related