Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source...
Transcript of Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source...
![Page 1: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/1.jpg)
Salvus: A flexible open-source package for full-waveform modelling and inversionMichael Afanasiev, Christian Boehm, Martin van Driel, Lion Krischer, Dave May, Max Rietmann, Korbinian Sager, and Andreas Fichtner
1.12.2014
![Page 2: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/2.jpg)
Full waveform inversion
@2t u(x)�r · �(x)� F = 0
�(x) = c(x) : ru(x)
![Page 3: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/3.jpg)
Full waveform inversion
@2t u(x)�r · �(x)� F = 0
�(x) = c(x) : ru(x)
A problem of optimal control over c, constrained by the wave equation.
![Page 4: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/4.jpg)
Motivation
�(x) = c(x) : ru(x)
Takes on a very different character…
![Page 5: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/5.jpg)
Motivation Acoustic and attenuative 2D/3D propagation through tissue
Elastic/Acoustic/Gravity coupling, 3D, attenuative
Elastic regional scale with topography
2D/3D, Elastic/Acoustic coupling, Topography, Attenuation
�(x) = c(x) : ru(x)
Takes on a very different character…
![Page 6: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/6.jpg)
Motivation Acoustic and attenuative 2D/3D propagation through tissue
Elastic/Acoustic/Gravity coupling, 3D, attenuative
Elastic regional scale with topography
2D/3D, Elastic/Acoustic coupling, Topography, Attenuation
![Page 7: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/7.jpg)
Motivation Acoustic and attenuative 2D/3D propagation through tissue
Elastic/Acoustic/Gravity coupling, 3D, attenuative
Elastic regional scale with topography
2D/3D, Elastic/Acoustic coupling, Topography, Attenuation
![Page 8: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/8.jpg)
▪ Flexible and modular design, with support for the concurrent simulation of multiple coupled PDEs — composability
▪ Scalable, dimension independent spatial and temporal discretization
▪ Simple integration with external optimization libraries
▪ Correct and consistent solutions
▪ Speed
Requirements
![Page 9: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/9.jpg)
Flexible and modular design, with support for the concurrent simulation of multiple coupled PDEs
![Page 10: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/10.jpg)
Modular design
Quad
FEM Basis 1
Physics 1
Additional physics…
Coupling physics…
Shape Mapping 1 Shape Mapping 2 Shape Mapping ….
FEM Basis 2 FEM Basis …
Hex
Physics 2 Physics …
![Page 11: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/11.jpg)
Modular design (Composability)
Shape::computeJacobian(); Shape::interpolateParameters(); …
Element::computeSymmetricGradient(); Element::applyGradTestAndIntegrate(); …
Physics::computeStrain(); Physics::computeStress(); …
AdditionalPhysics::modifyStress(); …
CouplingPhysics::computeCouplingTerm(); …
…need to change any one of these independently.
![Page 12: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/12.jpg)
Modular design: a solution with template mixins
template <typename Element> MatrixXd Elastic2D<Element>::computeStiffnessTerm(const Eigen::MatrixXd &u) {
// strain ux_x, ux_y, uy_x, uy_y. mStrain.leftCols<2>() = Element::computeGradient(u.col(0)); mStrain.rightCols<2>() = Element::computeGradient(u.col(1));
// compute stress from strain. mStress = computeStress(mStrain);
// temporary matrix to hold directional stresses. Matrix<double,Dynamic,2> temp_stress(Element::NumIntPnt(), 2);
// compute stiffness. temp_stress.col(0) = mStress.col(0); temp_stress.col(1) = mStress.col(2); mStiff.col(0) = Element::applyGradTestAndIntegrate(temp_stress); temp_stress.col(0) = mStress.col(2); temp_stress.col(1) = mStress.col(1); mStiff.col(1) = Element::applyGradTestAndIntegrate(temp_stress);
return mStiff;
}
![Page 13: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/13.jpg)
Modular design: a solution with template mixins
template <typename Element> MatrixXd Elastic2D<Element>::computeStiffnessTerm(const Eigen::MatrixXd &u) {
// strain ux_x, ux_y, uy_x, uy_y. mStrain.leftCols<2>() = Element::computeGradient(u.col(0)); mStrain.rightCols<2>() = Element::computeGradient(u.col(1));
// compute stress from strain. mStress = computeStress(mStrain);
// temporary matrix to hold directional stresses. Matrix<double,Dynamic,2> temp_stress(Element::NumIntPnt(), 2);
// compute stiffness. temp_stress.col(0) = mStress.col(0); temp_stress.col(1) = mStress.col(2); mStiff.col(0) = Element::applyGradTestAndIntegrate(temp_stress); temp_stress.col(0) = mStress.col(2); temp_stress.col(1) = mStress.col(1); mStiff.col(1) = Element::applyGradTestAndIntegrate(temp_stress);
return mStiff;
}
![Page 14: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/14.jpg)
Modular design: a solution with template mixinstemplate <typename Physics> MatrixXd Attenuation<Physics>::computeStiffnessTerm(const Eigen::MatrixXd &u) {
strain = Physics::computeGradient(u);
/* Modify strain by subtracting from memory variable equations… */
stress = Physics::computeStress(strain);
stiff = Physics::applyGradTestAndIntegrate(stress);
return stiff;
}
![Page 15: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/15.jpg)
Modular design: a solution with template mixins
Building up elements…
QuadP1 QuadP2 TensorBasis Acoustic Elastic2D Elastic3D Cpl2Acoustic Cpl2Elastic Attenuation …
![Page 16: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/16.jpg)
Modular design: a solution with template mixins
ElementAdapter< Attenuation< Elastic2D< Quad< QuadP2>>>> AtnElasticQuadP1;
Adding attenuation to surface elements….
Building up elements…
QuadP1 QuadP2 TensorBasis Acoustic Elastic2D Elastic3D Cpl2Acoustic Cpl2Elastic Attenuation …
![Page 17: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/17.jpg)
Modular design: a solution with template mixins
Adding coupling to CMB….
ElementAdapter< CoupleToAcoustic< Elastic2D< Quad< QuadP1>>>> AtnElasticQuadP1;
Building up elements…
QuadP1 QuadP2 TensorBasis Acoustic Elastic2D Elastic3D Cpl2Acoustic Cpl2Elastic Attenuation …
ElementAdapter< Attenuation< Elastic2D< Quad< QuadP2>>>> AtnElasticQuadP1;
Adding attenuation to surface elements….
![Page 18: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/18.jpg)
Modular design: a solution with template mixins// Get values from distributed array. mesh->pullElementalFields();
// Compute element integrals. for (auto &elm: elements) {
// Get relevant values. u = mesh->getFields(elm->closure());
// Compute stiffness. ku = elm->computeStiffnessTerm(u);
// Compute surface integral. s = elm->computeSurfaceIntegral(u);
// Compute source term. f = elm->computeSourceTerm(time);
// Compute acceleration. a = f - ku + s
// Assemble. mesh->pushFields(elm->closure());
}
// Push values to distributed array. mesh->pushElementalFields();
All in one element loop…
![Page 19: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/19.jpg)
Flexible spatial and temporal discretization
![Page 20: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/20.jpg)
Flexible spatial discretization: Builtin python-based meshing tools
Exploration scale (with topography)
Planets (Mars, Europa, …)
2D applications
![Page 21: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/21.jpg)
Flexible spatial discretization: PETSc DMPLEX
An example with coupling
Red: fluid Blue: solid
![Page 22: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/22.jpg)
An example with coupling
Flexible spatial discretization: PETSc DMPLEX
Red: fluid Blue: solid
![Page 23: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/23.jpg)
f1 (fluid)
f2 (solid) (fluid) (solid)
v1
v2 v4
v3 v5
v6
e1
e2
e3
e4
e5
e6
e7f1 f2
(fluid) (solid)
Flexible spatial discretization: PETSc DMPLEX
![Page 24: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/24.jpg)
v1
v2 v4
v3 v5
v6
e1
e2
e3
e4
e5
e6
e7f1 f2
v1 v2 v3 v4 v5 v6
e1 e2 e3 e4 e5 e6 e7
f1 f2
f1
f2
(fluid) (solid)
v1
v2 v4
v3 v5
v6
e1
e2
e3
e4
e5
e6
e7f1 f2
(fluid) (solid)
f1 (fluid)
f2 (solid)
Flexible spatial discretization: PETSc DMPLEX
![Page 25: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/25.jpg)
v1
v2 v4
v3 v5
v6
e1
e2
e3
e4
e5
e6
e7f1 f2
v1 v2 v3 v4 v5 v6
e1 e2 e3 e4 e5 e6 e7
f1 f2
f1
f2
(fluid) (solid)
v1
v2 v4
v3 v5
v6
e1
e2
e3
e4
e5
e6
e7f1 f2
(fluid) (solid)
f1 (fluid)
f2 (solid)
element_vector.push_back( ElasticCplAcousticQuadP1(options));
Flexible spatial discretization: PETSc DMPLEX
![Page 26: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/26.jpg)
Integration with external optimization routines (salvus_opt)
![Page 27: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/27.jpg)
27
Wavefield CompressionTime-domain full-waveform inversion requires massive storage capabilities to store the forward wavefield.
Goal: Find a good tradeoff between memory requirements and computational overhead by using customized compression methods.
![Page 28: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/28.jpg)
||
Line-Search with Inexact Gradients
28
contour lines of misfit functional
minimum
mk
inexact gradient
exact gradient (unknown)
![Page 29: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/29.jpg)
||
Line-Search with Inexact Gradients
29
contour lines of misfit functional
minimum
mk
inexact gradient
exact gradient (unknown)
![Page 30: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/30.jpg)
||
Line-Search with Inexact Gradients
30
contour lines of misfit functional
minimum
mk
inexact gradient
exact gradient (unknown)
![Page 31: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/31.jpg)
||
Line-Search with Inexact Gradients
31
contour lines of misfit functional
minimum
mk
inexact gradient
exact gradient (unknown)
Compression thresholds can be chosen adaptively based on ➢ the norm of the inexact gradient, ➢ the ratio of actual and predicted misfit reduction.
Convergence can be ensured if the relative error is smaller than 50%.
![Page 32: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/32.jpg)
Wavefield Compression• Substantial reduction of memory
requirements and I/O operations at negligible extra costs.
• 200 TB per event
• Using approximate gradients does not significantly slow down the rate of convergence to solve the inverse problem.
• The error in the inexact gradients can be controlled such that the lossy compression does not significantly affect the inverted results.
32
p = 0
p = 1
p = 2
p = 4
✦ Prediction-correction on hierarchical grids
✦ Requantization ✦ Re-interpolation with sliding-
window cubic splines
![Page 33: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/33.jpg)
First name Surname (edit via “Insert” > “Header & Footer”) 1.12.2014
f1
f2
f1 (fluid)
f2 (solid)
Integration with optimization libraries
-1 0 1
w/o compression
0
�6
�12
depth
[km
]
(i)
cf-544
(ii)
cf-1043
(iii)
cf-1708
(iv)
cf-2100
0
5
10 15 20
0
�6
�12
length [km]
depth
[km
]
(v)
cf-2506
0
5
10 15 20
length [km]
(vi)
cf-3277
0
5
10 15 20
length [km]
(vii)
cf-4714
0
5
10 15 20
length [km]
• Consistent discrete adjoint equation • Built-in interface to wavefield compression to reduce memory requirements • Extensions for Hessian-vector products (currently under development)
![Page 34: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/34.jpg)
Comprehensive testing suite
![Page 35: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/35.jpg)
Comprehensive Test Suite
▪ Every contribution is run against a comprehensive test suite, driven by CatchTM ▪ Analytical integrations on element volumes, faces, edges ▪ Analytical time-dependent solutions ▪ Proper interpolation of sources/receivers
▪ Eases collaboration from the level of student to domain specialist
![Page 36: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/36.jpg)
First name Surname (edit via “Insert” > “Header & Footer”) 1.12.2014
Comprehensive Test Suite
Testing integration with optimization libraries
![Page 37: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/37.jpg)
Speed
![Page 38: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/38.jpg)
Speed
ElementAdapter<Attenuation<Elastic2D<Quad<QuadP1>>>> AtnElasticQuadP1;
Templates resolved at compile time
SPECFEM3D CARTESIAN SALVUS
Stiffness Calculation
(microseconds)1.7 15
![Page 39: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/39.jpg)
Speed
ElementAdapter<Attenuation<Elastic2D<Quad<QuadP1>>>> AtnElasticQuadP1;
Templates resolved at compile time
SPECFEM3D CARTESIAN
SALVUS (precomputed
Jacobian)
Stiffness Calculation
(microseconds)1.7 5
![Page 40: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/40.jpg)
Speed
ElementAdapter<Attenuation<Elastic2D<Quad<QuadP1>>>> AtnElasticQuadP1;
Templates resolved at compile time
SPECFEM3D CARTESIAN
SALVUS (Deville
optimized strides)
Stiffness Calculation
(microseconds)1.7 2.2
![Page 41: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/41.jpg)
Speed
ElementAdapter<Attenuation<Elastic2D<Quad<QuadP1>>>> AtnElasticQuadP1;
Templates resolved at compile time
SPECFEM3D CARTESIAN
SALVUS (Deville
optimized strides)
Stiffness Calculation
(microseconds)1.7 2.2
Haven’t really tried yet…
![Page 42: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/42.jpg)
Applications
![Page 43: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/43.jpg)
Applications
![Page 44: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/44.jpg)
Applications
![Page 45: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/45.jpg)
Applications
![Page 46: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/46.jpg)
Outlook
![Page 47: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/47.jpg)
Accelerators?#include <stdio.h> #include <iostream> #include <cuda.h>
class QuadP1 { public:
__device__ __host__ void jacobian() { printf("%s\n", "Computing QuadP1 Jacobian."); };
};
class TriangleP1 { public: __device __host__ void jacobian() { printf("%s\n", "Computing TriangeP1 Jacobian."); }; };
template <typename ConcreteElement> class Quad: public ConcreteElement {
public: __device__ __host__ void gradient() { printf("%s\n", "Computing quad gradient."); ConcreteElement::jacobian(); };
};
template <typename ConcreteElement> class Tri: public ConcreteElement public: __device__ __host__ void gradient() { printf("%s\n", "Computing triangle gradient."); ConcreteElement::jacobian(); }; };
![Page 48: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/48.jpg)
Accelerators?#include <stdio.h> #include <iostream> #include <cuda.h>
class QuadP1 { public:
__device__ __host__ void jacobian() { printf("%s\n", "Computing QuadP1 Jacobian."); };
};
class TriangleP1 { public: __device __host__ void jacobian() { printf("%s\n", "Computing TriangeP1 Jacobian."); }; };
template <typename ConcreteElement> class Quad: public ConcreteElement {
public: __device__ __host__ void gradient() { printf("%s\n", "Computing quad gradient."); ConcreteElement::jacobian(); };
};
template <typename ConcreteElement> class Tri: public ConcreteElement public: __device__ __host__ void gradient() { printf("%s\n", "Computing triangle gradient."); ConcreteElement::jacobian(); }; };
// Get values from distributed array. mesh->pullElementalFields();
// Compute element integrals. for (auto &elm: elements) {
// Get relevant values. u = mesh->getFields(elm->closure());
// Compute stiffness. ku = elm->computeStiffnessTerm(u);
// Compute surface integral. s = elm->computeSurfaceIntegral(u);
// Compute source term. f = elm->computeSourceTerm(time);
// Compute acceleration. a = f - ku + s
// Assemble. mesh->pushFields(elm->closure());
}
// Push values to distributed array. mesh->pushElementalFields();
Graphics cards? Knights Landing?
![Page 49: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/49.jpg)
Outlook
▪ Frequency domain (billions of dofs…)
▪ Bridge the gap between research and production codes
▪ Nonconforming meshes
▪ Applications of FWI to new and interesting domains
▪ Additional physics (GPR, electromagnetic, …)
![Page 50: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/50.jpg)
Very warm thanks to:
▪ Kuangdai Leng (University of Oxford) ▪ Dan Chown (Achates power) ▪ Matt Knepley (Rice university) ▪ Tarje Nissen-Meyer (University of Oxford)
![Page 51: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/51.jpg)
Features and Methods• trust-region and line search methods • reusing pre-computed information whenever possible
-> interpolation instead of backtracking -> book-keeping of Earth models
• built-in regularization and smoothing • constraint handling using projection methods and
homogenization -> constraints on m are cheap, “feasible-point methods”
• handling of inexact derivatives and change of objective function
51
![Page 52: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/52.jpg)
Speed: Scaling
Maintain the scaling characteristics of PETSc
Parallelization via PETSc is essentially free
![Page 53: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/53.jpg)
53
Wavefield Compression
![Page 54: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/54.jpg)
||
Line-Search with Inexact Gradients
54
contour lines of misfit functional
minimum
BFGS update with exact information
mk
![Page 55: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/55.jpg)
||
Line-Search with Inexact Gradients
55
contour lines of misfit functional
minimum
BFGS update with inexact information
BFGS update with exact information
mk
![Page 56: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/56.jpg)
||
mk
Line-Search with Inexact Gradients
56
contour lines of misfit functional
minimum
BFGS update with inexact information
BFGS update with exact information
Classical workaround for Newton-type methods:
![Page 57: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/57.jpg)
Coupling to other physics…#include <Model/ExodusModel.h> #include <Physics/AcousticElastic2D.h> #include <Utilities/Options.h> #include <Mesh/Mesh.h>
using namespace Eigen;
template <typename BasePhysics> std::vector<std::string> AcousticToElastic2D<BasePhysics>::PullElementalFields() const { return {"ux", "uy", "v"}; }
template <typename BasePhysics> void AcousticToElastic2D<BasePhysics>::setBoundaryConditions(Mesh *mesh) { for (auto e: mesh->CouplingFields(BasePhysics::ElmNum())) { mEdg.push_back(std::get<0>(e)); mNbr.push_back(mesh->GetNeighbouringElement(mEdg.back(), BasePhysics::ElmNum())); mNbrCtr.push_back(mesh->getElementCoordinateClosure(mNbr.back()).colwise().mean()); } BasePhysics::setBoundaryConditions(mesh); }
template <typename BasePhysics> Eigen::MatrixXd AcousticToElastic2D<BasePhysics>::computeSurfaceIntegral(const Eigen::Ref<const Eigen::MatrixXd> &u) {
// col0->ux, col1->uy, col2->potential. Eigen::MatrixXd rval = Eigen::MatrixXd::Zero(BasePhysics::NumIntPnt(), 2); for (int i = 0; i < mEdg.size(); i++) { rval.col(0) += mRho_0[i] * BasePhysics::applyTestAndIntegrateEdge(u.col(2), mEdg[i]); rval.col(1) += mRho_0[i] * BasePhysics::applyTestAndIntegrateEdge(u.col(2), mEdg[i]); }
return -1 * rval;
}
![Page 58: Salvus: A flexible open-source package for full- waveform ... · Salvus: A flexible open-source package for full-waveform modelling and inversion Michael Afanasiev, Christian Boehm,](https://reader033.fdocuments.in/reader033/viewer/2022041601/5e314a9af1cba2330506de92/html5/thumbnails/58.jpg)
Modular design: a solution with template mixinstemplate <typename Element> MatrixXd Elastic2D<Element>::computeStress(const Eigen::Ref<const Eigen::MatrixXd> &strain) {
mc11 = Element::ParAtIntPts("C11"); mc12 = Element::ParAtIntPts("C12"); mc22 = Element::ParAtIntPts("C22"); mc33 = Element::ParAtIntPts("C33");
Matrix<double,Dynamic,3> stress(Element::NumIntPnt(), 3); VectorXd uxy_plus_uyx = strain.col(1) + strain.col(2);
stress.col(0) = mc11 * strain.col(0) + mc12 * strain.col(3) + mc13 * uxy_plus_uyx);
stress.col(1) = mc12 * strain.col(0) + mc22 * strain.col(3) + mc23 * uxy_plus_uyx);
stress.col(2) = mc13 * strain.col(0) + mc23 * strain.col(3) + mc33 * uxy_plus_uyx);
return stress;
}