Learning Similarity Metrics forNumerical Simulations

Georg Kohl Kiwon Um Nils Thuerey

Technical University of Munich

Overview – Motivation

Similarity assessment of scalar 2D simulation data from PDEs

Overview – Motivation

→

→

→

Similarity assessment of scalar 2D simulation data from PDEs

Overview – Motivation

Typical metrics like L¹ or L² operate locally → structures and patterns are ignored

Recognition of spatial contexts with CNNs

Mathematical metric properties should be considered

0.1 0.10.1

0.2

→

Overview – Method

Numerical Simulation

Initial condition varied in isolation

Ground truth distances

Simulation with PDE Solver

Data sequence

[ pi pi+1Δi pi+2Δi ⋯ pi+10Δi ]

[0.0 0.1 0.2 ⋯ 1.0 ]

⋯

Varia

tion

defin

es

Overview – Method

Numerical Simulation

Initial condition varied in isolation

Ground truth distances

Simulation with PDE Solver

Data sequence

[ pi pi+1Δi pi+2Δi ⋯ pi+10Δi ]

[0.0 0.1 0.2 ⋯ 1.0 ]

⋯

Varia

tion

defin

es

Learning

Learned distances

CNN

CNN

[0.01 0.12 0.24 ⋯ 0.99 ]

Pairing FeaturenormalizationShared weights

Featurecomparison

Overview – Method

Numerical Simulation

Initial condition varied in isolation

Ground truth distances

Simulation with PDE Solver

Data sequence

[ pi pi+1Δi pi+2Δi ⋯ pi+10Δi ]

[0.0 0.1 0.2 ⋯ 1.0 ]

⋯

Varia

tion

defin

es

Learning

Learned distances

CNN

CNN

[0.01 0.12 0.24 ⋯ 0.99 ]

Pairing FeaturenormalizationShared weights

Featurecomparison

Loss

Optimization

Overview – Method

Numerical Simulation

Initial condition varied in isolation

Ground truth distances

Simulation with PDE Solver

Data sequence

[ pi pi+1Δi pi+2Δi ⋯ pi+10Δi ]

[0.0 0.1 0.2 ⋯ 1.0 ]

⋯

Varia

tion

defin

es

Learning

Learned distances

CNN

CNN

[0.01 0.12 0.24 ⋯ 0.99 ]

Pairing FeaturenormalizationShared weights

Featurecomparison

Evaluation

Spearman’s ranking correlation[0.95 ]

Loss

Optimization

Overview – Results

Single example: distance comparisonPlume (a) Reference Plume (b)

LSiM (ours) L² Ground Truth0

1Distance to Reference

Plume (a)Plume (b)

Overview – Results

Single example: distance comparison Combined test data: correlation evaluationPlume (a) Reference Plume (b)

LSiM (ours) L² Ground Truth0

1Distance to Reference

Plume (a)Plume (b)

L² SSIM LPIPS LSiM (ours)0.40

0.50

0.60

0.70

0.80

Spea

rman

's ra

nk c

orre

latio

n

Related Work

“Shallow” vector space metrics

– Metrics induced by Lp-norms, peak signal-to-noise ratio (PSNR)

– Structural similarity index (SSIM) [Wang04]

Evaluation with user studies for PDE data

– Liquid simulations [Um17]

– Non-oscillatory discretization schemes [Um19]

Image-based deep metrics with CNNs

– E.g. learned perceptual image patch similarity (LPIPS) [Zhang18]

[Wang04] Wang, Bovik, Sheikh, and Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004[Um17] Um, Hu, and Thuerey. Perceptual Evaluation of Liquid Simulation Methods. ACM Transactions on Graphics, 2017[Um19] Um, Hu, Wang, and Thuerey. Spot the Difference: Accuracy of Numerical Simulations via the Human Visual System. CoRR, abs/1907.04179, 2019[Zhang18] Zhang, Isola, Efros, Shechtman, and Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR, 2018

Data Generation

Time depended, motion-based PDE with one varied initial condition

Initial conditions OutputFinite difference solver with time discretization

[ p0 p1 ⋯ pi ] t1 t2 t t o0

Data Generation

Time depended, motion-based PDE with one varied initial condition

[ p0 p1 ⋯ pi+n⋅Δi ] t1 t2 t t on

Incr

easi

ng p

aram

eter

cha

nge

Dec

reas

ing

outp

ut s

imila

rity

o1[ p0 p1 ⋯ pi+Δi ] t1 t2 t t

Initial conditions OutputFinite difference solver with time discretization

[ p0 p1 ⋯ pi ] t1 t2 t t o0

Data Generation

Time depended, motion-based PDE with one varied initial condition

Chaotic behavior in controlled environment → added noise to adjust data difficulty

noise1,1(s) noise1,2(s) noise1 , t(s)

[ p0 p1 ⋯ pi+n⋅Δi ] t1 t2 t t onnoisen ,1(s) noisen ,2(s) noisen ,t (s)In

crea

sing

par

amet

er c

hang

e

Dec

reas

ing

outp

ut s

imila

rity

noise2,1(s) noise2,2(s) noise2 , t(s)

o1[ p0 p1 ⋯ pi+Δi ] t1 t2 t t

Initial conditions OutputFinite difference solver with time discretization

[ p0 p1 ⋯ pi ] t1 t2 t t o0

Training Data

Eulerian smoke plume Liquid via FLIP [Zhu05]

Advection-diffusion transport Burger’s equation

[Zhu05] Zhu and Bridson. Animating sand as a fluid. ACM SIGGRAPH, 2005

Test Data

Liquid (background noise) Advection-diffusion transport (density)

Shape data Video data

TID2013 [Ponomarenko15]

[Ponomarenko15] Ponomarenko, Jin, Ieremeiev, et al. Image database TID2013: Peculiarities, results and perspectives. Signal Processing-Image Communication, 2015

Method – Base Network

Siamese architecture (shared weights) → Convolution + ReLU layers

Feature extraction from both inputs

Existing network possible → specialized model works better

BaseNetwork

BaseNetwork

Feature maps:sets of 3rd order tensors

Method – Feature Normalization

Adjust value range of feature vectors along channel dimension

– Unit length normalization → cosine distance (only angle comparison)

– Element-wise std. normal distribution → angle and length in global length distribution

BaseNetwork

BaseNetwork

Feature maps:sets of 3rd order tensors

Feature mapnormalization

Feature mapnormalization

Method – Latent Space Difference

Actual comparison of feature maps → element-wise distance

Must be a metric w.r.t. the latent space → ensure metric properties

or are useful options

BaseNetwork

BaseNetwork

Feature maps:sets of 3rd order tensors

Element-wiselatent spacedifference

Difference maps:set of 3rd order tensors

|~x−~y| (~x−~y )2

Feature mapnormalization

Feature mapnormalization

Method – Aggregations

Compression of difference maps to scalar distance prediction

Learned channel aggregation via weighted average

Simple aggregations with sum or average

BaseNetwork

BaseNetwork

Feature maps:sets of 3rd order tensors

Feature mapnormalization

Feature mapnormalization

Element-wiselatent spacedifference

Difference maps:set of 3rd order tensors

Channelaggregation:

weighted avg.

1 Learned weight per feature map

Average maps:set of 2nd order tensors

Spatialaggregation:

average

Layeraggregation:summation

Distanceoutput

dResult:scalar

Layer distances:set of scalars

d1 d2 d3

Loss Function

Ground truth distances c and predicted distances d

Mean squared error term → minimize distance deviation directly

Inverted correlation term → maximize linear distance relationship

L(c , d) = λ1(c−d)2 + λ2(1− (c−c̄)⋅(d−d̄)‖c−c̄‖2 ‖d−d̄‖2 )

Results

Evaluation with Spearman’s rank correlation

Ground truth against predicted distances

MetricValidation data sets Test data sets

Smo Liq Adv Bur TID LiqN AdvD Sha Vid All

L2 0.66 0.80 0.74 0.62 0.82 0.73 0.57 0.58 0.79 0.61

SSIM 0.69 0.74 0.77 0.71 0.77 0.26 0.69 0.46 0.75 0.53

LPIPS 0.63 0.68 0.68 0.72 0.86 0.50 0.62 0.84 0.83 0.66

LSiM 0.78 0.82 0.79 0.75 0.86 0.79 0.58 0.88 0.81 0.73

Real-world Evaluation

ScalarFlow [Eckert19] WeatherBench [Rasp20]

Johns Hopkins Turbulence Database (JHTDB) [Perlman07]

[Eckert19] Eckert, Um, and Thuerey. Scalarflow: A large-scale volumetric data set of real-world scalar transport flows [...]. ACM Transactions on Graphics, 2019[Rasp20] Rasp, Dueben, Scher, Weyn, Mouatadid, and Thuerey. Weatherbench: A benchmark dataset for data-driven weather forecasting. CoRR, abs/2002.00469, 2020[Perlman07] Perlman, Burns, Li, and Meneveau. Data exploration of turbulence simulations using a database cluster. ACM/IEEE Conference on Supercomputing, 2007

Real-world Evaluation

Retrieve order of spatial and temporal frame translations

Six interval spacings per data repository

180-240 sequences each

Mean and standard deviation over correlation of each spacing

L² SSIM LPIPS LSiM (ours)0.6

0.7

0.8

0.9

1.0

ScalarFlow JHTDB WeatherBench

Aver

age

Spea

rman

cor

rela

tion

Future Work

Accuracy assessment of new simulation methods

Parameter reconstructions of observed behavior

Guiding generative models of physical systems

Extensions to other data

– 3D flows and further PDEs

– Multi-channel turbulence data

Thank you for your attention!

Join the live-sessions for questions and discussion

Source code available at https://github.com/tum-pbs/LSIM

https://github.com/tum-pbs/LSIM

Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26