Human Visual System Models in Computer Graphics

Tunç O. Aydın MPI Informatik Computer Graphics Department

HDR and Visual Perception Group

Outline Reality vs. Perception

– Why even bother modeling visual perception

The Human Visual System (HVS) – How the “wetware” affects our perception

HVS models in Computer Graphics – Visual Significance of contrast

– Contrast Detection

Our contributions – Key challenges

High Low

Difference Image (Color coded)

Invisible Bits & Bytes

Reference (bmp, 616K) Compressed (jpg, 48K)

Variations of Perception

No one-to-one correspondence between visual perception and reality !

“Perceived Visual Data” instead of luminance or arbitrary pixel values

The Human Visual System (HVS) Experimental

Methods of Vision Science – Micro-electrode

– Radioactive Marker

– Vivisection

– Psychophysical Experimentation

HVS effects (1): Glare

Video Courtesy of Tobias Ritschel

Disability Glare (blooming)

Disability Glare Model of Light

Scattering

– Point Spread Function in spatial domain

– Optical Transfer Function in Fourier Domain [Deeley et al. 1991]

Spatial Frequency [cy/deg]

Mo

du

lati

on

(2): Light Adaptation

Time Adaptation Level:

10-4 cd/m2 Adaptation Level:

17 cd/m2

Perceptually Uniform Space Transfer function:

Maps Luminance to Just Noticeable Differences (JNDs) in Luminance. [Mantiuk et al. 2004, Aydın et al. 2008]

Res

po

nse

[JN

D]

Luminance [cd/m2]

(3): Contrast Sensitivity

CSF(spatial frequency, adaptation level, temporal freq., viewing dist, …)

Co

ntr

ast

Spatial Frequency

November 6, 2011

Steady-state CSFS: Returns the Sensitivity (1/Threshold contrast), given the adaptation luminance and spatial frequency [Daly 1993].

Contrast Sensitivity Function (CSF)

(4): Visual Channels Cortex Transform

(5): Visual Masking

Loss of sensitivity to a signal in the presence of a “similar frequency” signal “nearby”.

Modeling Visual Masking Example: JPEG’s

pointwise extended masking:

C’: Normalized Contrast

HVS Models in Graphics/Vision

HDR LDR

Tone Mapping Compression

Rate the

Quality

Quality Assessment Panorama Stitching

Visual Significance Pipeline

kN

n

k

ref

k

tst RRR

/1

1

ˆ

Contrast Detection Pipeline

Log

Thre

sho

ld E

leva

tio

n

Log Contrast Contrast Difference

Pro

bab

ility

of

Det

ecti

on

N

n

nPP1

11ˆ

CONTRIBUTIONS: VISUAL SIGNIFICANCE

Visually Significant Edges Key Idea: Use the

magnitude of the HVS model’s response as the measure of edge strength, instead of gradient magnitude.

Result (1): Significant improvement in application results, especially for HDR images

Result (2): Only minor improvements observed in LDR retargeting and panorama stitching.

[Aydın, Čadík, Myszkowski, Seidel. 2010 ACM TAP]

Calibration Procedure

CSF from the Visible Differences Predictor [Daly’93]

JPEG’s pointwise extended masking Calibration: CSF derived for

sinusoidal stimuli, not for edges. Perceptual experiment for

measuring edge thresholds

Calibration Function

Calibration function: R: Visual Significance for sinusoidal stimulus, R’: Visual Significance for edges.

Subjective Measurements

Polynomial Fit

Metric Predictions

Polynomial Fit

Ideal Metric Response

Calibrated Metric

Response

Image Retargeting

Visual Significance Maps

High Low

Display Visibility under Dynamically Changing Illumination

Key Idea: Extending steady-state HVS models with temporal adaptation model

Result: A visibility class estimator integrated into a software that simulates illumination inside an automobile.

[Aydın, Myszkowski, Seidel. 2009 EuroGraphics]

L

Background Luminance

cvi for Steady-State Adaptation Contrast vs. Intensity

(cvi): function assumes perfect adaptation

Contrast vs. Intensity and adaptation (cvia) accounts for maladaptation

L + ∆L

Threshold Luminance

CLcvi :

CLLcvia a ,:

cvia for Maladaptation

cvia

cvi

Adaptation over time

Low

High

Visu

al Significan

ce t = 0 t = 0.2s t = 0.4s t = 0.8s

Rendering Adaptation

[Pająk, Čadík, Aydın, Myszkowski, Seidel. 2010 Electronic Imaging]

Dark Adaptation Bright Adaptation

CONTRIBUTIONS: CONTRAST DETECTION

Quality Assessment (IQA, VQA)

Rate the

Quality

+ Reliable - High cost

Key Idea: Find a transformation from Luminance to pixel values, such that: – An increment of 1 pixel

value corresponds to 1 JND Luminance in both HDR and LDR domains.

– The pixel values in LDR domain should be close to sRGB pixel values

Result: Common LDR Quality metrics (SSIM, PSNR) extended to HDR through the PU space transformation

Perceptually Uniform Space

[Aydın, Mantiuk, Seidel. 2008 Electronic Imaging]

Perceptually Uniform Space Derivation:

for i = 2 to N

Li = Li-1 + tvi(Li-1);

end for

Fit the absolute value and subject sensitivity to sRGB within CRT luminance range

Key Idea: Instead of the traditional contrast difference, use distortion measures agnostic to dynamic range difference.

Result: An IQA that can meaningfully compare an LDR test image with an HDR reference image, and vice versa. Enables objective evaluation of tone mapping operators.

Dynamic Range Independent IQA

[Aydın, Mantiuk, Myszkowski, Seidel. 2008 SIGGRAPH]

5x

LDR HDR Lu

min

ance

Lum

inan

ce

LDR HDR

HDR vs. LDR

Problem with Visible Differences

HDR Reference LDR Test HDR-VDP

95%

75%

50%

25%

Det

ecti

on

Pro

bab

ility

Co

ntr

ast

Loss

Local Gaussian Blur

Distortion Measures

Reference Test Contrast Loss

Reference Test Contrast Amplification

Reference Test Contrast Reversal

Novel Applications Tone Mapping Inverse

Tone Mapping

Video Quality Assessment

DRIVQM [Aydin et al. 2010] DRIVDP [Aydin et al. 2008] (frame-by-frame)

HDR Video (tone mapped for

presentation)

Key Idea: Extend the Dynamic Range Independent pipeline with temporal aspects to evaluate video sequences.

Result: An objective VQM that evaluates rendering quality, temporal tone mapping and HDR compression.

Dynamic Range Independent VQA

[Aydın, Čadík, Myszkowski, Seidel. 2010 SIGGRAPH Asia]

CSF: ω,ρ,La → S

– ω: temporal frequency,

– ρ: spatial frequency,

– La: adaptation level,

– S: sensitivity.

Contrast Sensitivity Function

CSF: ω,ρ,La → S

– ω: temporal frequency,

– ρ: spatial frequency,

– La: adaptation level,

– S: sensitivity.

Contrast Sensitivity Function

Spatio-temporal CSFT

CSF: ω,ρ,La → S

– ω: temporal frequency,

– ρ: spatial frequency,

– La: adaptation level,

– S: sensitivity.

Contrast Sensitivity Function

Steady-state CSFS

( )

Contrast Sensitivity Function

=

CSF(ω,ρ,La = L) CSFT(ω,ρ,La = 100 cd/m2) f(ρ,La)

x

÷ f =

CSFS(ρ,La) CSFS(ρ,100 cd/m2)

La = 100 cd/m2

CSFT(ω,ρ)

Extended Cortex Transform

Sustained and Transient Temporal Channels [Winkler 2005] Spatial

Evaluation of Rendering Methods

No temporal filtering With temporal filtering [Herzog et al. 2010]

Predicted distortion map

Evaluation of Rendering Qualities

High quality Low quality Predicted distortion map

Evaluation of HDR Compression

Medium Compression High Compression

Validation Study Noise, HDR video compression, tone mapping

“2.5D videos”

HDR-HDR, HDR-LDR, LDR-LDR

Psychophysical Validation

(1) Show videos side-by-side on a HDR Display

(2) Subjects mark regions where they detect differences

[Čadík, Aydın, Myszkowski, Seidel. 2011 Electronic Imaging]

Validation Study Results Stimulus DRIVQM PDM HDRVDP DRIVDP 1 0.765 -0.0147 0.591 0.488

2 0.883 0.686 0.673 0.859

3 0.843 0.886 0.0769 0.865

4 0.815 0.0205 0.211 -0.0654

5 0.844 0.565 0.803 0.689

6 0.761 -0.462 0.709 0.299

7 0.879 0.155 0.882 0.924

8 0.733 0.109 0.339 0.393

9 0.753 0.368 0.473 0.617

Average 0.809 0.257 0.528 0.563

Conclusion Starting Intuition: Working on “perceived” visual data,

instead of “physical” visual data.

Limitations and Future Work What about the rest of

the brain? – Visual Attention

– Prior Knowledge

– Gestalt Properties

– Free will

– …

User interaction?

Depth perception

Acknowledgements Advisors

– Karol Myszkowski, Hans Peter Seidel

Collaborators

– Martin Čadík, Rafał Mantiuk, Dawid Pająk, Makoto Okabe

AG4 Members

– Current and past

AG4 Staff

– Sabine Budde, Ellen Fries, Conny Liegl, Svetlana Borodina, Sonja Lienard.

Thesis Committee

– Phillip Slusallek, Jan Kautz, Thorsten Thormählen.

Family

– Süheyla and Vahit Aydın, Irem Dumlupınar

THANK YOU.

Tunç O. Aydın <tunc@mpii.de>