Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol...

Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range

Video Video EncodingEncoding

Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range

Video Video EncodingEncoding

Rafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter SeidelRafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,

Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter Seidel

INFORMATIKINFORMATIK

[log cd/m^2]Luminance

-6 -4 -2 0 2 4 6 8

Bright projector

LCD Monitor

CRT Monitor

Human eye

High Dynamic RangeHigh Dynamic RangeHigh Dynamic RangeHigh Dynamic Range

High vs Low High vs Low Dynamic Range VideoDynamic Range VideoHigh vs Low High vs Low Dynamic Range VideoDynamic Range Video

LDR Video

Intended for

existing displays

Relative pixel brightness

HDR Video

Intended for

the human eye

Photometric or radiometric units [cd/m2, Watt/m2sr]

High Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range Video

Goal: Efficient encoding of full dynamic range of luminance perceived by the human observer

Luminance-6 -4 -2 0 2 4 6 8

[log cd/m^2]

• 1st demo

OverviewOverviewOverviewOverview

1. HDR Pipeline

2. HDR Video Encoding• Luminance Quantization

• Edge Coding

3. Results • vs. MPEG-4

• vs. OpenEXR

4. Demo & Applications

RelatedRelated Work WorkRelatedRelated Work Work

HDR Pipeline

Acquisition Storage Display


HDR Pipeline


• Global Illumination

• HDR Cameras

•HDRC (IMS Chips)

•Lars III (Silicon Vision)

•Autobrite (SMal Camera Technologies)

•LM9628 (National)

•Digital Pixel System (Pixim)

•Technology overview [Nayar2003]

HDRC – IMS Chips


HDR Pipeline


• Still images

•Radiance – RGBE [Ward91]

•OpenEXR [Bogart2003]

•logLuv TIFF [Ward98]

•HDR JPEG [Ward2004]

• Video

•No video format


HDR Pipeline


• LDR Displays

•But Tone Mapping necessary

• HDR displays start to appear

•University of British Columbia [Seetzen2004]

HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework

Detail level 1: Input & Output

White: MPEG

Orange: HDR Encoder

HDR

LDR bitstreamVideo encoder


Detail level 2: Color Transform

White: MPEG

Orange: HDR Encoder

bitstreamColorTransform Encoder

Video

HDR

LDR YCrCb

L u'v'p


Detail level 3: Edge Coding

White: MPEG

Orange: HDR Encoder

ColorTran. Comp.

MotionCodingDCT

Run-length

EdgeCoding

LDR

HDR

bitstream

Variablelength

ColorTran. Comp.

MotionCodingDCT

Run-length

EdgeCoding

LDR RGB

HDR XYZ

bitstream

Variablelength

Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color

Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color

How to represent color data?• Floating Points – ineffective compression• Integers – ok, but require quantization

How to quantize color data?• Quantization errors < threshold of perception• Use uniform color space (L*u*v*, L*a*b*) [Ward98]• Find minimum number of bits

Color (u*v*) – 8 bits are enough

Encoding of Encoding of LuminanceLuminanceEncoding of Encoding of LuminanceLuminance

How to quantize luminance?• Gamma correction?• Logarithm?

Integer representation

log

Lu

min

anc

e Y

-4

-2

0

2

4

6

8

log(Y)?

Threshold Versus IntensityThreshold Versus IntensityThreshold Versus IntensityThreshold Versus Intensity

Psychophysical measurements• The smallest perceivable difference Y for a

certain adaptation level YA

• tvi [Ferwerda96, CIE 12/2.1]

YA - Adaptation

Luminance

Ytvi

-2-4 0 2 4 6 8-2

0

2

4

6

log Adaptation Luminance YA

log

Thr

esh

old

Y

Maximum quantization error

Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization

-4

-2

0

2

4

6

8

0 2048Integers Lp

log

Lu

min

anc

e Y

f

Ytvi )(

maxe

Just below threshold of perception

Maximum quantization error

Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization

-4

-2

0

2

4

6

8

0 2048Integers Lp

log

Lu

min

anc

e Y

f

Ytvi )( ))((2)( 1 ltvif

dl

ld

maxe

decrease threshold

mapping to)(

space in

f

YLl

LLl

P

P

Just below threshold of perception

Capacity function [Ashkihmin02]

Grayscale Standard Display Function [DICOM03]

10 – 11 bits are enough

Luminance QuantizationsLuminance QuantizationsComparisonComparisonLuminance QuantizationsLuminance QuantizationsComparisonComparison

RGBE

-2 0 2 4-4 6 8log

Con

tras

t Thr

esho

ld

log Adapting Luminance

-4

-2

0

2

32-bit LogLuv

cvi11-bit percep. quant.

ColorTran. Comp.

MotionCodingDCT

Run-length

EdgeCoding

LDR RGB

HDR XYZ

bitstream

Variablelength

Edge CodingEdge CodingEdge CodingEdge Coding

Edge CodingEdge Coding: : MotivationMotivationEdge CodingEdge Coding: : MotivationMotivation

HDR video can contain sharp contrast edges• Light sources, shadows

DCT coding of sharp contrast may cause high frequency artifacts

DCT coding Edge coding

Edge CodingEdge Coding: : SolutionSolutionEdge CodingEdge Coding: : SolutionSolution

Solution: Encode sharp edges in spatial domain, the rest in frequency domain

Original Signal

Sharp edge signal

Smoothed signal

Run-length

encoding

DCT

encoding

Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm

original

horizontal decomposition

edge block

horiz. edges

edge block

vert. edges

horizontal DCT

vertical decomposition

vertical DCT

I

II

III

IV

ResultsResultsResultsResults

2x size of tone-mapped MPEG-4 video

20-30x saving compared to intra-frame compression (OpenEXR)

27

1

0.55

OpenEXR

HDR Enc.

MPEG-4

Bit-stream Size

Demo & ApplicationsDemo & ApplicationsDemo & ApplicationsDemo & Applications

Display dependent rendering

Choice of tone-mapping

Extended postprocessing

ConclusionsConclusionsConclusionsConclusions

HDR video compression• Modest changes to MPEG-4

• Lpu’v’ color space– Luminance quantization (10-11 bits)

• Edge coding

Applications• On-the-fly tone mapping

• Blooming, motion blur, night vision

• Tuned for display– LDR / HDR Display

AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments

HDR Images and Sequences• Paul Debevec• SpheronVR• Jozef Zajac• Christian Fuchs• Patrick Reuter

HDR Camera• HDRC(R) VGAx

courtesy of IMS CHIPSwww.hdrc.com

Comments and help• Volker Blanz

• Scott Daly

• Michael Goesele

• Jeffrey Schoner

Thank youThank youThank youThank you

http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/

Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol...

Documents

Transcript of Perception-motivated High Dynamic Range Video Encoding Rafal Mantiuk, Grzegorz Krawczyk, Karol...