High dynamic images between devices and vision limits

Le immagini ad alta dinamica tra i limiti dei dispositivi e quelli

della visioneAlessandro Rizzi

Dipartimento di Informatica e ComunicazioneUniversità degli Studi di Milano

Friday, June 10, 2011

Outline

HDR imaging

HDR in practice: measuring the limits

Using HDR

The dynamic range

Define HDR ?

do we need a threshold number ?

Define HDR ?

do we need a threshold number ?

Define HDR

A rendition of a scene with greater dynamic range than

the reproduction media

That is ?

Annibale Carracci (1560-‐1609) PaesaggioFriday, June 10, 2011

Photo: C. OleariFriday, June 10, 2011

Annibale Carracci (1560-‐1609) PaesaggioFriday, June 10, 2011

Source/lamp Average Luminance cd/m2

Xenon short arc 200 000 ÷ 5 000 000 000Sun 1 600 000 000Metal halide 10 000 000 ÷ 60 000 000Incandescent 20 000 000 ÷ 26 000 000compact Fluorescent 20 000 ÷ 70 000Fluorescent 5 000 ÷ 30 000Sunlit clouds 10 000Candle 7 500blue sky 5 000Preferred values for indoor lighIng

50 ÷ 500

White paper at sun 10 000White paper at 500 lx 100White paper at 5 lx 1

Courtesy: C. Oleari

Light levels

Dynamic ranges

?Friday, June 10, 2011

Range limits and quantization: the ‘salame’ metaphor

Range compression from incorrect pixel perspective

Very wide range obtained with isolated stimuliimpossible to obtain in an image

The “salame” metaphor

Dynamic range Quantization

The “salame” metaphor

Dynamic range Quantization

More bits do not mean wider rangeLess bits do not mean shorter range

SceneDR

SensorDR

16 bit

28=2562-3 log unit

216=65536 4-5 log unit

SceneDR

SensorDR

16 bit

28=2562-3 log unit

216=65536 4-5 log unit

SceneDR

SensorDR

2-3 log unit

4-5 log unit

SceneDR

SensorDR

16 bit

SceneDR

SensorDR

8 bit 16 bit

SceneDR

SensorDR

SceneDR

SensorDR

8 bit 16 bit

SceneDR

SensorDR

The HDR idea

http://www.adolfo.trinca.name/public/2010/11/ahdrdiagram.jpg

The HDR ideaHow ?

general solution ?rendering intent ?

http://www.digitalcameratracker.com/how-to-create-high-definition-range-hdr-photos/

Two sides of the coin

• Objective data: recording/displaying physical light colorimetric distribution

• Subjective data: reproducing appearance (or different rendering intent)

Mapping the world: the characteristic curve

H & D curve

http://www.dpreview.com/reviews/olympuse3/page21.asp

Olympus E-3

Exposure problem

History of HDR imaging

HDR 1858H.P. Robinson “Fading Away

Mees (1920) 2 negative print

“The Fundamentals of Photography”

Ansel Adams

ISCC 11/05-McCann

Ansel Adams - Zone System

Jones and Condit, 1941Measurements of dynamic range of real scenes

0.0 3.01.5log range

REFLECTANCE RANGE OF PRINTS

Maximum

Average of 126 outdoor scenes

Minimum

SCENE RANGE OF WORLD

L.A.Jones & H.R.Condit, JOSA,1941

Retinex starting idea

Green record

ratio = 0.62

digit ~ luminance 119 119

Ratios are constant in sun and shadeFriday, June 10, 2011

1980Friday, June 10, 2011

Retinex cameraFriday, June 10, 2011

Capturing and reproducing the scene

Sensors dynamic range

Limited !

Is HDR a technological problem ?

Expanding sensors dynamic range• Sensors that compress their response to light due to their

logarithmic transfer function;• Multimode sensors that have a linear and a logarithmic

response at dark and bright illumination levels, (switches between linear and logarithmic modes of operation);

• Sensors with a capacity well adjustment method;• Frequency-based sensors, sensor output is converted into

pulse frequency; • Time-to-saturation [(TTS); time-to-first spike] sensors,

signal is the time the to saturated pixel; • Sensors with global control over the integration time; • Sensors with autonomous control over the integration time,

where each pixel has control over its own exposure.Spivak A, Belenky A, Fish A & Yadid-Pecht O (2009) Wide dynamic-range CMOS image sensors:

A comparative performance analysis, IEEE Trans. on Electron Devices, 56, 2446-2461.Friday, June 10, 2011

The HDR idea

The HDR ideaHow ?

Multiple image acquisition

•Multiple Exposures

•Use Multiple Times

•Recover scene radiances at all pixelsfrom camera digits

CameraDigit = radiance* time( )

New goal: Accurately measure radiances

Multiple Exposures

Flux = Luminance * time

Scene Luminance = Flux / time

Scene Luminance = Camera Digit / time

Multiple Exposures

One Spot (ScaleD)

0.0001 0.0010 0.0100 0.1000 1.0000 10.0000 100.0000 1000.0000Exposure Flux [(cd/m2) * sec]

1/8 sec1/4 sec1/2 sec1 sec2 sec4 sec8 sec16 sec32 sec64 secFIT

Flux = Luminance * time

Camera Digit

HDR file formats

Source: Reinhard et al., High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics)

HDR file formats

Source: Reinhard et al., High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics)

Acquisition limits

The glare problem

Effect of illumination

Assumes 0.0 glare

1.0 refl * 1.0 illum = 1.0 cd/m2

0.2 refl *0.01 illum = 0.002 cd/m2

Glare is image dependent

Assumes 0.001 glare

1.0 refl * 1.0 illum = 1.0 cd/m2

0.2 refl *0.01 illum = 0.002 cd/m2

0.002 cd/m2 *0.001 = 0.000002

1.0 cd/m2 *0.001 = 0.001

Ratio Signal/Glare

Assumes 0.001 glare

1.0 cd/m2)/(0.000002) = 5*10^5

( 0.002 cd/m2)) / (0.001) = 2

Sowerby, “Dictionary of Photography”, 1956

Parasitic Images

Camera limits•Glare•Unwanted scattered light in camera

•air - glass reflections •lens (number of elements)•aperture •angle off optical axis

•camera wall reflections•sensor surface reflections

•We must measure actual veiling glare limit

Measuring overall camera glare

HDR Test Setup

Synthetic HDR(High-Dynamic Range)

Images

18,619:1

digit 255 = 2094.2 cd/m2

= 18,619

digit 0 = 0.11 cd/m2

2094.2 cd/m2

0.11 cd/m2

Goal Image

Targets

18,619:1

16 sec exposure - Target 1scaleBlackFriday, June 10, 2011

16 sec exposure - Target 4scaleBlackFriday, June 10, 2011

16 sec exposure

TextTarget 1B

Target 4B

Target 4W

Constant Luminance - Variable SurroundFriday, June 10, 2011

Minimum Glare

Mild Glare

Maximum Glare

Measure In-camera Accuracy

SceneScene

Dynamic Range

In-cameraAccurate

MaximumError

(% radiance)

1scaleB 20:1 20:1 0

4scaleB 18,619:1 3,000:1 300% Min

4scaleW 18,619:1 100:1 10,000% Max

4.3 log10 scene ----> 3.0 log10 image

Side Dupe FilmFriday, June 10, 2011

Slide Dupe FilmFriday, June 10, 2011

One Negative Capture4scale Black - Single Negative

-1.00 -0.50 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50Log Cd/m2

3.5 Log10 units

Dynamic Range (OD)Friday, June 10, 2011

HDR from cameras

• Range of usable captured information

• Range of accurate luminance information

(much smaller)

• Scene dependent

Courtesy: M. Fairchild

Glare insertion

Gregory Ward Larson, Holly Rushmeier, and Christine Piatko, “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes”, IEEE Trans on VISUALIZATION AND COMPUTER GRAPHICS, VOL. 3, NO. 4, oct-dec 1997

Display: measuring the human limits

Magnitude estimates (100-1)Friday, June 10, 2011

•Luminance does not correlate uniquely with appearance

•No global tone scale can render the appearance

Magnitude Estimation of AppearanceChange Surrounds

0102030405060708090

0.10 1.00 10.00 100.00 1000.00 10000.00Log Luminance (cd/m2)

Min [0 cd/m2] Max [2094 cd/m2]

We need a new range target

•White surround•adds glare•changes surround

(simultaneous contrast)

•Vary dynamic range with•constant glare•contrast surround

Center/Surround Basic Unit

Fixed contrast surround 88%

Gray test areas 12%(small differences)

90o rotation

Testing different glares% of white surround

Single =

Double =(superimposed)

5.4 log10 range

2.7 log10 range

Single & Double Density Transparencies

5.4 & 2.7 log10 Ranges Constant Glare & Surround

0123456relative optical density

50% Single Density

50% whitesurround

White[100] = 0.0 rOD - Black [1] = 2.89 rOD

50% Double Density 50% Single Density

50% whitesurround

2.3 log10 units

White Double Density White Single Density

2.0 log10 units

100% whitesurround

Black Double Density Black Single Density

0% whitesurround

5.0 log10 units

Black Double Density Black Single Density

0% whitesurround

5.0 log10 units

Over 20 not big improvement

Measurements of apparent range

(depends on area of white)

•100% = 2.0 log10 units

• 50% = 2.3 log10 units

• 8% = 2.9 log10 units

DD DD DD DDFriday, June 10, 2011

Test summary

•Double transmission contrast

•Double dynamic range

•very small change in appearance range

•Visual limit ~ area of white surround

•area of white controls glare

What is on the retina: calculated retinal luminance

What comes to the retina is different from the image

High glare Low glare

Glare vs. Contrast

Veiling glare increases gray luminance

Contrast decreases gray appearance

Contrast offsets glare

Discussion

• Glare lowers the physical contrast • Spatial comparisons increase the

contrast of appearance.

• The two act in opposition. • Change with distance are different and

the cancellation is far from exact.

1 Vos, J.J. and van den Berg, T.J.T.P, CIE Research note 135/1, “Disability Glare”, ISBN 3900734976 (1999).

PIGMENTBlue eyed Caucasian 1.21Blue green Caucasian 1.02Mean over all Caucasian 1.00Brown eyed Caucasian 0.50Non Caucasian with pigmented skin and dark brown eyes 0.00

Glare Spread Function

Plotted in log scale

False-color LookUpTable (LUT)

Dynamic Range = 5.4 ODor 251,189:1

Visualize HDR targets

Same LUT applied to SD & DD

Retinal image

Visualize Retinal ImagesFriday, June 10, 2011

Change LUT for Retinal ImagesFriday, June 10, 2011

Two scene-dependent spatial mechanisms:glare and contrast

Glare masks the strength of spatial contrast

Scene Retina Appearance 1,000,000:1 100:1 1,000:1

SpatialGlare

SpatialContrast

Ranges

Tone-rendering problem and spatial comparisons

Choosing a rendering intent

Land experiment

Projector

Land experiment

Projector

Land experiment

Projector

ES=100 EL=100EM=100

Land experiment

Projector Colorimeter

ES=100 EL=100EM=100

Land experiment

LS=255 LL=255LM=115

ES=100 EL=100EM=100

Land experiment

LS=255 LL=255LM=115

ES=100 EL=100EM=100

Observer

Land experiment

LS=255 LL=255LM=115

ES=100 EL=100EM=100

Observer

Land experiment

ES=50 EL=50EM=111

Observer

Land experiment

ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment

ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment

PINKGRAY

ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land experiment

visual sensation

HVS: local compression of range

Tone mapping vs Tone rendering

No tone mapping operator (global) can mimic vision

We need an image dependent tone renderer operator (local)

Black and White Mondrian

HP 945 Images without “Frames of Reference”Friday, June 10, 2011

Some examples

Bob Sobol, HP

R. Sobol, “ Improving the Retinex algorithm for rendering

wide dynamic range photographs”, in Human Vision and Electronic

Imaging VII, B. E. Rogowitz and T. N. Pappas, ed., Proc. SPIE 4662-41, 341-348,

Original ACEOriginal ACE

STRESS Tone Rendering

Judging the results

Beauty contest

C. Gatta, A. Rizzi, D. Marini, “Perceptually inspired HDR images tone mapping with color correction”, Journal of Imaging Systems and Technology, Volume 17 Issue 5, pp. 285-294 (2007).

HDR is in the middle

Imagein CPUmemory

Scene DisplaySpatialImagein CPU

SpatialAlgorithm

GlareSensor

Pre-LUT

Post-LUTgraphics

Summary

• To understand HDR we need a new perspective!

1. Veiling glare limits the range on the retina 2. Neural processing (spatial) determines appearance 3. Neural is stronger than it appears [neural cancels glare] 4. General Solution requires spatial process [mimic vision] 5. Tone-Scale is limited, we need Tone-rendering [scene dependent]

Take home points

• HDR limits are not (only) technological

• Glare limits both acquisition and vision

• Glare is scene dependent

• Human vision use spatial comparison to overcome this limit

• Tone renderer operator can use the same approach

Take home points

HDR works very well

• because preserves image information

• not because are more accurate (not possible)

References• J. J. McCann, A. Rizzi, “Camera and visual veiling glare in HDR images”

Journal of the Society for Information Display 15/9, 721–730 (2007).

• J. J. McCann, “Art, Science and Appearance in HDR” Journal of the Society for Information Display 15/9, 709–719 (2007).

• A. Rizzi, J. J. McCann, “Glare-limited Appearances in HDR Images”, Journal of the Society for Information Display, 17/1, pp. 3-12, (2009).

• J. J. McCann, A. Rizzi, “Retinal HDR Images: Intraocular Glare and Object Size” Journal of the Society for Information Display, 17/11, pp. 913-920, (2009).

The art and science of HDR imagingJ.J. McCann, A. Rizzi

(expected publication date autumn 2011)

Thank you

alessandro.rizzi@unimi.it

High dynamic images between devices and vision limits

Technology

Transcript of High dynamic images between devices and vision limits

Sizing and Operating Conditions for Reference Designs · Dynamic Limits forSkill Groups andPrecision Queues PerAgent Thenumberofskillgroupsandprecisionqueuesperagentsignificantlyaffectsthefollowingsubcomponents

Dynamic Management of Input/Output Devices for Wearable ...589182/FULLTEXT01.pdfDe Morais W.O. Dynamic Management of Input/Output Devices for Wearable Computers Page 2 of 47 Acknowledgements

Determine Performance of Content-Aware Devices and Networks · Determine Performance of Content-Aware Devices and Networks IxLoad • Determine Performance Limits of Load Balancers,

Operating Limits and Dynamic Average- Value Modelling of ...

Connecting End Customer Devices Without Limits

An 558: Implementing Dynamic Reconfiguration in Arria II Devices · 2021. 7. 5. · Page 2 AN 558: Implementing Dynamic Reconfiguration in Arria II Devices December 2010 Altera Corporation

Dynamic Energy Monitoring for Smart Devices using IoTscse.ntu.edu.sg/NewsnEvents/PublishingImages/SCSEFYP/Apr2019/… · Dynamic Energy Monitoring for Smart Devices using IoT Technology

Integration of FACTS Devices Into a Dynamic Power

Total Dose Effects on Devices and Circuits - Principles and Limits of Ground Evaluation-

High Dynamic Range Imaging: Towards the Limits of the Human … · 2013-03-14 · High Dynamic Range Imaging: Towards the Limits of the Human Visual Perception Rafał Mantiuk Max-Planck-Institut

Understanding Dynamic Games: Limits, Continuity, and ...scholar.harvard.edu/...to_long_run_collaboration.pdf · Understanding Dynamic Games: Limits, Continuity, and Robustness Economists'

Avaya.Prep4sure.7593X.v2018-07-06.by.Christopher › avaya › 7593x › Avaya.Prep4sure.7593… · A. It limits the network discovery to older ERS devices only. B. It limits the

Dynamic time warping and PIC 16F676 for control of devices

Dynamic Management of Input/Output Devices for Wearable Computers

DYNAMIC DEVICES - PICKUPS AND KICKERS* · DYNAMIC DEVICES - PICKUPS AND KICKERS* Glen Lambectson Lawrence Berkeley Laboratory ... Office of High Energy and Nuclear Physics, High Energy

Exploring the Ultimate Limits of Adiabatic Circuits · Special Session: Exploring the Ultimate Limits of Adiabatic Circuits Michael P. Frank ... offer the potential to reduce dynamic

Dynamic behaviour and operational limits of stabiliser · PDF fileDYNAMIC BEHAVIOUR AND OPERATIONAL LIMITS OF STABILISER FINS Guilhem Gaillarde* Maritime Research Institute Netherlands

Fundamental limits on dynamic inference from single-cell snapshots · Fundamental limits on dynamic inference from single-cell snapshots Caleb Weinreba, Samuel Wolocka, Betsabeh K.

Dynamic Security Feed for Office 365...O365 is a security feature within OneConﬁg’s platform for SRX and vSRX devices. How it Works Juniper SRX and vSRX devices include a dynamic

Implantable Electronic Cardiac Devices for Prevention of Sudden Cardiac Death and Treatment of Cardiac Arrhythmias. What are the Limits to Use It? Limits.