1

Spatial Modelsof Color

Alessandro RizziJohn J. McCann

International Symposium "Vision by Brains and Machines"November 13th - 17th Montevideo, Uruguay

Thanks to:

Carlo Gatta, Silvia Zuffi, Daniele Marini, Edoardo Provenzi, Angelo Moretti

color is the product of light at certain wavelenghts

color is the product of light at certain wavelenghts

NewtonNewton

color is the product of observer’s mindcolor is the product of observer’s mind

GoetheGoethe

Vision process

Newton

Goethe

Physical Perceptual

OUTLINE• The reality of color• The input• The appearance• Colorimetry vs

appearance• The models• The behavior• Applications

2

The physical reality of color

380 nm to 780 nm

Visible light

Solar SPD before and after passing atmosphere

a - zenith sunny sky b - northern sunny sky c – cloudy skyd – direct sunlight (sunny)

(normalized at 556nm)

SPD in nature

Fluorescent bulb

6500°K compensated fluorescent bulb

Low pressure sodium

Fluorescent bulb

Artificial light SPD

Macbeth Color Checker Spectral Reflectances

Dark skin

Light skin

Bluesky Foliage

Blueflower

Bluishgreen

Orange PurplishblueModerate

red PurpleYellowgreen

Orangeyellow

Blue Green Red Yellow Magenta Cyan

White Neutral 8Neutral

6.5Neutral

5Neutral

3.5 Black

ρ(λ)E(λL(λ )) =

The basic ambiguity

3

Color does not exist by itself

• Same SPD different color perception

• Same color perception different SPD

The input

Two very different devices Weber-Fechner law

cost=∆II

Stimulus difference

Stimulus magnitude

∆I

I

Light and dark adaptation

Light

Retina

4

Retinal cone and rod distribution

Retinal cone and rod distribution No (or bad) color without 3 separate channels

http://www.toledo-bend.com/colorblind/Ishihara.html

0 : Spectrum (45) gm©

400 500 600 700

1.00

0.00

Sensitivity curves of the human eye

Maximum red peak falls on green

Cone spectral sensitivity(normalized)

Cone spectral sensitivity

Max signal ratio

L/M=2

5

1.1:11.1:152.7%52.7%

16.5:116.5:194.3%94.3%

L:ML:M

The appearance

Mach bands Hermann illusion

6

Lightness perception

Simultaneous contrast

Color does not exist by itself

• Same SPD different color perception

• Same color perception different SPD

(J. Albers)

Assimilation(White effect)

Assimilation High vs low level

7

Sensation vs perception• Sensation: ”mode of mental functioning that

is directly associated with stimulation of the organism”;

• Perception: ”mode of mental functioning that include the combination of different sensations and the utilization of past experience in recognizing the objects and facts from which the present stimulation arises”.

committee on Colorimetry of the Optical Society of America in “The Science of Color”

Levels of appearance

Which radiance comes from the two sides ?STIMULUS = VERY DIFFERENT

which paint should a painter use to depict an image of the float ?

SENSATION = SLIGHTLY DIFFERENT

the float is painted with the same paint ?PERCEPTION = THE SAME

sensation: before recognition of shadows, junctions, geometries, belongingness, etc..

SCAs deal with sensation

Importance of the context

8

High level ?

Courtesy of John McCann

Colorimetryvs

appearance

Wright (1928) e Guild (1931) experiment

435,8 nm

700 nm

546,1 nm

monochromatic light

Wright(1928)

Guild(1931)

CIE 1931

9

(monitor Hitachi)

Phosphors spectral emissions

)(λeL)( )( )( )( λλλλ bBgGrRC ++=

∫∫∫

=

=

=

λλλ

λλλ

λλλ

dzLZ

dyLY

dxLX

e

e

e

)()(

)()(

)()(

1

1

1

∫∫∫

=

=

=

λλλ

λλλ

λλλ

dzCZ

dyCY

dxCX

)()(

)()(

)()(

2

2

2

∫∫∫∫∫∫

===

===

===

λλλλλλ

λλλλλλ

λλλλλλ

dzCdzLZZ

dyCdyLYY

dxCdxLXX

e

e

e

)()()()(

)()()()(

)()()()(

21

21

21

Metameric colors

Color does not exist by itself

• Same SPD different color perception

• Same color perception different SPD

Spectral data CMF triplets reproduced colors

CMF

CMF

CMF goal: preserve metamerism

C.I.E. 1931 C.I.E. 1964

Stiles and Burch (1955)

Demarco, Smith and Pokorny (1992) Stockman, MacLeod and Johnson (1993)

Vos and Walraven (1970)

Thornton(2002)

10

CMF spatial normalization

before after

overall

single colors

Bipartite fieldexperiment

CIE 1931

Color spaces generations1 - Physical color spaces (XYZ, RGB ..)2 - Psychophysical color spaces (CIELab,

Munsell ..)3 - Appearance (spatial) color

models/algorithms (CAM, Retinex, ACE ..)

?)(λeL

Vision Process

ρ(λ)

Physical Perceptual

11

Color depends on context

color is subject to several changes that depend on

contextSpatial computation

Lightnessconstancy

HVS adaptational mechanisms

HVS

HVS

Not completely

Hipparchus line Gray World

100

95 190

50 100

5 10

Perceived intensity

EstimateIllum. Int.

Illuminantintensity

Different illuminants

HVS

Colorconstancy

HVS

HVS adaptational mechanisms

Not completely

12

Which reference ?

D65, D50 ?

which is the “real color” ?

color constancy is controlled by channel maxima

Maximov shoeboxes

White Patch

HVS normalizes towards an hypothetical white reference area.

Von Kries

Sun / Shade

119119

InputGlobal vs local Global vs local

13

Edwin Land

Retinex Theory

John McCann

SCIENTIFIC AMERICAN 1959

Edwin Land experimentB/W film

Red filter Green filter

projection

Red filter

Slide projectors

Edwin Land projection

Edwin Land projection Second Land experiment

14

Land’s experiment

LS=255 LL=255LM=115

Projector Colorimeter

ES=100 EL=100EM=100

Observer

PINK PINKGRAY

Projector Colorimeter

ES=50 EL=50EM=111

Observer

LS=128 LL=128LM=128

Land’s experiment

The models

Computing the appearance

Retinex

ACE(Automatic Color Equalization)

Color comes from the comparison of the three lightness scales

“Color is the correlation number for several rank orders of lightness” (Land, 1977)

Retinex theoryThe term RETINEX comes from retina-and-cortex since is not clear

where the passage from radiant flux to lightness takes place

RadianceRadiant light power through a unitary solid angle (for a certain direction)measured in [W/m2/sr]

LightnessVisual sensation quality related to a surface that appear lighter or darker in relation to a “local reference white”

Reflectanceratio between radiant flux and reflected radiant flux (spectral)

Physical Perceptual

15

Let’s bust a myth

Retinex DOES NOT separate

illumination from reflectance(like human visual system)

ρ(λ)E(λL(λ )) =

Sequential products(Land and McCann 1971)

Random pathsSequential products

serching the global/local neighborooh to compute the ratios

The Retinex algorithm

• Relative lightness along a path is the sum of the log ratios of intensities:

• Reset mechanism (WP)

∑∈

+=pathx x

xjisml I

Ir 1, ,, log.δ

δ = 1, if logI x +1I x

> threshold

0 else

j

i

IkIk+1

The Retinex algorithm

• Relative lightness (R) at a point is the mean value of the relative lightnesses (r) computed along N random paths to the point, separately for each band long, medium, short:

N

rR

N

k

jisml

isml

k∑== 1

,,,

,,

16

A key point:how to realize locality

[S. Zeki]

Cortical receptive field distribution

Brownian paths Retinex MLV

Random Spray Retinex Center/surround Retinex (Jobson, Rahman, Woodel)

[ ]),(),(log),(log),( yxIyxFyxIyxR iii ∗−=

22 /),( crKeyxF −=

R(x,y) is Retinex output, I(x,y) is the image, F(x,y) is a neighbor function

• In 1986 Land propose a version without paths;• lightness is computed in a circular neighborood defined by 1/r2

Basic idea: receptive fields

17

Why it is not a spatial color algorithm

Filter shape does not depend on the image

Implementation coming from the latter work of Land

Multiresolution Retinex (McCann 99)No pathsMultiresolutionStarting from the smaller:

ratio-product-reset-averageon each levelAt the end oversampling and go to the upper level

Digital flash (HP)

18

Computing the appearance

Retinex

ACE(Automatic Color Equalization)

Chromatic/SpatialAdaptation

Dynamic ToneReproduction ScalingIc Rc Oc

subset selection

s(·) function

GW WP scaling referencelocal/globalbalancing

ACE structure Pixel recomputation

) (s=)(iRc ∑≠∈ ijsubsetj ,

)()( jIiI cc −

),( jid

For each channel C

∑≠∈

−=

ijsubsetj

ccc jid

jIiIsiR, ),(

))()(()(

>−

=−

−

otherwise1

thresholdLLifLL

δ1ki,ki,

1ki,

ki,jc

jcj

c

jc

ki,c

Ic(i)-Ic(j) lateral inhibition mechanism

Chromatic/Spatial recomputation (1)

ACERetinex

ratio product1ki,

ki,

jc

jc

LL

−

∑≠∈

−=

ijsubsetj

ccc jid

jIiIsiR, ),(

))()(()(

Reset non-linearity, WP

s(·) non-linearity, WP, GW

ACERetinex

x

j1,1j 2,1

ji,1j i,k+1

ji,k

... ...

...

...

Resetmechanism

Chromatic/Spatial recomputation (2)

19

Contrast tuningACE

Linear Saturation Signumd1

x

j1,1j 2,1

ji,1j i,k+1

ji,k

... ...

...

...

ACERetinex

d(·) local/global balancingEuclidean distance

∑≠∈

−=

ijsubsetj

ccc jid

jIiIsiR, ),(

))()(()(

Random paths local/global balancing

Chromatic/Spatial recomputation (3)

Chromatic/SpatialAdaptation

Dynamic ToneReproduction ScalingIc Rc Oc

subset selection

s(·) function

GW WP scaling referencelocal/globalbalancing

ACE structureWP/GW scaling:

Estimated “white” White PatchEstimated “gray” Gray World

)](5.127

5.127[ )( max iRRroundiO cc +=

Dynamic tone reproduction scaling

medium gray point Rmax

R Histogram O Histogram

127 2550

Spatial color models common structure

• First phase: pixel recomputationaccording to the visual (spatial) image characteristics

• Second phase:rescaling according to– available dynamic – Global principles (WP, GW)– rendering intent

The behavior

20

Filtering goals

• Qualitative

• Quantitative (parameters, I/O calibration,

dynamic, etc...)

Color Constancy

Original Filtered(255,255,255)

No constraint and no a priori information required

Color constancy by spatial comparison

equal

different

Unwanted color removal

Alternative solution: apply SCA only on Lightness channel

Local filtering effectOriginal Filtered

RGB differences filtered-Original around “128” gray

Contrast correction

Original Filtered Original Filtered

global

local local

global

21

Shadow removal

Rearrange contrast locallyDo not separate illuminant and reflectance

Orig Filt O-F

Image dynamic adjustment

Tone remapping (LC)

Original

Filtered ACE

Original

ALL

GW required

Spatial dequantization

WP WP + GW

Equal radiance, different appearance Filtering visual Illusion

211

43

128

128

Original Filtered

Original

Filtered

22

HVS spatial computationnever adapts completely

does not estimate illuminantchange according to image content

STABLE UNSTABLE

Colour ConstancyLightness ConstancyContrast constancy ?

Simoultaneous ContrastAssimilationEdge effects

ROBUSTNESSApplications

“Incendio di Borgo” (Raffaello 1514)

Application: image DB

Retinex Prefiltering

Matching ?

Matching !

Application: image DB

Prefilteringfor computer vision or medical imaging Prefiltering for edge detection

23

InterfacesUser preferences

0

10

20

30

40

50

60

70

80

luminosità colori leggibilità preferenzaglobale

originaleACE

~8000 cd/m2

~1000 cd/m2

L=205

L=105

8:1 → 2:1

[Gatta]

HDR images remapping Automatic digital movie restoration

Another example

Hue histogram original image

Original image After filtering

Hue histogram result image

Thank you

rizzi@dti.unimi.it

International Symposium "Vision by Brains and Machines"November 13th - 17th Montevideo, Uruguay