Cameras and Image Processing

Computational Photography

CSE 291

Lecture 2

Announcements

• Assignment 1 will be released today

– Due Apr 8, 11:59 PM

CSE 291, Spring 2020 2

Traditional photography

• The acquisition of images by recording light (or other electromagnetic radiation) either electronically (using an image sensor) or chemically (using film)

• Typically, a lens is used to focus the light reflected or emitted from objects in the scene being imaged into an image on the light sensitive surface inside a camera during a timed exposure

CSE 291, Spring 2020 3

Geometric image formation

• How do 3D world points project to 2D image points?

CSE 291, Spring 2020 4

The projective camera

• Extrinsic Parameters: Since camera may not be at the origin, there is a rigid transformation between the world coordinates and the camera coordinates

• Intrinsic parameters: Since scene units (e.g., cm) differ image units (e.g., pixels) and coordinate system may not be centered in image, we capture that with a 3x3 transformation comprised of focal length, principal point, pixel aspect ratio, and skew

=

T

Z

Y

X

w

y

x

parameters extrinsic

by drepresente

tionTransforma Rigid

0100

0010

0001

parameters intrinsic

by drepresente

tionTransforma

3 x 3 4 x 4

CSE 291, Spring 2020 5

The reason for lenses

CSE 291, Spring 2020 6

We need light, but big pinholes cause blur

Thin lens, image of point

CSE 291, Spring 2020 7

O

All rays passing through lens and starting at Pconverge upon P’

So light gather capability of lens is given the area of the lens and all the rays focus on P’ instead of become blurred like a pinhole

F

P

P’

Thin lens, image plane

CSE 291, Spring 2020 8

OF

P

P’

Image Plane

Q’

Q

A price: Whereas the image of P is in focus,the image of Q isn’t.

Thin lens, aperture

CSE 291, Spring 2020 9

O

P

P’

Image Plane • Smaller Aperture-> Less Blur

• Pinhole -> No Blur

Lens distortion

• Radial distortion

• Tangential distortion

CSE 291, Spring 2020 10

Images of straight lines should be

straight

Photometric image formation

• What color is the projected point?

CSE 291, Spring 2020 11

Need bidirectional reflectance distribution function (BRDF) at

point on surface

Lighting, reflectance, and shading

CSE 291, Spring 2020 12

BRDF

Image acquisition

CSE 291, Spring 2020 13

Digitization, one row of image

CSE 291, Spring 2020 14

Digitization, whole image

CSE 291, Spring 2020 15

Number of quantization levels

CSE 291, Spring 2020 16

256 128

64 32

16 8

4 2

Image sensing pipeline

CSE 291, Spring 2020 17

Image processing

Image processing

• A discipline in which both the input and output of a process are images

– There are usually other input parameters to the process

CSE 291, Spring 2020 18

Demosaicing

CSE 291, Spring 2020 19

Image sensor

Color filter array (CFA)

Bayer pattern

CFA Interpolated (lower case)pixel values

Image processing

• Color spaces

• Gamut mapping

• White balancing and color balancing

CSE 291, Spring 2020 20

Image coding

• Common standards– Video

• Recommendation ITU-R BT.601 (standard-definition television (SDTV))

• SMPTE standard 240M (precursor to Rec. 709)• Recommendation ITU-R BT.709 (high-definition television

(HDTV))

– Image• sRGB• Adobe RGB• Wide gamut RGB (or Adobe Wide Gamut RGB)• ProPhoto RGB (or reference output medium metric (ROMM)

RGB)

CSE 291, Spring 2020 21

Different codings, different pixel values

CSE 291, Spring 2020 22

Color specification: chromaticity

• Chromaticity coordinates

– (x, y, z)

where x + y + z = 1

– Usually specified by (x, y)

where z = 1 – x – y

The CIE 1931 color space chromaticity diagram

CSE 291, Spring 2020 23

Chromaticities

• Set of chromaticities

– Red

– Green

– Blue

– White (point)

CSE 291, Spring 2020 24

Standard illuminantsHue of each white point, calculated with luminance Y = 0.54

CSE 291, Spring 2020 25

Chromaticities of common video standards

CSE 291, Spring 2020 26

Color CIE x CIE y CIE z

Red 0.6400 0.3300 0.0300

Green 0.3000 0.6000 0.1000

Blue 0.1500 0.0600 0.7900

White 0.3127 0.3290 0.3583

Recommendation ITU-R BT.709

Color CIE x CIE y CIE z

Red 0.6400 0.3300 0.0300

Green 0.2900 0.6000 0.1100

Blue 0.1500 0.0600 0.7900

White 0.3127 0.3290 0.3583

Recommendation ITU-R BT.601625 lines

Color CIE x CIE y CIE z

Red 0.6300 0.3400 0.0300

Green 0.3100 0.5950 0.0950

Blue 0.1550 0.0700 0.7750

White 0.3127 0.3290 0.3583

Recommendation ITU-R BT.601525 lines

Color CIE x CIE y CIE z

Red 0.6300 0.3400 0.0300

Green 0.3100 0.5950 0.0950

Blue 0.1550 0.0700 0.7750

White 0.3127 0.3290 0.3583

SMPTE standard 240M

D65

D65 D65

D65

Chromaticities of common image standards

CSE 291, Spring 2020 27

Color CIE x CIE y CIE z

Red 0.6400 0.3300 0.0300

Green 0.3000 0.6000 0.1000

Blue 0.1500 0.0600 0.7900

White 0.3127 0.3290 0.3583

sRGB

Color CIE x CIE y CIE z

Red 0.7347 0.2653 0.0000

Green 0.1152 0.8264 0.0584

Blue 0.1566 0.0177 0.8257

White 0.3457 0.3585 0.2958

Wide gamut RGB(or Adobe Wide Gamut RGB)

Color CIE x CIE y CIE z

Red 0.7347 0.2653 0.0000

Green 0.1596 0.8404 0.0000

Blue 0.0366 0.0001 0.9633

White 0.3457 0.3585 0.2958

ProPhoto RGB(or reference output media metric (ROMM RGB))

Color CIE x CIE y CIE z

Red 0.6400 0.3300 0.0300

Green 0.2100 0.7100 0.0800

Blue 0.1500 0.0600 0.7900

White 0.3127 0.3290 0.3583

Adobe RGB

D65

D50 D50

D65

Chromaticity diagrams

Rec. 709 and sRGB35.9% of visible colors

Adobe RGB52.1% of visible colors

CSE 291, Spring 2020 28

Chromaticity diagrams

Rec. 709 and sRGB35.9% of visible colors

Wide gamut RGB77.6% of visible colors

CSE 291, Spring 2020 29

Chromaticity diagrams

Rec. 709 and sRGB35.9% of visible colors

ProPhoto RGB90% of visible colors

CSE 291, Spring 2020 30

Academy Color Encoding Specification (ACES)

Color CIE x CIE y CIE z

Red 0.73470 0.26530 0.00000

Green 0.00000 1.00000 0.00000

Blue 0.00010 -0.07700 1.07690

White 0.32168 0.33767 0.34065

ACES

Approximately D60

100% of visible colors

CSE 291, Spring 2020 31

Nonlinear encoding

• All of these standards use nonlinear encoding (gamma encoding)– Video

• Recommendation ITU-R BT.601 (standard-definition television (SDTV))

• SMPTE standard 240M (precursor to Rec. 709)

• Recommendation ITU-R BT.709 (high-definition television (HDTV))

– Image• sRGB

• Adobe RGB

• Wide gamut RGB (or Adobe Wide Gamut RGB)

• ProPhoto RGB (or reference output medium metric (ROMM) RGB)

CSE 291, Spring 2020 32

Nonlinear encoding and conversion to linear

• Typical CRT monitors have a transfer function of gamma = 2.2

• Image and video standards were designed to be directly displayed on CRTs

– Pixel values are encoded to approximate gamma = 2.2

• Nonlinear to linear (floating-point) using a lookup table

• Linear to nonlinear calculation

CSE 291, Spring 2020 33

LinearNonlinear

Nonlinear R’G’B’ color space and linear RGB color space

• Example: sRGB

CSE 291, Spring 2020 34

RGB linear

sRGB nonlinear

Slope of sRGB nonlinearin log-log space

XYZ color space

• Encompasses all color sensations the average person can experience

• Standard reference– Many other color space definitions are based on XYZ

• Y is luminance• Z is quasi-equal to blue stimulation• X is a linear combination of cone response curves chosen to be

nonnegative• The plane parallel to the XZ plane and that Y lies on contains all

possible chromaticities at that luminance

CSE 291, Spring 2020 35

RGB color space to XYZ color space

𝑋𝑌𝑍

=

𝑟𝑋 𝑔𝑋 𝑏𝑋𝑟𝑌 𝑔𝑌 𝑏𝑌𝑟𝑍 𝑔𝑍 𝑏𝑍

𝑅𝐺𝐵

𝑋𝑌𝑍

=

𝑟𝑥(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑥(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑥(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)𝑟𝑦(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑦(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑦(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)

𝑟𝑧(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑧(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑧(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)

𝑅𝐺𝐵

𝑋𝑌𝑍

=

𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧

𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍

𝑅𝐺𝐵

CSE 291, Spring 2020 36

RGB color space to XYZ color space

• Substitute in RGB white and XYZ white

CSE 291, Spring 2020 37

𝑊𝑅𝐺𝐵 =111

𝑊𝑋𝑌𝑍 =

𝑤𝑥𝑤𝑦1𝑤𝑧𝑤𝑦

𝑤𝑥𝑤𝑦1𝑤𝑧𝑤𝑦

=

𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧

𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍

111

𝑤𝑥

𝑤𝑦

1𝑤𝑧

𝑤𝑦

=

𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧

𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍

, solve for

𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍

M_RGB_to_XYZ =

𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧

𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍

Chromatic adaptation

• Estimating the appearance of a sample under a different illuminant– Convert between different

white points

• LMS color space– Response of the three types of

cones in the human eye• Long, medium, and short

wavelengths

• XYZ to LMS– Bradford transformation

matrix

• Chromatic adaptation– Adaptation matrix

CSE 291, Spring 2020 38

Color mapping (using chromaticities)

• Image 1 to image 2– Calculate transformation matrix from image 1 RGB to XYZ

(M_RGB1_to_XYZ1) and transformation matrix from image 2 RGB to XYZ (M_RGB2_to_XYZ2)• White points in XYZ are the same as in RGB

– If white points are the same• M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * M_RGB1_to_XYZ1

– Else (white points are different, include chromatic adaptation)• Bradford transformation matrix (M_XYZ_to_LMS)• Map white points to LMS and calculate adaptation matrix

(M_LMS1_to_LMS2)• Compose transformation matrices

– M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * inv(M_XYZ_to_LMS) * M_LMS1_to_LMS2 * M_XYZ_to_LMS * M_RGB1_to_XYZ1

CSE 291, Spring 2020 39

Note: luminance Y and luma Y’

• Luminance is calculated from linear RGB

– Y coordinate of XYZ

• Luma is calculated from nonlinear R’G’B’

• Example: sRGBY = 0.21263903 * R + 0.71516871 * G + 0.072192319 * B

Y’ = 0.21263903 * R’ + 0.71516871 * G’ + 0.072192319 * B’

(Coefficients are the middle row of M_RGB_to_XYZ)

CSE 291, Spring 2020 40

Remember

• Common video and image standards use nonlinear encoding and may use different chromaticities

• Chromaticities define gamuts that contain a percent of the visible colors– Some contain more than others

• Mapping pixel data between these standards requires mapping to the XYZ color space and possibly the LMS color space

• Luminance is different than luma

CSE 291, Spring 2020 41