•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

●

●

●

●

•

• sparse matrix

•

•

•

•

•

•

•

• →

→

•

•

•

•

•

•

4.7 mm2

canon.com

1360 mm24.7 mm2

Larger aperture Long exposure Flash

How to collect more light? Capture multiple photos

video denoising[Kokaram, 1993]

[Dabov et al., 2007][Liu et al., 2014]

exposure bracketing[Debevec et al., 2007][Gallo and Sen 2016]

image stacking

weatherandsky.com

1 frame

16 frames

[Fruchter and Hook, 2002]

● burst with constant exposure○ more robust merge○ underexposed up to 8x

● reference image○ physically consistent fallback

● raw images○ merge in raw too

● burst with constant exposure○ more robust merge○ underexposed up to 8x

● burst with constant exposure○ more robust merge○ underexposed up to 8x

● reference image○ physically consistent fallback

Our approach

exposure bracketing● higher SNR● challenging merge

● HDR capture as noise reduction [Hasinoff et al. 2010] [Zhang et al. 2010]

Underexposure for HDR

single underexposed shot● low SNR

underexposed burst● moderate SNR● more robust merge ...

...

12 MPix, 10-bit

System overview

customautoexposure

processing

align & merge

12 MPix, 14-bit

zeroshutterlagcapture

finish

12 MPixring

buffer

Autoexposure and tonemapping

Example-based autoexposure

AE database~5,000 scenes

scenedescriptor

AE result

scenedescriptor

dual AE result - controls capture and rendition

shortexposure

longexposure

AE result

raw input

● synthetic exposures from AE○ single merged input image○ digital gains

● automatically set tuning parameters

Local tonemapping

pyramid blending for HDR tonemapping[Mertens et al., 2007]

[Paris et al. 2011][Aubry et al. 2014]

Align and merge

Burst alignment

reference frame

R+Gr+Gb+B

input burst

local alignmentto reference

reference frame

alternate frame

raw

120x90 (or 60x45)12 MPix3 MPix

Coarse to fine alignment

4x

4x

● 4 pyramid levels● upsample with multiple hypotheses [Tao et al., 2012]

metric search range subpixel?

L1

L2 9x9

3x3 no

yes

2x

Example alignment

aligned to reference

reference frame

Tiled Fourier-based merge● divide into 16x16 or 32x32 tiles

○ 50% overlap - every pixel covered by 4 tiles● merge in Fourier domain

...

reference tile

aligned alternate tiles

merged tile

Robust per-frequency merge

robust pairwise merge

reference frame

aligned average

Finish

Finish pipeline

demosaic,white balance

chroma denoise

lens shading, color correction

local tonemapping

global tonemapping

sharpen

suppress aberrations

color tuning YUV→U'V' LUT

merged raw

final result

12 MPix, 14-bit

12 MPix, 8-bit

auto exposure

luma

chroma

Results

HDR+ Off

HDR+ On

HDR+ Off

HDR+ On

Released Nov. 2018

Siggraph Asia 2019

HDR+ Night Sight

Night Sight● Dedicated mode

○ positive shutter lag○ heavier processing

● More light○ bigger bursts○ much longer exposures

● Motion metering● Stronger merge

○ more aggressive○ super-resolution merge

● ML-based AWB● Night "look"

○ brighter AE○ more HDR

● less motion?○ longer exposure○ better SNR

Measure scene motion in real time, to choose the exposure time

Motion metering

motion metering off8.3 ms, ISO 346

motion metering on59 ms, ISO 53

● more motion?○ shorter exposure○ less motion blur

● less motion?○ longer exposure○ better SNR

Measure scene motion in real time, to choose the exposure time

Motion metering

● more motion?○ shorter exposure○ less motion blur

motion metering off17 ms, ISO 183

motion metering on6 ms, ISO 512

Motion metering - "tripod detection"

● When camera is very stable, allow even longer exposures● Combine signals from gyro (camera motion) and images

(camera+scene motion)○ Discount motion due to shutter tap

● For fixed time budget, prefer fewer frames○ Less read noise start of burst

77

Without tripod detection 15 frames x 333ms

With tripod detection 6 frames x 1000ms

78

Without tripod detection 15 frames x 333ms

With tripod detection 6 frames x 1000ms

Challenges for Night Sight merge

object motion camera motion low SNR

Merge increases →High noise, motion robust Low noise, motion blur

80

Motion-Adaptive Burst Merging

Modifying the merge strength

HDR+ Increased merge

HDR+Temporal strength = 16

Increased mergeTemporal strength = 200

Ours: Tile-wise merge using mismatch maps

Modified merge: spatially varying merge strength

Super Res Zoom on Pixel 3

Siggraph 2019

White balance gains are applied to make “white” appear white

Captured image

Predicted “illumination”

Predicted imageInverse rgb gains

=/

White balance is an ill-posed problem

Learning based white balance: "Fast Fourier Color

Constancy", Barron & Tsai, 2017

Extended to low-light scenes

Challenges of Color Constancy in low light

Noisier images

➡ Train on real images that have noise

➡ New training data set for low light (3,500 images)

Various illuminants:Highly colorful illuminants result in color channels with practically zero signal (“missing channels”)

➡ New error metric: Anisotropic Reproduction Error (ARE)

Auto white balance results

Left: camera default, Right: ours

Baseline system(HDR+, Hasinoff et

al, 2016)

13 frames,exposure time:

1/15 s

Adding new tone mapping to

baseline

Results are noisy

Adding motion metering

Exposure time 1/3 s

Noise and detail improve

Adding motion-robust

merge

Noise and detail improve further

Adding motion-robust

merge

Noise and detail improve further

Adding low-light auto white

balance

Color improves

Side by side comparison

Previously described result (HDR+, Hasinoff et al 2016) Our result

Results across a variety of scenes: indoor

Previously described result (HDR+, Hasinoff et al 2016) Our result

Results across a variety of scenes: outdoor

Previously described result (HDR+, Hasinoff et al 2016) Our result

“Learning to see in the dark”, Chen et al, 2018

End to end network

Training set is pairs on short and long exposures

Comparison to “Learning to See in the Dark” (Chen et al, 2018)

Chen et al 2018 Our result

Comparison to “Learning to See in the Dark” (Chen et al, 2018)

Chen et al 2018 Our result

Astrophotography

Astrophotography

Extending exposure time

● Up to 4 minutes total exposure: ○ Capture even more light to improve signal to noise○ Extend the capture time per frame

● Detect when the camera is static using gyro signals● Detect when the scene is static

● Traditional: extend the exposure time● Computational:

● Burst photography● Motion metering● Motion-adaptive merge● Tone mapping for nighttime scenes● Device stability detection● “Warm pixel” removal

● Machine learning:● Learning based white balance● Semantically aware image processing of the skies

● Implemented on a mobile device

In Summary,To Capture Low Light Scenes:

2019 Smartphone Camera of the year award“If you shoot Night Sight - even during daylight hours - you'll be rewarded with some of the best detail retention and balanced noise reduction we've seen from a smartphone... A new astrophotography mode is not just cool but inspiring, and also benefits any nighttime scene where longer exposures can be used. The combination of super-res zoom and a new telephoto module make 'zoomed in' photos better than many peers.” dpreview.com

HDR

• Mann, Picard “On Being ‘Undigital’ with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures”, IS&T 1995

• Debevec, Malik, “Recovering High Dynamic Range Radiance Maps from Photographs”, SIGGRAPH 1997

• Robertson, Borman, Stevenson, “Estimation-Theoretic approach to Dynamic Range Improvement Using Multiple Exposures”, Journal of Electronic Imaging 2003

• Mitsunaga, Nayar, “Radiometric self Calibration”, CVPR 1999

• Reinhard, Ward, Pattanaik, Debevec (2005). High dynamic range imaging: acquisition, display, and image-based lighting. Elsevier/Morgan Kaufmann

• Fattal, Lischinski, Werman, “Gradient Domain High Dynamic Range Compression”, ACM SIGGRAPH 2002

• Durand, Dorsey, “Fast Bilateral Filtering for the Display of High Dynamic Range Images”, ACM SIGGRAPH 2002

Super-resolution

• Baker, Kanade, Limits on super-resolution and how to break them“ IEEE Transactions on Pattern Analysis and Machine Intelligence 24(9), 1167–1183 (2002)

• Ben-Ezra, Lin, Wilburn, Zhang,, “Penrose pixels for super-resolution” EEE Transactions on Pattern Analysis and Machine Intelligence 33(7), 1370–1383 (2011)

• Ben-Ezra, Zomet, Nayar, “Jitter Camera: High Resolution Video from a Low Resolution Detector”, CVPR 2004

• Ben-Ezra, Zomet, Nayar, “Video super-resolution using controlled subpixel detector shifts” IEEE Trans. PAMI27(6), 977–987 (2005)

• Elad, Feuer, “Restoration of single super-resolution image from several blurred, noisy and down-sampled measured images” IEEE Trans. Im. Proc. 6(12), (1997)

Other

• Hasinoff, Sharlet, Geiss, Adams, Barron, Kainz, Chen, Levoy “Burst photography for high dynamic range and low-light imaging on mobile cameras”, SIGGRAPH Asia 2016

• Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, Marc Levoy, “Handheld mobile photography in very low light.” SIGGRAPH

Asia 2019.

• Liba, Orly, Longqi Cai, Yun-Ta Tsai, Elad Eban, Yair Movshovitz-Attias, Yael Pritch, Huizhong Chen, and Jonathan T. Barron. "Sky Optimization: Semantically Aware Image Processing of Skies in Low-Light Photography." In Proceedings of the

IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 526-527. 2020.

• Chen, et al. "Learning to see in the dark." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

• Wronski B, Garcia-Dorado I, Ernst M, Kelly D, Krainin M, Liang CK, Levoy M, Milanfar P . “Handheld multi-frame super-resolution”, SIGGRAPH 2019.

• Ben-Ezra and Nayar, "Motion Deblurring using Hybrid Imaging”, CVPR 2003

• Yuan, Sun, Quan, Shum, “Image Deblurring with Blurred/Noisy Image Pairs”, ACM SIGGRAPH 2007

• Hasinoff, Kutulakos, “Confocal Stereo”, ECCV 2006

• Hasinoff, Kutulakos, “A Layer-Based Restoration Framework for Variable-Aperture Photography”, ICCV 2007

• Raskar, Tan, Feris, Yu, Turk, “Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging”, ACM SIGGRAPH 2004

• Pettschnigg, Agrawala, Hoppe, Szeliski, Cohen, Toyama, “Digital Photography with Flash and No-Flash Image Pairs”, ACM SIGGRAPH 2004

• Eisemann, Durand, “Flash Photography Enhancement via Intrinsic Relighting”, ACM SIGGRAPH 2004