Computational photography research at Stanford
Marc LevoyComputer Science DepartmentStanford University
January 2012
! Marc Levoy
Outline
! Light field photography
! FCam and the Stanford Frankencamera
! User interfaces for computational photography
! Ongoing projects• FrankenSLR• burst-mode photography• languages and architectures for programmable cameras• computational videography and cinematography• multi-bucket pixels
2
Light field photography using a handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brédif,Gene Duval, Mark Horowitz and Pat Hanrahan
(Proc. SIGGRAPH 2005and TR 2005-02)
! Marc Levoy
Light field photography[Ng SIGGRAPH 2005]
Prototype camera
4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125µ square-sided microlenses
! 2010 Marc Levoy
Digital refocusing
!
!
! 2010 Marc Levoy
Example of digital refocusing
! 2010 Marc Levoy
Refocusing portraits
! 2010 Marc Levoy
Light field photography
(Flash demo)
• trades off (excess) spatial resolution forability to refocus and adjust the perspective
• sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm ÷ 2.5µ pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixelor 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix
! Marc Levoy
Commercialization
Camera 2.0Marc Levoy
Computer Science DepartmentStanford University
! 2010 Marc Levoy
! facilitate research in experimental computational photography
! for students in computational photography courses worldwide
! proving ground for plugins and apps for future cameras
The Stanford Frankencameras
14
Frankencamera F2 Nokia N900 “F”
! 2010 Marc Levoy
CS 448A - Computational Photography
15
! 2010 Marc Levoy
CS 448 projectsM. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine
J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo! ! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park! ! ! High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que ! ! ! ! User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis ! ! ! ! ! ! Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving
16
! 2010 Marc Levoy
Frankencamera architecture
17
Image Sensor
Lens
Metadata
Actions
Flash
...
ApplicationProcessor
Con!gure1
Expose2
Readout3
Image Processing
StatisticsCollection
Imaging Processor
4
+
Devices
Requests
Images andStatistics
! 2010 Marc Levoy18
Sensor sensor;Flash flash;vector<Shot> burst(2);
burst[0].exposure = 1/200.;burst[1].exposure = 1/30.;
Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;burst[0].actions.insert(fire);
sensor.stream(burst);
while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame();}
Frankencamera software: the FCAM API
! 2010 Marc Levoy
Demonstration applications
19
• Canon 430EX (smaller flash) strobed continuously
• Canon 580EX (larger flash) fired once at end of exposure
! 2010 Marc Levoy
Fcam is being actively ported to...
! Android (at least by NVIDIA)
! iOS?
! Nokia/Microsoft?
! Android point-and-shoot camera?
20
! Marc Levoy
Prospects for computational photography
! the megapixel wars are over (and it’s about time)
! computational photography is the next battlegroundin the camera industry (it’s already starting)
21
! Marc Levoy
! this “headroom” permits more computation per pixel,or more frames per second, or less custom hardware
22
Premise: available computing power in cameras is rising faster than megapixels
0
3
6
9
12
2005 2006 2007 2008 2009 20100
17.5
35
52.5
70
billi
ons o
f pix
els /
seco
nd
meg
apix
els
Avg megapixels in new cameras, CAGR = 1.2NVIDIA GTX texture fill rate, CAGR = 1.8(CAGR for Moore’s law = 1.5)
! Marc Levoy
Interfaces for computational photography
! the megapixel wars are over (long overdue)
! computational photography is the next battlegroundin the camera industry (it’s already starting)
! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera
23
! Marc Levoy
Interfaces for computational photography
! the megapixel wars are over (long overdue)
! computational photography is the next battlegroundin the camera industry (it’s already starting)
! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera
! but aren’t cameras too complicated already?• won’t computational photography make this worse?
24
! Marc Levoy
Sony NEX-5! manual is 159 pages long
• and full of grammatical mistakes
! scene modes are not explained well• Sunset: Shoots the red of the sunset beautifully. (page 43)
! user interface is difficult to navigate
25
1234567
89
10111213
! Marc Levoy
ProHDR for the iPhone 4S
! most popular HDR capture method on the market
! app developers are making photography easy again!26
" 1 " " 2 "
27
Free
Free
28
Free
29
30
Free
circa 500,000 downloads in
2011
31
iPhone 4, single HD video frame
32
SynthCam, align & average ~30 frames
SNR increases as sqrt(# of frames)
SynthCam
Canon 5DII, 1/400 sec
Canon 5DII, average of 15 shots of 1/6 sec
(Marc Levoy)
(Marc Levoy)
! 2010 Marc Levoy
Removing foreground objectsby translating the camera
• align the shots• match histograms• apply median filter
! 2010 Marc Levoy
Ongoing projects
! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)
39
Micron MT9001• cell phone quality• $150
Cypress LUPA 4000• DSLR quality• $1500• arbitrary ROIs and non-destructive readout
! 2010 Marc Levoy
Ongoing projects
! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)
! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)
40
! 2010 Marc Levoy
Burst-mode photography
41
! 2010 Marc Levoy
Ongoing projects
! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)
! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)
! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision
42
! 2010 Marc Levoy
Ongoing projects
! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)
! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)
! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision
! computational videography & cinematography• stabilization, stereo, view interpolation, free-viewpoint video
43
! Marc Levoy
3D video stabilization[Agarwala 2011]
44 (http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/demo.mp4)
! 2009 Marc Levoy
Camera 3.0:multi-bucket pixels
! a CMOS pixel architecturewith multiple accumulation nodes per pixel
! can flip between TX1 and TX2 very fast• all pixels flip at once
45
storage nodes
photodetector
TX2TX1
transfer gates
! 2009 Marc Levoy
Flash/noflash photography[Agrawal SIGGRAPH 2005]
46
TX2TX1put flash exposure
in middle of ambient exposure
another app:HDR capture
(with no differences in object motion
between exposures)
!"#$%&'()*$+,-'.+"/
High Dynamic Range Photography
47
Short exposure Long exposure HDR Image
!"#$%&'()*$+,-'.+"/
Motion artifacts in HDR
48
!"#$%&'()*$+,-'.+"/
HDR Photography using 4-bucket pixels
49
Images with different exposures tightly interleaved
!"#$%&'()*$+,-'.+"/
4-Bucket Pixel Layout (3D View)
50
http://graphics.stanford.edu/projects/camera-2.0/
Image Sensor
Lens
Metadata
Actions
Flash
...
ApplicationProcessor
Con!gure1
Expose2
Readout3
Image Processing
StatisticsCollection
Imaging Processor
4
+
Devices
Requests
Images andStatistics
Sensor sensor;Shot low, med, high;
low.exposure = 1/80.;med.exposure = 1/20.;high.exposure = 1/5.;
sensor.capture(low);sensor.capture(med);sensor.capture(high);
Frame frames[3];frames[0] = sensor.getFrame();frames[1] = sensor.getFrame();frames[2] = sensor.getFrame();
fused = mergeHDR(frames);
Top Related