Computational Photography

Matthias Zwicker University of Bern

Fall 2012

Today Course organization

Course overview

Image formation

Course organization Instructor

Matthias Zwicker (zwicker@iam.unibe.ch)

Teaching Assistant

Daniel Donatsch (donatsch@iam.unibe.ch)

mailto:zwicker@iam.unibe.chmailto:knaus@iam.unibe.ch

Course organization Lecture

Mondays, 14:00-16:00

Engehaldenstrasse 8, Room 3

Exercises

Mondays, 16:00-17:00

Engehaldenstrasse 8, Room 3

Class web page Class overview

http://www.cgg.unibe.ch/teaching/computational-photography

http://www.cgg.unibe.ch/teaching/computational-photography

ILIAS Use your campus account to log in

Join course Magazin Weitere Institutionen; Weiterbildungen und Studiengnge BeNeFri Joint Master in Computer Science HS2012 2012HS: 31051 Computational Photography

Lecture slides

Exercise description & material

Additional reading material

Forum

Any questions and discussions related to class material and exercises

https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=1https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=711https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447636https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447636https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447639https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447650

Exercises 6 assignments

Programming projects

Matlab Available in ExWi pool

Exercises on paper

Exercises Final grade: 40% exercises, 60% final exam To qualify for final exam: need 70% of

exercise score Late penalty

50% of original score Exceptions for military service, illness

Collaboration Discussion among students is encouraged Each student must write up and turn in his/her

own solution If we detect copied material, you will need to talk

to us and explain your material in person; if we are not satisfied, you will not get credit

Final exam Written, 105 minutes

Bring two A4 sheets (4 pages) of hand written notes

Relevant material: slides and exercises

Wikipedia links not part of class material, but may be useful to better understand concepts discussed in class

Date: February 2013

Prerequisites Familiarity with

Linear algebra (matrix calculations, linear systems of equations, least squares problems)

Programming experience

Today Course organization

Course overview

Image formation

Computational photography Topics of this class

Role of computation, algorithms in digital photography today

Algorithms to extend and improve capabilities of digital photography in the future

Photography Traditionally

Measuring light

Optics focuses light on sensor

Sensor records image

Sensors

Digital Film

http://en.wikipedia.org/wiki/Single-lens_reflex_camera http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera

http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera

Computational photography More than digital photography

Arbitrary computation between light measurement and final image Light measured on sensor is not final image Computation enhances and extends capabilities of

digital photography Two types of computation

1. Post-process after traditional imaging 2. Design of new camera devices that require

computation to form an image Overview of recent research

http://en.wikipedia.org/wiki/Computational_photography

http://en.wikipedia.org/wiki/Computational_photography

Removing imaging artifacts Denoising & deblurring

http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

Blurry + Output Noisy Algorithm

http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

Removing imaging artifacts High dynamic range images & tone mapping

Image manipulation Panoramas

http://en.wikipedia.org/wiki/Image_stitching

http://en.wikipedia.org/wiki/Image_stitching

Computational optics

Coded aperture

Captured image, slightly blurry everywhere

Computational optics

Recovered depth

Refocused image Sharp foreground, blurry background

Focus of class Fun with digital photography and

computer programming

Algorithms and computational techniques with potential applications in the consumer domain

Mostly software, less hardware

Recent research

What you will learn Basic understanding of photography, light,

and color Practical experience with implementation of

algorithms for image processing & computational photography

Cool and creative applications of mathematical tools Fourier transforms Linear and non linear filtering Optimization techniques (least squares, iteratively

re-weighted least squares, graph cuts) Probabilistic models

Many applications beyond processing images!

Related areas, not covered Image processing for scientific applications

Physics, biology, etc.

Optics, lens design

Photosensors, sensor design

Computational imaging

Tomography, radar, microscopy

3D imaging

Using photo processing tools, e.g. Photoshop

Artistical aspects of photography

Syllabus 1. Introduction, image formation 2. Color & color processing 3. Dynamic range & contrast 4. Sampling, reconstruction, & the frequency domain 5. Image restoration: denoising & deblurring 6. Image manipulation using optimization 7. Gradient domain image manipulation 8. Warping & morphing 9. Panoramas 10. Automatic alignment 11. Probabilistic image models 12. Light fields 13. Capturing light transport

http://www.cgg.unibe.ch/teaching/computational-photography

http://www.cgg.unibe.ch/teaching/computational-photography

Cameras, image artifacts

Image formation

Color Color perception, color spaces, color

measurement, color processing

Dynamic range & contrast HDR imaging

http://en.wikipedia.org/wiki/High_dynamic_range_imaging http://en.wikipedia.org/wiki/Tone_mapping

http://en.wikipedia.org/wiki/High_dynamic_range_imaginghttp://en.wikipedia.org/wiki/Tone_mapping

Sampling, reconstruction Sampling artifacts

Frequency domain analysis

Spatial Domain Frequency Domain

Image restoration Denoising & deblurring

Blurry input Deblurred output

Estimated blur kernel (scaled) http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

Image manipulation using optimization Photomontage, matting, colorization

http://grail.cs.washington.edu/projects/photomontage/

http://www.cs.huji.ac.il/~yweiss/Colorization/

http://grail.cs.washington.edu/projects/digital-matting/image-matting/

http://grail.cs.washington.edu/projects/photomontage/http://www.cs.huji.ac.il/~yweiss/Colorization/http://grail.cs.washington.edu/projects/digital-matting/image-matting/

Gradient domain manipulation Poisson equation

http://portal.acm.org/citation.cfm?id=882269

http://portal.acm.org/citation.cfm?id=882269

Warping & morphing

Panoramas Automatic alignment, stitching

http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/

http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/

Probabilistic models Faces, textures

http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

Beyond 2D images

Light fields

http://www-graphics.stanford.edu/papers/fourierphoto/

http://www-graphics.stanford.edu/papers/fourierphoto/http://www-graphics.stanford.edu/papers/fourierphoto/http://www-graphics.stanford.edu/papers/fourierphoto/

Capturing light transport Dual photography

http://www-graphics.stanford.edu/papers/dual_photography/

http://www-graphics.stanford.edu/papers/dual_photography/http://www-graphics.stanford.edu/papers/dual_photography/http://www-graphics.stanford.edu/papers/dual_photography/

Today Course organization

Course overview

Image formation

Models of light

Question Why is there no image on a white piece of

paper?

Question Why is there no image on a white piece of

paper?

Receives all light rays

Images from all viewpoints

Need to select light rays for specifice image, viewpoint

How?