Computational Photography and Video - ETH Zurich€¦ · What is computational photography •...

ComputationalPhotographyandVideo

Prof.MarcPollefeys

Dr.GabrielBrostow

Todaysschedule

IntroductionofComputationalPhotography Coursefacts Syllabus DigitalPhotography

Whatiscomputationalphotography

Convergenceofimageprocessing,computervision,computergraphicsandphotography

Digitalphotography: Simplyreplacestraditionalsensorsandrecordingbydigitaltechnology

Involvesonlysimpleimageprocessing Computationalphotography

Moreelaborateimagemanipulation,morecomputation

Newtypesofmedia(panorama,3D,etc.) Cameradesignthattakecomputationintoaccount

(severalslidesinspired/borrowedfromFredoDurand,MIT)

Tonemapping

Before After

DurandandDorsey.Siggraph02

http://www-users.mat.uni.torun.pl/~olka/http://www.loc.gov/exhibits/empire/

Flash/NoFlash

Petschniggetal.Siggraph04

Photomontage

Agarwalaetal.Siggraph04

Panoramicimages

BrownandLoweICCV03

Defocusmatting

McGuireetal.Siggraph05

Videotextures

Schoedletal.Siggraph00

Motionmagnification

Liuetal.Siggraph05

Todaysschedule


Administrivia

Staff Prof.MarcPollefeys Dr.GabrielBrostow RolandAngst

Timeandlocation: Lectures:Wednesday1315inCABH57 Exercises:Thursday1112inCABH56

Webpage:http://www.inf.ethz.ch/personal/pomarc/courses/CompPhoto/

http://www.inf.ethz.ch/personal/pomarc/courses/CompPhoto/

Courseorganization

Lectures Exercises

Firstafewassignments Laterprojectandpaperpresentations

Smallclassproject Individualorsmallgroups

Gradingpolicy

50%assignments 10%paperpresentation 40%classproject(report+presentation) Bonusforparticipation

Noseparateexam

Textbook

Notextbookrequired Slidesavailableoncoursewebpage Lotmoreresourcesonline

Interestingreference:ComputationalPhotography:MasteringNewTechniquesforLenses,Lighting,andSensors.RaskarandTumblin,toappearsoon,AKPeters.

Todaysschedule


Topics

Imageformation,Imagesensor,Optics Pixelresolution,Exposure,Aperture,Focus,DynamicRange Color,whitebalance,Bayerpattern,demosaicking, Motionblur,shutter,deblurring Dynamicrange,HDRimaging,tonemapping,bilateralfiltering Imagepyramids,opticalflow,gradients Mattingandcompositing,graphcuts Warpingandmorphing,panoramas Texturesynthesis Illumination,flash/noflash,depthedges Codedaperture,defocus Videotextures,timelapse,videosummarization Lightfield imaging

Todaysschedule


Overview

Lensandviewpointdetermineperspective Apertureandshutterspeeddetermineexposure Apertureandothereffectsdeterminedepthoffield Sensorrecordsimage

(thisandfollowingslidesborrowedfromFredoDurand,MIT)

Reference

http://en.wikipedia.org/wiki/Lens_(optics)

Theslidesuseillustrationsfromthesebooks

http://en.wikipedia.org/wiki/Lens_(optics)

Morereferences

Plan

Pinholeoptics Lenses Exposure

Whyistherenoimageonawhitepieceofpaper?

Itreceiveslightfromalldirections

FromPhotography,Londonetal.

Pinhole


Focallength

pinhole

Film/sensor scene

f

s

Focallength:pinholeoptics

Whathappenswhenthefocallengthisdoubled? Projectedobjectsize Amountoflightgathered

pinhole

Film/sensor scene

f d

s

2f

isdoubledisdividedby4

Questions?

Pinholesize?


Diffractionlimit

Optimalsizeforvisiblelight:sqrt(f)/28(inmillimiters)wherefisfocallength

FromWandell

Problemwithpinhole?

Notenoughlight! Diffractionlimitssharpness

Solution:refraction!


Lenses

gathermorelight!

Butneedtobefocused


Thinlensoptics

Simplificationofgeometricalopticsforwellbehavedlenses Allparallelraysconvergetoonepointonaplanelocatedatthe

focallengthf

Allraysgoingthroughthecenterarenotdeviated Hencesameperspectiveaspinhole

f

Howtotracerays

Startbyraysthroughthecenter

Howtotracerays

Startbyraysthroughthecenter Choosefocallength,traceparallels

f

Howtotracerays Startbyraysthroughthecenter Choosefocallength,traceparallels Yougetthefocusplaneforagivensceneplane

Allrayscomingfrompointsonaplaneparalleltothelensarefocusedonanotherplaneparalleltothelens

f

Focusing

Tofocuscloserthaninfinity Movethesensor/filmfurther thanthefocallength

f

Thinlensformula

fDD

Thinlensformula

fDD

Similartriangleseverywhere!

Thinlensformula

fDD


y

y

y/y=D/D

Thinlensformula

fDD


y

y

y/y=D/D

y/y=(Df)/D

Thinlensformula

fDD

1D D

1 1f

+ =

Minimumfocusingdistance

Bysymmetry,anobjectatthefocallengthrequiresthefilmtobeatinfinity.

sensor

Raysfrominfinity

Raysfromobjectatf

Extensionstubes

Allowustoputsensorfartherfocuscloser

Fieldofview&focusing

Whathappenstothefieldofviewwhenonefocusescloser? It'sreduced

sensorfocused at infinity

sensorfocused close

Questions?

http://www.pinhole.cz/en/pinholecameras/dirkon_01.html

Focallengthinpractice

24mm

50mm

135mm

Perspectivevs.viewpoint

Telephotomakesiteasiertoselectbackground(asmallchangeinviewpointisabigchangeinbackground).


Movescameraasyouzoomin HitchcockVertigoeffect


Portrait:distortionwithwideangle Why?

Wideangle Standard Telephoto

Focallength&sensor Whathappenswhenthefilmishalfthesize? Application:

Realfilmis36x24mm Onthe20D,thesensoris22.5x15.0mm Conversionfactoronthe20D? OntheSD500,itis1/1.8"(7.18x5.32mm) Whatisthe7.723.1mmzoomontheSD500?

pinhole

Film/sensor

scene

f d

s

Sensorsize

Similartocropping

source:canonredbook

http://www.photozone.de/3Technology/digital_1.htm

Recap

Pinholeisthesimplestmodelofimageformation Lensesgathermorelight

Butgetonlyoneplanefocused Focusbymovingsensor/film Cannotfocusinfinitelyclose

Focallengthdeterminesfieldofview Fromwideangletotelephoto Dependsonsensorsize

Moreinthelenslecture

Questions?

Exposure

Gettherightamountoflighttosensor/film Twomainparameters:

Shutterspeed Aperture(areaoflens)

Shutterspeed

Controlshowlongthefilm/sensorisexposed Prettymuchlineareffectonexposure Usuallyinfractionofasecond:

1/30,1/60,1/125,1/250,1/500 Getthepattern?

Onanormallens,normalhumanscanhandholddownto1/60 Ingeneral,theruleofthumbsaysthatthelimitistheinverseoffocallength,e.g.1/500fora500mm

Maineffectofshutterspeed

Motionblur


Effectofshutterspeed

Freezingmotion

1/125 1/250 1/500 1/1000

Walking people Running people Car Fast train

Shutter

Varioustechnologies Goal:achieveuniformexposureacrossimage

FromCameraTechnology,Goldberg

Flashsynchspeed?

Fastestshutterspeedforwhichtheshutteropenscompletelyatsomeinstant.

Forfasterspeeds,itopensandclosesatthesametimeandexposesaslit.

Modernhighspeedflashsynchusesmultipleflashbursts


Aperture

Diameterofthelensopening(controlledbydiaphragm) Expressedasafractionoffocallength,infnumber

f/2.0ona50mmmeansthattheapertureis25mm f/2.0ona100mmmeansthattheapertureis50mm

Disconcerting:smallfnumber=bigaperture Whathappenstotheareaoftheaperturewhengoingfromf/2.0to

f/4.0? Typicalfnumbersare

f/2.0,f/2.8,f/4,f/5.6,f/8,f/11,f/16,f/22,f/32 Seethepattern?

Maineffectofaperture

Depthoffield


Depthoffield

lenssensor

Pointinfocus

Objectwithtexture

Depthoffield

lenssensor

Pointinfocus

Objectwithtexture

Weallowforsometolerance

lenssensor

Pointinfocus

Objectwithtexture

Maxacceptablecircleofconfusion

Depthoffield

Depthoffocus

Depthoffield

Whathappenswhenweclosetheaperturebytwostop? Aperturediameterisdividedbytwo Depthoffieldisdoubled

lenssensor

Pointinfocus

Objectwithtexture

Diaphragm

Depthoffield


Depthoffield&focusingdistance

Whathappenswhenwedividefocusingdistancebytwo? Similartriangles=>dividedbytwoaswell

lenssensor

Pointinfocus

Halfdepthoffield

Halfdepthoffield

Depthoffield&focusingdistance

Whathappenswhenwedividefocusingdistancebytwo? Similartriangles=>dividedbytwoaswell


SLRviewfinder&aperture

Bydefault,anSLRalwaysshowsyouthebiggestaperture

Brighterimage Shallowdepthoffieldhelpjudgefocus Depthoffieldpreviewbutton:

Stopsdowntotheapertureyouhavechosen Darkerimage Largerdepthoffield

Questions?

Exposure

Twomainparameters: Aperture(infstop) Shutterspeed(infractionofasecond)

ReciprocityThesameexposureisobtainedwitanexposuretwiceaslongandanapertureareahalfasbig

Hencesquarerootoftwoprogressionoffstopsvs.poweroftwoprogressionofshutterspeed

Reciprocitycanfailforverylongexposures


Reciprocity

Assumeweknowhowmuchlightweneed Wehavethechoiceofaninfinityofshutterspeed/aperturepairs

Whatwillguideourchoiceofashutterspeed? Freezemotionvs.motionblur,camerashake

Whatwillguideourchoiceofanaperture? Depthoffield,diffractionlimit

Oftenwemustcompromise Openmoretoenablefasterspeed(butshallowDoF)

Questions?

Metering

Photosensitivesensorsmeasuresceneluminance UsuallyTTL(throughthelens) Simpleversion:centerweightedaverage

Assumption?Failurecases? Usuallyassumesthatasceneis18%gray Problemwithdarkandbrightscenes

Metering

Centeredaverage

Spot

Smartmetering Nikon3Dmatrix Canonevaluative

Incident Measureincominglight

Fromtheluminouslandscape

Nextslide

ChoiceonNikon

http://www.mir.com.my//

Nikon3DColorMatrix

http://www.mir.com.my/rb/photography/hardwares/classics/NikonF5/metering/

Learningfromdatabaseof30,000photos

Multiplecaptors(segments) Exposuredependson

Brightnessfromeachsegments Color Contrast Distance Focus(whereisthesubject)

Exposure&metering

Thecamerameteringsystemmeasureshowbrightthesceneis InApertureprioritymode,thephotographersetstheaperture,

thecamerasetstheshutterspeed

InShutterspeedprioritymode,thephotographerssetstheshutterspeedandthecameradeducestheaperture

Inbothcases,reciprocityisexploited InProgrammode,thecameradecidesbothexposureand

shutterspeed(middlevaluemoreorless)

InManual,theuserdecideseverything(butcangetfeedback)

Prosandconsofvariousmodes

Aperturepriority Directdepthoffieldcontrol Cons:canrequireimpossibleshutterspeed(e.g.withf/1.4fora

brightscene) Shutterspeedpriority

Directmotionblurcontrol Cons:canrequireimpossibleaperture(e.g.whenrequestinga

1/1000speedforadarkscene) Notethatapertureissomewhatmorerestricted

Program Almostnocontrol,butnoneedforneurons

Manual Fullcontrol,buttakesmoretimeandthinking

Recap:Metering

Measurescenebrightness Someadvancedmodesthattakemultiplesourcesofinformation

Stillanopenproblem

Questions?

Sensitivity(ISO)

Thirdvariableforexposure Lineareffect(200ISOneedshalfthelightas100ISO) Filmphotography:tradesensitivityforgrain

Digitalphotography:tradesensitivityfornoise

From

dpreview.com

Computational Photography and Video Todays scheduleWhat is computational photographyTone mappingFlash/No-FlashPhotomontagePanoramic imagesDefocus mattingSlide Number 9Video texturesMotion magnificationTodays scheduleAdministriviaCourse organizationGrading policyTextbookTodays scheduleTopics Todays scheduleOverviewReferenceMore referencesPlanWhy is there no image on a white piece of paper?PinholeFocal lengthFocal length: pinhole opticsQuestions?Pinhole size? Diffraction limitProblem with pinhole?Solution: refraction!Lenses Thin lens opticsHow to trace raysHow to trace raysHow to trace raysFocusingThin lens formulaThin lens formulaThin lens formulaThin lens formulaThin lens formulaMinimum focusing distanceExtensions tubesField of view & focusingQuestions?Focal length in practicePerspective vs. viewpointPerspective vs. viewpointPerspective vs. viewpointFocal length & sensorSensor sizehttp://www.photozone.de/3Technology/digital_1.htmRecapQuestions?ExposureShutter speedMain effect of shutter speedEffect of shutter speedShutter Slide Number 63Flash synch speed?ApertureMain effect of apertureDepth of fieldDepth of fieldSlide Number 69Depth of fieldDepth of fieldDepth of field & focusing distanceDepth of field & focusing distanceSLR viewfinder & apertureQuestions?ExposureReciprocitySlide Number 78Slide Number 79Slide Number 80Questions?MeteringSlide Number 83MeteringNikon 3D Color MatrixExposure & meteringPros and cons of various modesRecap: MeteringQuestions?Sensitivity (ISO)

Computational Photography and Video - ETH Zurich€¦ · What is computational photography •...

Documents

Transcript of Computational Photography and Video - ETH Zurich€¦ · What is computational photography •...