soima11-digital-imagingbesser.tsoa.nyu.edu/howard/Talks/11soima-digital-imaging.pdf · • MPEG 2...
Transcript of soima11-digital-imagingbesser.tsoa.nyu.edu/howard/Talks/11soima-digital-imaging.pdf · • MPEG 2...
7/13/11
1
DigitalImaging—Compression&Sampling
Averyshortintroduc?on
HowardBesserNYUMovingImageArchiving&Preserva?onhHp://besser.tsoa.nyu.edu/howard/Talks/hHp://www.nyu.edu/?sch/preserva?on/
1Besser‐SOIMA2011‐Compression&Sampling
WhatisDigitalEncoding?‐
• WhyDigitalisdifferentthanAnalog– DigitalAudioasexample
• CompressionforAudioandS?llImages– FramesandFrameRate,BitRates,ChromaSubsampling
• Theprocessofdigitalphotography• DigitalCompressioncoding
• Wrappers
• (notdiscussingcopyphotography)2Besser‐SOIMA2011‐Compression&Sampling
Besser‐ICCROMRIO,6/8/073
SamplingSoundWaveoverTime
Besser‐ICCROMRIO,6/8/074
Frequency : every for seconds
DigitalSamplePointsonAnalogWave
DigitalAudioSamplingconvertfromcon?nuoustodiscrete
Besser‐SOIMA2011‐Compression&Sampling 5 Besser‐ICCROMRIO,6/8/076
Digi?za?on,samplingevery4seconds,1bit
Sample Time
Voltage Signal (in base 2)
0 4.3 1 4 6 1 8 6.5 1 12 1.5 0 16 4.2 1 20 4 1
Resolution
7/13/11
2
Besser‐ICCROMRIO,6/8/077
AudioDigi?za?onsebngsdependonseveralfactors
• Thequalityontherecording,andthequalityyouneedforyourusers– Whenyouneedtocapturesoundsthatarelesscomplex(shorterandlesscomplexsounds),youcanloweryoursignalresolu?onand/orsamplingrate(thehumanvoiceonaninterviewrequireslesssignalresolu?on/fidelitythanmostmusic
• Digi?za?onsebngsforspeecharerela?velylowunlessyouintendtocapturebackgroundsoundsaswell(backgroundmusicwhispers,gunshots,…)
Besser‐ICCROMRIO,6/8/078
Resolu'on
• Humanspeechcanbeeffec?velyreproducedatarateof5.5kHz(kilo‐Hertz)
• 5500samples/sec
• Mostnaturalworldsoundsandmediumfidelitymusiccanbereproducedat11kHzwithacceptablelossesoffidelity
• 11000samples/sec.
• highfidelitymusicatCDaudioqualitythesamplingratemustbeabsoluteminimumof44.1kHz,givingaplaybackrateof22kHz,whichisjustabovethelimitofhumanhearing,butmostprofessionalandacademicorganiza?onsrecommendaboutdoublethat(96kHz).Butfor44.1kHZ:
• 44100samples/sec.• 22000samples/secisjustabovethelimitofhumanhearing
Besser‐ICCROMRIO,6/8/079
Quan'za'on
• Thenumberofdis?nctsoundlevelsthatcanberepresentedisdeterminedbythenumberofbytesusedtostorethequan?za?onvalue.
• themostcommonquan?za?onstrategyuses16bits(2bytes).Ex:11111111
• 216=65536discretelevels
Audio:Pulse‐CodeModula?on(PCM)
• Samplingrate(numberofsamplespersecond)– 48kHz,96kHz
• Bit‐depth(rangeofpossiblevaluesforeachsample)– 16bit,24bit
Besser‐SOIMA2011‐Compression&Sampling 10
Besser‐ICCROMRIO,6/8/0711
Standards/BestPrac?cesforSoundDigi?za?on
• Aremanagedbyprofessionalcommuni?eswithcommoninterests– Interna?onalOralHistoryAssocia?onhHp://www.iohanet.org/resources/technical.html
– Interna?onalAssocia?onofSoundandAudio‐VisualArchives(IASAA)hHp://www.iasa‐web.org/audio‐preserva?on‐tc04
– SoundDirec,ons:BestPrac,cesforAudioPreserva,onh7p://www.dlib.indiana.edu/projects/sounddirec,ons/papersPresent/index.shtml
– UniversityLibraries,…
S?llImageEncoding
Besser‐SOIMA2011‐Compression&Sampling 12
7/13/11
3
FilmvsCCDArray
Besser‐SOIMA2011‐Compression&Sampling 13
CCDforcapture
Besser‐SOIMA2011‐Compression&Sampling 14
• fromhHp://micro.magnet.fsu.edu/primer/digitalimaging/concepts/fullframe.html
Pixels
Besser‐SOIMA2011‐Compression&Sampling 15
Pixels
Besser‐SOIMA2011‐Compression&Sampling 16
Resolu?on
• dpi,ppi,spi(scannervs.camera)
Besser‐SOIMA2011‐Compression&Sampling 17
Example:50‐800ppi
Besser‐SOIMA2011‐Compression&Sampling 18
7/13/11
4
Bit‐depth
• 8‐bit,24‐bit,36‐bit
Besser‐SOIMA2011‐Compression&Sampling 19
Bit‐depthexample
Besser‐SOIMA2011‐Compression&Sampling 20
Full‐colorspectrumvs8‐bits
Besser‐SOIMA2011‐Compression&Sampling 21
RGBColor
Besser‐SOIMA2011‐Compression&Sampling 22
Compressionforcapturingcon?nuous‐tonescenes
• JPEG/JFIF– Predic?vecoding– DCT‐
Besser‐SOIMA2011‐Compression&Sampling 23
Compressionforcapturingcon?nuous‐tonescenes
• JPEG/JFIF– Predic?vecoding– DCT‐
Besser‐SOIMA2011‐Compression&Sampling 24
7/13/11
5
ChromaSubSampling/DCTencoding
• Mostly,eyeismoresensi?vetoluminencethantochroma
• Divideimageintomatrix(8x8cellblocks;moreblocksforH.264[butalsoincludesub‐blocks])
• Separateluminence(Y)fromChroma(Cb,Cr)
• Encodeluminenceforeachindividualcell,butencodechromaforgroupsofcells(chromasubsampling)
Besser‐SOIMA2011‐Compression&Sampling 25
Encodemoredataforluminencethanchroma
• 4:4:4isfullysampled
• 4:2:2useshalfthechromasamplesoneachchromachannel
• 4:2:0uses¼thechromasamplesoneachchromachannel
‐ImagefromTonyGladvinGeorge
Besser‐SOIMA2011‐Compression&Sampling 26
DCTCodingar?facts
Besser‐SOIMA2011‐Compression&Sampling 27
Ar?factsbecomelessvisibleaswezoomout
Besser‐SOIMA2011‐Compression&Sampling 28
Ar?factsbecomelessvisibleaswezoomout
Besser‐SOIMA2011‐Compression&Sampling 29
FileFormats
• TIFF• JFIF(JPEG)• GIF• J2K• Raw• DNG(Adobe)
Besser‐SOIMA2011‐Compression&Sampling 30
7/13/11
6
Differentfileformats
Besser‐SOIMA2011‐Compression&Sampling 31
OtherAr?facts:Aliasing
• Aliasingduetohighfrequencyclearedupwithlow‐passfilteringbeforedigi?zing
‐hHp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/sab/report.htmlBesser‐SOIMA2011‐Compression&Sampling 32
CameraMetadata
• XMP(Adobe’seXtensibleMetadataPlaworm)• IPTC(Interna?onalPressTelecommunica?onsCouncil)– hHp://www.iptc.org/site/Photo_Metadata/Embedded_Metadata_Manifesto_%282011%29
• EXIF(ExchangableImageFileFormat)‐
Besser‐SOIMA2011‐Compression&Sampling 33
EXIFmetadataexample
Besser‐SOIMA2011‐Compression&Sampling 34
DigitalVideoCompression:Codecs
• theschemeusedforcompressingandencodingthematerial.Usuallyrequiresdecodingsoywareorhardwaretoplayit.– Hardwaredecodingindigitalvideotapeplayers– Usuallysoywaredecodinginfile‐baseddigitalvideo
• withwebbrowsers:mightbebuiltintobrowser(jpg)orrequirebrowserplug‐in(flv)
• Otherwise,requiresplayersoywareorhardwarecard(J2K)• Encodingisusuallyasymetrical
– Takeslotsof?me&resourcestoencode
– …butdecodingcanbedonein(near)real?meonstandardwkst
Besser‐SOIMA2011‐Compression&Sampling 35 Besser‐ICCROMRIO,6/8/0736
MovingImageDigitalFiles
• Eachframe(25‐30fps)isessen?allystoredsimilarlytoas?lldigitalimageframe(intermsofrepresenta?onandcompression)
• Compressedformatssaveonstoragespacebynotsavingeverysingleframe– Maystoreonlythesmalldifferencesfromthepreviousframe
– Maystoreperiodicfullframesandpredict/interpolatebtwnthestoredframes
7/13/11
7
DigitalVideoCompression
• Spa?alimagecompression(intraframe)• Temporalmo?oncompression(interframe)
• Andmostcompressionschemesgiveyouarangeofchoicesinhowheavilytocompress
Besser‐SOIMA2011‐Compression&Sampling 37
Differentcompressionlevels(CarlFleischhauer)
38Besser‐SOIMA2011‐Compression&Sampling
Temporalmo?oncompression(interframe)‐basics
Besser‐SOIMA2011‐Compression&Sampling 39
Interframecompression‐‐moredetails
• I‐Frame(keyframe)—encodedass?llimage• P‐Frame/Delta‐Frame—differencesfrompreviousframe
• B‐Frame—a“virtual”framethataveragesdifferencebtwnframesbeforeandayer
‐imagefromPeHeriAimonenBesser‐SOIMA2011‐Compression&Sampling 40
Besser‐ICCROMRIO,6/8/0741
Intra‐FrameCompression InterframeCompression
• I‐Frame(orkeyframe)isonlyframethatdoesn’tdependuponotherframestodisplayit,butittakesupthelargestamountofstorage/bandwidth– NeedI‐Frametorecoverfromproblems
• IntervalsbtwnI‐Frames(4,6,8,…,150)canbespecifiedinH.264orFLVformats
Besser‐SOIMA2011‐Compression&Sampling 42
7/13/11
8
EncodingAr?facts&RelatedIssues
• Ifevenonebitgoesbad,onecanlosesignalun?lnextkey‐frame(because,withinter‐framecompression,alltheframesbeforethenextkey‐frameonlycontainpar?alinforma?on)
• Browsingwillonlyfindkey‐frames
• Canseeablockingeffectofchromasubsamplingaroundthe?meoflumainterference‐
Besser‐SOIMA2011‐Compression&Sampling 43
CodingAr?facts
"JensHenneman”,PerformanceArtpiecebyFredArmisen)careofPeterOleksik.720x980HDvideo,compressedusingMPEG2codec(23.98fps@2mbit/s),withonepassCBR
Besser‐SOIMA2011‐Compression&Sampling 44
CodingAr?factsfollowcolorlightflash
Besser‐SOIMA2011‐Compression&Sampling 45
CodingAr?factsfollowcolorlightflash
Besser‐SOIMA2011‐Compression&Sampling 46
Tamiment‐35mmtelecinedtoDigibeta,thentranscodeduncompressedintocomputer.Then,filewascompressedusingMPEG2codec(3.7mbit/s)
Besser‐SOIMA2011‐Compression&Sampling 47
AspectRa?oissuesTerryGilliam’sBrazil,digi?zed4:3VHSusingaDV50codecs?ll
displayedwithbadaspectra?o
Besser‐SOIMA2011‐Compression&Sampling 48
7/13/11
9
Besser‐ICCROMRIO,6/8/0749
VideoDigi?za?on&Transfer Framesize&rate
• Framesize‐‐pixelarrayplusra?o(4:3,16:9)• Framerate(29.97fps,24fps,…)
• Somedigitalcompression&storageformatsrequireaspecificframesize
Besser‐SOIMA2011‐Compression&Sampling 50
BitRate
• Datatransferrate‐‐Amountofdatapersecond(comingacrossanetwork,orfromdisk/tapetoscreen)
• UsuallymeasuredinMb/sec(8Mbps,30Mbps,50Mbps,200Mbps)– oyennotedas30Mbps(uncompressed),or8Mbps(compressed)
Besser‐SOIMA2011‐Compression&Sampling 51 Besser‐ICCROMRIO,6/8/0752
• AVI Microsoft format • MOV Quicktime format • MPEG compressed standardized format • RM Real Media compressed format • ASF Microsoft streaming format • DV25 25 Mbits/sec, fixed compression, 4:1:1 color
encoding, 3.6 MB per sec, 60 min. mini-DV cassette has 13 GB of storage • DV50 50 Mbits/sec, 4:2:2 or 3:3:1 color encoding • DV100 100 Mbits/sec, to be used for HDTV
production
From Steve Puglia from Adobe’s “A Digital Video Primer” and http://www.cit.cornell.edu/atc/materials/dig/videoformats.shtml
FileFormats
JimWheeler’slistofDigitalVideoFormats
Besser‐SOIMA2011‐Compression&Sampling 53 Besser‐ICCROMRIO,6/8/0754
MPEG
• MPEG2‐‐currentlymostcommonlyusedfordistribu?on,butdifficulttouseforproduc?on
• MPEG4‐‐usedwithQuickTimeandWindowsMediafiles
• MPEG7&21‐‐metadatastandards,notusedforcompression
7/13/11
10
Encodingformats
• MPEG‐1,op?mizedforCD‐ROMat1.5Mb/sec,1992,usedforVCDs
• MPEG‐2,designedfor10Mb/sec,1994,usedforDVDsandmanybroadcasttvformats
• MPEG‐4,madeforspeedsasslowas64Kb/sec,canseparateouts?lls/audio/video,goodforanima?onandstreaming
• H.264(MPEG‐4part10),usedforBlu‐Rayandsomebroadcasttv
• J2k(mo?onJPEG2000)usedinDigitalCinema(DCDM)• WMV
• .mov• Flash(SWF),Real,…
Besser‐SOIMA2011‐Compression&Sampling 55
Encodingformatgeneralproper?es
• FrameIntegrity‐‐DPX2andJ2kencodeindividualframesasseparatefilesordis?ncten??es
• Mostversa?le(varietyofimplementa?onchoices)‐‐Quick?meandMPEG‐4
56Besser‐SOIMA2011‐Compression&Sampling
Transcoding
• Canalwaysbedonewithenoughhorsepower• Butsomequalitycanbelost,dependinguponcompression,movingbtwncolorspaces(RGBandY’CbCr),etc.
Besser‐SOIMA2011‐Compression&Sampling 57
Streaming
• Latencyisprimaryissue(whichsome?mesnecessitatescaching)
• FormatsincludeFlash(SWF7),Real,MPEG‐4
Besser‐SOIMA2011‐Compression&Sampling 58
Wrappers
• Forwrappingboththe“essence”andthemetadataaboutitintothesamecontainer
• Wrapperscaneitherhaveeverythingembeddedwithinasinglefile,ortheycanhavepointerstofilesoutsidethewrapper
• Wrappersinclude:– MXF(broadcas?ngworld,canwrapanything)
– AAF(videoedi?ngworld,canwrapanything)– METS(libraryworld,canwrapanything)– Quick?me(canwrap.mov,DV,MPEG‐2/4,J2K,Flash)
– AVI(canwrapDV,J2K,DivX,mp3)
– ASF(proprietaryMicrosoy)wrapperforWMVBesser‐SOIMA2011‐Compression&Sampling 59 Besser‐ICCROMRIO,6/8/0760
DigitalCinema
• 24fps• 2K(2,000X1,500)perframefortheatricaldistribu?on
• 4K(4,000X3,000)perframeformasterfile(36M/frameuncompressed)
7/13/11
11
Howmuchstoragedoyouneed?
• 1hourvideo• 640X480• 24bitscolor• 25fps• =83Gbytes
Besser‐SOIMA2011‐Compression&Sampling 61
Administra?ngDigitalVideoover?me
• Preserva?onofdigitalvideofilesshouldfollowstandardrepositoryprac?cesforpreservingalltypesofdigitalcontent– OAIScompliantrepositoryarchitecture
– Metadatawrappers(includingpreserva?onmetadatasuchasPREMISthatdocumentseachchangeorreformabngorreinstalla?onortransforma?on)
– Periodicmonitoringofcheck‐sums– RegularRefreshing– Migra?onand/orEmula?onstrategies
Besser‐SOIMA2011‐Compression&Sampling 62
ImportantGuide(1/2)
63Besser‐SOIMA2011‐Compression&Sampling
ImportantGuide(2/2)
64Besser‐SOIMA2011‐Compression&Sampling
Re/Sources
• Besser,Howard.Talks,hHp://besser.tsoa.nyu.edu/howard/Talks/
• Besser,Howard.Introduc?ontoImaging,hHp://www.geHy.edu/research/publica?ons/electronic_publica?ons/introimages/index.html
• AICGuidetoDigitalPhotographyandConserva?onDocumenta?on,2ndEdi?on,hHp://www.conserva?on‐us.org/index.cfm?fuseac?on=Store.viewProduct&product_ID=19276&prod_cat_ID=13
• Fleischhauer,CarlandCarolineArms,SustainabilityofDigitalFormats,PlanningforLibraryofCongressCollec?ons:ContentCategories‐MovingImages,hHp://www.digitalpreserva?on.gov/formats/content/s?ll.shtml
• IMAP/EAIPreserva?onGuidehHp://www.eai.org/resourceguide/preserva?on.html
• MIAPWebsitehHp://www.nyu.edu/?sch/preserva?on/
Besser‐SOIMA2011‐Compression&Sampling 65