Sampling Issues in Image and Video z – 2-D sampling at ... · PDF fileSampling Issues in...
Transcript of Sampling Issues in Image and Video z – 2-D sampling at ... · PDF fileSampling Issues in...
11
ENEE631 Digital Image Processing (Spring'06)
Sampling Issues in Image and VideoSampling Issues in Image and Video
Spring ’06 Instructor: K. J. Ray Liu
ECE Department, Univ. of Maryland, College Park
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [2]
Overview and LogisticsOverview and Logistics
Last Time:– Motion analysis– Geometric relations and manipulations
Today:– 2-D sampling at Rectangular grid– Lattice theory for multidimensional sampling at non-rectangular grid– Sampling and resampling for video
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [3]
Sampling: From 1Sampling: From 1--D to 2D to 2--D and 3D and 3--DD
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [4]
Review: 1Review: 1--D SamplingD SamplingTime domain– Multiply continuous-time signal with periodic impulse train
Frequency domain– Duality: sampling in one domain tiling in another domain
FT of an impulse train is an impulse train (proper scaling & stretching)
Review Oppenheim “Sig. & Sys”Chapt.7 (Sampling)Chapt.3,4,5 (FS,FT,DFT)
x(t)
p(t) = Σk δ ( t - kT)T
xs(t)
P(ω) = Σ k δ ( ω - 2kπ/T) *2π/T
2π/TX(ω)
ω
Xs(ω)
2π/T
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
22
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [5]
Review: 1Review: 1--D Sampling TheoremD Sampling Theorem
1-D Sampling Theorem– A 1-D signal x(t) bandlimited within [-ωB,ωB] can be uniquely
determined by its samples x(nT) if ωs > 2ωB (sample fast enough).– Using the samples x(nT), we can reconstruct x(t) by filtering the
impulse version of x(nT) by an ideal low pass filter
Sampling below Nyquist rate (2ωB) cause Aliasing
Xs(ω) with ωs < 2ωB Aliasing
ωs=2π/T
ωB
Xs(ω) with ωs > 2ωB Perfect Reconstructable
ωs=2π/T
ωB-ωs
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [6]
Extend to 2Extend to 2--D Sampling with Rectangular GridD Sampling with Rectangular Grid
Bandlimited 2-D signal– Its FT is zero outside a bounded region ( |ζx|> ζx0, |ζy|> ζy0 ) in
spatial freq. domain– Real-word multi-dimensional signals often exhibit diamond or
football shape of supportWith spectrum normalization, we will get spherical shape of support
Jain’s Fig.4.6
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01/2
004)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [7]
22--D Sampling (contD Sampling (cont’’d)d)2-D Comb functioncomb(x,y; Δx, Δy) = Σm,n δ ( x - mΔx, y - nΔy ) ~ separable functionFT: COMB(ζx, ζy) = comb(ζx, ζy; 1/Δx, 1/Δy) / ΔxΔy
Sampling vs. Replication (tiling) – Nyquist rates (2ζx0 and 2ζy0) − Aliasing
Jain’s Fig.4.7
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [8]
22--D Sampling: Beyond Rectangular GridD Sampling: Beyond Rectangular GridSampling at nonrectangular grid – May give more efficient sampling
density when spectrum region of support is not rectangular
Sampling density measured by #samples needed per unit area
– E.g. interlaced grid for diamond-shaped region of support
equiv. to rotate 45-deg. of rectangular gridspectrum rotate by thesame degree
From Wang’s book preprint Fig.4.2
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01/2
004)
33
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [9]
General Sampling LatticeGeneral Sampling LatticeLattice Λ in K-dimension space R K
– A set of all possible vectors represented as integer weighted combinations of K linearly independent basis vectors
Generating matrix V (sampling matrix)V = [v1, v2, …, vk] => lattice points x = V n
e.g., identity matrix V ~ square lattice
Voronoi cell of a lattice– A “unit cell” of a lattice, whose translations cover the whole space– Consists of vectors that are closer to the origin than to other lattice points
cell boundaries are equidistant lines between surrounding lattice points
Sampling density d(Λ) = 1 / |det(V)|– |det(V)| measures volume of a cell; d(Λ) is # lattice points in unit volume
⎭⎬⎫
⎩⎨⎧
∈∀=∈=Λ ∑=
K
jkjj
K nn1
,| ZR vxx
From Wang’s book preprint Fig.3.1U
MC
P EN
EE63
1 Sl
ides
(cre
ated
by
M.W
u ©
2001
/200
4)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [10]
Sampling Density:d1 = 1d2 = 2 / √3
)(hexagonal 12/102/3
ar)(rectangul 1001
2
1
⎥⎦
⎤⎢⎣
⎡=
⎥⎦
⎤⎢⎣
⎡=
V
V
From Wang’s book preprint Fig.3.1
Example of LatticesExample of Lattices
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [11]
Frequency Domain View & Reciprocal LatticeFrequency Domain View & Reciprocal LatticeReciprocal lattice Λ# for a lattice Λ (with generating matrix V)– Generating matrix of Λ# is U = (VT)-1
– Basis vectors for Λ and Λ# are orthonormal to each other: VT U = I– Denser lattice Λ has sparser reciprocal lattice Λ# : det(U) = 1 / det(V)
Frequency domain view of sampling over lattice– Sampling in spatial domain Repetition in freq. Domain– Repetition grid in freq. domain can be described by reciprocal lattice
Aliasing and prefiltering to avoid aliasing– Aliasing happens when Voronoi cell of reciprocal lattice overlapped
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01/2
004)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [12]
Sampling EfficiencySampling EfficiencyConsider spherical signal spectrum support– Most real-world signals have symmetric freq. contents in many directions– The multi-dim spectrum can be approximated well by a sphere (with proper
scaling spectrum support)
Voronoi cell of reciprocal lattice need to cover the sphere to avoid aliasing– Tighter fit of the Voronoi cell to the sphere requires less sampling density
What lattice gives the best sphere-covering capability?Sampling Efficiency ρ = volume(unit sphere) / d(Λ) prefer close to 1
From Wang’s book preprint Fig.4.2 & 3.5
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01/2
004)
44
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [13]
Sampling Lattice ConversionSampling Lattice ConversionFrom Wang’s book
preprint Fig.4.4
Intermediate
Original
TargetedUMCP ENEE631 Slides (created by M.Wu © 2001)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [14]
Recall: 1Recall: 1--D D UpsampleUpsample and and DownsampleDownsample
From Crochiere-Rabiner “Multirate DSP” book Fig.2.15-16
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [15]
General Procedures for Sampling Rate ConversionGeneral Procedures for Sampling Rate Conversion
From Wang’s book preprint Fig.4.1
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [16]
Example: Frame Rate ConversionExample: Frame Rate ConversionVideo sampling: formulate as a 3-D sampling problemNote: different signal characteristics and visual sensitivities along spatial
and temporal dimensions (see Wang’s Sec.3.3 on video sampling)
General Approach to frame rate conversion– Upsample => LPF => Downsample
Interlaced 50 fields/sec 60 fields/sec– Analyze in terms of 2-D sampling lattice (y, t) – Convert odd field rate and even field rate separately
do 25 30 rate conversion twicenot fully utilize info. in the other fields
– Deinterlace first then convert frame ratedo 50 60 frame rate conversion: 50 300 60
– Simplify 50 60 by converting 5 frames 6 frameseach of output 6 frames is from two nearest frames of the 5 originalsweights are inversely proportional to the distance between I/O
– May do motion-interpolation for hybrid-coded video
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01/2
004)
55
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [17]
From Wang’s book preprint Fig.4.3
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [18]
Case Studies on Sampling and Case Studies on Sampling and ResamplingResamplingin Video Processingin Video Processing
Reading Assignment: WangReading Assignment: Wang’’s book Chapter 4s book Chapter 4
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [19]
Video Format Conversion for NTSC Video Format Conversion for NTSC PALPAL
Require both temporal and spatial rate conversion– NTSC 525 lines per picture, 60 fields per second– PAL 625 lines per picture, 50 fields per second
Ideal approach (direct conversion)– 525 lines 60 field/sec 13125 line 300 field/sec
625 lines 50 field/sec
4-step sequential conversion– Deinterlace => line rate conversion
=> frame rate conversion => interlace
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [20]
From Wang’s book preprint Fig.4.9
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
66
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [21]
Simplified Video Format ConversionSimplified Video Format Conversion
50 field/sec 60 field/sec– Simplified after deinterlacing to 5 frames 6 frames– Conversion involves two adjacent frames only
625 lines 525 lines– Simplified to 25 lines 21 lines– Conversion involves two adjacent lines only
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [22]
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [23]
Interlaced Video and Interlaced Video and DeinterlacingDeinterlacingInterlaced videoOdd field at 0 Even field at Δt Odd field at 2Δt Even field at 3Δt
…
Deinterlacing– Merge to get a complete frame with odd and even field
Examples from http://www.geocities.com/lukesvideo/interlacing.html
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [24]
DeDe--interlacing: Practical Approachesinterlacing: Practical Approaches
Spatial interpolation– Vertical interpolation within
the same field (1-D upsample by 2)– Line averaging ~ average the line
above and below D=(C+E)/2
Temporal interpolation– 2-frame field merging => artifacts– 3-frame field averaging D=(K+R)/2
fill in the missing odd field by averaging odd fields before and after
Spatial-temporal interpolation– Line-and-field averaging D=(C+E+K+R)/4
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
77
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [25]
MotionMotion--Compensated DeCompensated De--interlacing interlacing
Stationary video scenes– Temporary deinterlacing approach yield good result
Scenes with rapid temporal changes– Artifacts incurred from temporal interpolation– Spatial interpolation alone is better than involving temporal
interpolation
Switching between spatial & temporal interpolation modes– Based on motion detection result– Hard switching or weighted average– Motion-compensated interpolation
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
01)
ENEE631 Digital Image Processing (Spring'06) Lec24 – 2-D and 3-D Sampling [26]
Summary of TodaySummary of Today’’s Lectures LectureSampling and resampling issues in 2-D and 3-D– Sampling lattice and frequency-domain interpretation– Sampling rate conversion
Next Lecture: – Introduction to digital watermarking for image and video
Readings– Wang’s book: Sec. 3.1-3.3, 3.5; Chapter 4– “Computer Graphics” Chapter 5 (Hearn-Baker, Prentice-
Hall, 2nd Ed)
UM
CP
ENEE
631
Slid
es (c
reat
ed b
y M
.Wu
©20
04)