ELE 488 F06
ELE 488 Fall 2006Image Processing and Transmission
(11-28 -06)
Digital Video
•Motion Pictures
•Broadcast Television
•Digital Video
11/28
ELE 488 F06
Motion Picture Television Digital Video
• Broadcast Television (analog)– why invent new technology? – movie at home, mass market– influence of movie on development
• Key Steps– convert pictures to electric signal– send electric signal – convert electric signal to picture
• Comparison with motion picture• High Definition Television - analog digital, compression
• Video telephone - analog predecessor• Video conference - travel cost, people cost• Cable (narrowcast), satellite, interactive, ...
ELE 488 F06
NTSC (National Television Systems Committee)
• 525 lines
– 2 dots less than 1/2000 of distance from eye are not separated (merge into one)
– Assume view at distance 4 times the screen height. No need to have more than 500 lines
– NTSC set 525 lines (475 active)
• Movies in 1940 has 4:3 aspect ratio (width to height)
• 25 or more pictures per second to see continuous motion
• 50 or more pictures per second to avoid flicker
– movies use 24 frames/sec, each shown twice
• 30 frames/sec with 2:1 interlace (60 even-odd fields/sec)
ELE 488 F06
Bandwidth of Broadcast Television
• Without interlace (progressive scan), 60 frames/sec– 500 lines alternating black and white gives 250 full cycles– each horizontal line has 250 x 4/3 ~ 350 full cycles– 60 (frames/sec) x 500 (line) x 350 = 10 MHz (video ONLY)
• With 2:1 interlace, 5 MHz for video
• FCC assigns 6 MHz per broadcast channel– real usable bandwidth is less, MUCH less– actual resolvable lines per vertical height ~250
• Color insertion - must compatible with B/W receiver– Change R-G-B to Y-Cb-Cr – Y is luminance (brightness), Cb and Cr are chrominances– B/W sets converts Y to picture, color sets converts Y-Cb-Cr to
R-G-B then display
ELE 488 F06
Digital Video
• What drives digital video?– Information technology:
electronics, communication, storage, new functionality, …– HDTV
• R-G-B component video– 640 x 480 (pixel) x 3 (color) x 8 (bits/color) x 30 = 221 Mb/sec
• Y-Cb-Cr with subsampled Cb and Cr– 640 x 480 (pixel) x 1.5 (color) x 8 (bits/color) x 30 = 110 Mb/sec
• Compression - MPEG (motion picture expert group)– MPEG-1: CD-ROM, 1.5Mb/sec, 1.2Mb/sec for video,
352x240 (CIF), progressive scan, motion compensation– MPEG-2: extension of MPEG-1, interlace, HD– MPEG-4: object/region based– H.2xx
ELE 488 F06
Video Coding
• Video consists of frames In(i,j)– Code each frame as a still picture – motion JPEG
• Each frame is close to the previous frame– Code the difference FDn(i,j) = In(i,j) – In-1(i,j)
– Differential coding (DPCM, predictive coding)
( In-1(i,j) is the predicted value of In(i,j) )
– Need to code the first frame
ELE 488 F06
Encoding Three Frame Types
Differential encoding of video I – Intra Frame, code by itselfP – Prediction Frame, code by referring to previous I or P frameB – Bi-direction Frame, code by referring to forward AND backward I or P frames
I
BP PP P
B B B B B B B B B
I
ELE 488 F06
Coding of I-frame – same as still image
ELE 488 F06
I Frames
• I frames are Intra-coded using the JPEG coded• I frames can be decoded without reference to other
frames of the video.• Sometimes called anchor frames
I frame: JPEG
Frame 31
A group of pictures (GOP) begins with an I-frame and ends before the next I-frame
A typical GOP length is 15 framesWith only 1 I-frame per GOP (the first frame)
ELE 488 F06
Coding P Frames
• Each frame is close to the previous frame– Code frame difference (differential coding – DPCM)
• Occlusion – parts of current frame is blocked in previous frame
– need future frame to “predict” FDn(i,j) = In(i,j) – In+1(i,j)
current frame In frame difference In - In-1
ELE 488 F06
Coding P Frames
• Each frame is close to the previous frame– Code frame difference (differential coding – DPCM)
current frame In frame difference In - In-1
ELE 488 F06
Coding of P Frames
• Video consists of frames In(i,j)– Code each frame as a still picture – motion JPEG
• Each frame is close to the previous frame– Code the difference FDn(i,j) = In(i,j) – In-1(i,j)
– Differential coding (predictive coding)
– In-1(i,j) is the predicted value of In(i,j)
• Observe: – Most part of frame is unchanged– Except for moving objects– Motion Compensated Coding MPEG
ELE 488 F06
Motion Compensated Video Coding
Observe: Most of picture remains unchanged
But some objects have moved.
So code Displaced Frame Difference
Motion Compensated Coding
previous frame current frame
ELE 488 F06
Displaced Frame Difference
ELE 488 F06
Displaced Frame Difference
ELE 488 F06
ELE 488 F06
P Frames
I frame: JPEG P frame: motion compensated. macro-blocks and macro-blockmotion vectors are indicated
Frame 31 Frame 34
P frames are coded using two methods: - block motion compensation + error coding - jpeg (intra-coded), without referring to previous frames P frames are also anchor frames
Divide P-frame into Macro-blocksMB ~16x16
ELE 488 F06
Finding Motion Vectors
• Matching a block from current frame with a displaced block in reference frame using: (a) sum of squared difference (SSD), or
(b) sum of absolute difference (SAD) (almost always used)• The displacement giving best match is the motion vector of the block • Search methods:
• Global search over the entire anchor frame• Restricted search over local neighborhood• Fast search – over a selected neighborhood,
anchor frame current frame
ELE 488 F06
Illustration: P-frame Macro-Blocks
Frame 34 P-frame
ELE 488 F06
MPEG: I and P frames (anchor frames)
ELE 488 F06
Block Matching Motion Estimation
current frame
Block for which motion vector to be determined
a position for comparison
previous frame
another position
Mean Absolute Diff erence:
MAD(k,l, x,y) = (1/ MN)
1M
0i
1N
0j
| I n(k+i,l+j ) – I n–1(k+x+i,l+y+j) |
Motion vector f or (k,l)th
block:
v(k,l) = arg min(x,y) MAD(k,l, x, y)
Blocks of size MxN
ELE 488 F06
Motion Compensated Encoding of P Frame
current frame
previous frame
Y
ELE 488 F06
Coding of P frame
reconstructed previous frame
Encoder contains decoder
ELE 488 F06
More Detail
ELE 488 F06
Need for Bi-directional Encoding
I
BP PP P
B B B B B B B B B
I
ELE 488 F06
Bidirectional Encoding
ELE 488 F06
Frame Transmit Order vs Viewing Order
View order
Decode order= transmit order
ELE 488 F06
B-frames
B-frames are coded in the same way as P-frames except that for each macro-block, search for the best matching block in both the preceding and succeeding anchor frames. Use the encoding that requires the fewest bits. Called bidirectional encoding.
ELE 488 F06
Block Matching Motion Estimation
current frame
Block for which motion vector to be determined
a position for comparison
previous frame
another position
Mean Absolute Diff erence:
MAD(k,l, x,y) = (1/ MN)
1M
0i
1N
0j
| I n(k+i,l+j ) – I n–1(k+x+i,l+y+j) |
Motion vector f or (k,l)th
block:
v(k,l) = arg min(x,y) MAD(k,l, x, y)
Blocks of size MxN
ELE 488 F06
Complexity of Exhaustive Block-Matching
• Assumptions
– Block size NxN and image size S=M1xM2– Search step size is 1 pixel ~ “integer-pixel accuracy”– Search range +/–R pixels both horizontally and vertically
• Computation complexity
– # Candidate matching blocks = (2R+1)2 – # Operations for computing MAD for one block ~ O(N2)– # Operations for MV estimation per block ~ O((2R+1)2 N2)– # Blocks = S / N2 – Total # operations for entire frame ~ O((2R+1)2 S)
• i.e., overall computation load is independent of block size!
• E.g., M=512, N=16, R=16, 30fps => On the order of 8.55 x 109 operations per second!– Was difficult for real time estimation, but possible with parallel
hardware
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ELE 488 F06
Exhaustive Search: Cons and Pros
• Pros– Guaranteed optimality within search range and motion model
• Cons– Motion vectors are integer valued– High computation complexity
• On the order of [search-range-size * image-size] for 1-pixel step size
• How to improve accuracy?– Half pixel – significantly improvement– Quarter pixel – some improvement– Requires interpolation
• How to improve speed?– Fast search– Try to exclude unlikely candidates
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ELE 488 F06
Half pixel resolution in matching
B
a b
cdp
ELE 488 F06
M24
M15 M14 M13
M16
M11
M12
M5 M4 M3
M17 M18 M19
-6 M6 M1 M2 +6
M7 M8 M9
dx
dy
Fast Algorithm: 3-Step Search
• Search candidates at 9 positions
• Reduce step-size after each iteration– Start with step size
approx. half of max. search range
motion vector {dx, dy} = {1, 6}
Total number of computations: 9 + 82 = 25 (3-step) (2R+1)2 = 169 (full search)
(Fig. from Ken Lam – HK Poly Univ. short course in summer’2001)
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ELE 488 F06
Lowest resolution
Lower resolution
Original resolution
Hierarchical Block Matching• Problem with fast search at full resolution
– Small mis-alignment may give large displacement error esp. for texture and edge blocks
• Hierarchical (multi-resolution) block matching
– Match with coarse resolution to narrow down search range
– Match with high resolution to refine motion estimation
(From Wang’s Preprint Fig.6.19)
UM
CP
EN
EE
40
8G
Slid
es
(cre
ate
d b
y M
.Wu
& R
.Liu
© 2
00
2)
ELE 488 F06
Pixel Decimation
IEEE Trans. on Video Technology, April 1993, pp. 148- 157.
a block in current frame
part of a block in reference frame
ELE 488 F06
Pixel Decimation
ELE 488 F06
Subsampled Motion Field
ELE 488 F06
Subsampled Motion Field
ELE 488 F06
What else can you do with MPEG video?
• The MPEG encoder-decoder is asymmetric. – Encoder is much more complex than the decoder.
Determining motion vectors is a major task– Decoding is easy and fast. – The encoding only has to be done once, the decoding will
be done many times or at many locations.
• Symmetric application?
• Compression loses information. But– compressed video has information not readily available in
original video
Top Related