E.G.M. Petrakis Video Processing 1

Video Processing

Video is a rich information source frames (individual images)links between frames (cuts, fades,

dissolves)changes in color, shapes, motion of both

camera and objectsacquisition (shot angles, camera motion)each type of video has its own

characteristics (commercials, news, sports)

E.G.M. Petrakis Video Processing 2

Video Structure

Frame: typically 1/25 or 1/30 secondsShot: sequence of similar frames

elementary video unitsa single event

Clip / Scene: sequence of shots consecutive in time, space, action

Episode: consecutive scenes intro, news, reporter, weather

E.G.M. Petrakis Video Processing 3

Del Bimbo 99

Video Structure (cont.d)

E.G.M. Petrakis Video Processing 4

Video Retrieval

A video can be accessed at the structural level: browsing, retrieval

of shots, scenes, episodescontent level: according to camera

motion, motion of characters or objects, audio properties, scenes, semantics of color, texture, shape, object properties …

E.G.M. Petrakis Video Processing 5

Video Partitioning

Shot extraction and classification of editing effects due to camera breaks (cuts): abrupt transitionsgradual transitions: dissolves, wipes, fade-

in/outcamera movements: panning, tilting, zoom

Simple methods for camera breaks More sophisticated methods for gradual

transitions and camera movement

E.G.M. Petrakis Video Processing 6

shot1

shot2

Furht. et.al 96

Cut: Frames Between Shots

E.G.M. Petrakis Video Processing 7

Furht. et.al 96

Dissolve: Transition Frames

E.G.M. Petrakis Video Processing 8

Uncompressed Video Partitioning

Detect boundaries of consecutive camera shotscompare adjacent framesfor camera breaks compare color

histograms of adjacent framesfor gradual transitions and camera

motion histograms are less successfulOther techniques are based on edge

detection, motion analysis etc.

E.G.M. Petrakis Video Processing 9

Histograms

Grey level histograms for 3 successive frames

Frames 1 and 2 almost identical

Camera break between 2 and 3

Compute histogram differences

E.G.M. Petrakis Video Processing 10

Color Histograms

H(I,v) : number of pixels in I with intensity vMxN pixels

Grey-level images: 8 bits/pixel - 256 bins in histogram

Color images: 24 bits/pixel - 224 bins Convert color to YUV color space and

process intensity only: I = 0.299R + 0.587G + 0.114B

E.G.M. Petrakis Video Processing 11

1. Camera Breaks

Pair-wise pixel comparison (intensities)Histogram comparison for camera

breaksthreshold selectiontwin comparison approachmulti-pass approach

Motion vector analysis for camera motion and gradual transitionsHough transformVideo X-ray

E.G.M. Petrakis Video Processing 12

Pair-Wise Pixel Comparison

Count pixels changed from a frame to the next

A shot boundary is found if more than Tb pixels changed

Problem: sensitivity to camera/object motion and noisemany pixels change

otherwise0

l][k,P-l][k,P 1 1ii tifDPi

b

NM

lk iT

MN

lkDP

100],[

,

1,

E.G.M. Petrakis Video Processing 13

Pair-Wise Block Comparison

Compare blocks instead of pixelsμi ,μi+1: mean intensity

values in frames si,si+1: variances

Less sensitive to motion and noise

t: does not change for different video sources

t

SSDP

ii

SS

i

iiii

1

22

2211

E.G.M. Petrakis Video Processing 14

Pair-Wise Histogram Comparison

Even less sensitive to motion i: frame countj: intensity count in

HG=MxN intensities

TjH

jHjHSD

TjHjHSD

G

j i

iii

G

jiii

1 1

2

1

11

)(

)()(

or

)()(

Furht. et.al 96

E.G.M. Petrakis Video Processing 15

gradual transitions camera

breaks

Furht et.al. 96

Histogram Comparison

E.G.M. Petrakis Video Processing 16

Thresholds

Tolerate variations while ensuring good performancelow thresholds accept many false

positiveshigh thresholds reject true transitions

Threshold: varies from one video source to another e.g., cartoons exhibit larger frame

differences than films

E.G.M. Petrakis Video Processing 17

2. Gradual Transitions

Transitions not as high as in camera breaksdissolve, fade-in/out, other special effectshigh transitions in a neighborhood lower thresholds do not solve the problem

Furht. et.al 96

E.G.M. Petrakis Video Processing 18

Twin-Comparison

Two thresholds Tb for camera break detectionTs < Tb for special effects like dissolves, motion

Compare consecutive frames (e.g. histograms)if difference exceeds Tb: camera breakif difference exceeds Ts: potential cut

Accumulate differences from that frame until the transition becomes lower than Ts

A boundary is detected if the accumulated difference becomes higher than Tb

E.G.M. Petrakis Video Processing 19

camera break

specialeffect

shotboundary

Twin Comparison Example

Furht. et.al 96

E.G.M. Petrakis Video Processing 20

Threshold Selection

Based on the distribution of the frame-to-frame histogram changes Most changes are due to noise ~90%, scene

changes ~10%Scan entire video and compute distribution

of changes (e.g., histogram differences)Assume Gaussian distribution and compute: μ, σ

Compute the two thresholds asTb=μ+ασ, α=4-6Ts=βμ, β=1.5-2

E.G.M. Petrakis Video Processing 21

Motion Vector Analysis (object motion)

Detect camera breaks using motion vectors (MV)

Compute MVs by block matchingCompute correlation of the same

block bi from frame i to frame i+1

Assign a displacement vector Dbi to bi

The same for all blocks between frames

E.G.M. Petrakis Video Processing 22

Motion Smoothness

For each frame compute Wi

Nominator counts significant motion vectors in frame i

Denominator counts significant transitions in motion vectors

Wi0 indicates camera break

otherwise

DbDbifbw

otherwise

tifbw

bw

bwW

iii

si

b i

b ii

0

t 1)(

0

Db 1)(

)(

)(

m12

i1

2

1

E.G.M. Petrakis Video Processing 23

3. Camera Motion

Detect changes due to camera movementCamera zoom in/out,

tilting/panningTransitions resemble

gradual transitionsMore specific

techniquesAnalysis of motion

vectors

Furht. et.al 96

E.G.M. Petrakis Video Processing 24

Camera Panning

Most motion vector exhibit same directionΣb|θb - θm| < Θp

Θb is the direction of the vector of block b

θm is the direction of the entire set of blocks

The variation Θp should be close to 0

Furht. et.al 96

E.G.M. Petrakis Video Processing 25

Camera Zooming

Assume focus center within the frame and little object motion at the periphery of a frame

Compare v in top/bottom rows, left/right columns

Every column |vtop-vbottom| >= max(|vtop|,|vbottom|)

Every raw |vleft-vright| >= max(|vleft|,|vright|)Zooming: most vectors satisfy these

condition

E.G.M. Petrakis Video Processing 26

Compressed Video Partitioning

Frame comparison using DCT coefficients instead of blocksMPEG motion vectors Combination of the above

Same techniques using I frames only (faster)

Furht. et.al 96

E.G.M. Petrakis Video Processing 27

%100)(),(max

)()(

64

1 64

1

k lklk

lklkl ficic

ficicDiff

Pair-Wise DCT Comparison

Applies to the DCT coefficients of corresponding blocks in consecutive I frames which are f-distance apartk=1...64 coefficients

For each block computes

E.G.M. Petrakis Video Processing 28

Threshold Selection

Same techniques for threshold selection with uncompressed video Diffl > t : a block has changedt: does not vary with video sources

Shot boundary: the percentage of blocks that changed exceeds Tb

Tb varies for different video sources

bblocks l Tblocks

DifffiiD

%100#

),( #

E.G.M. Petrakis Video Processing 29

Example

Sharp peaks indicate camera breaks Cannot handle gradual transitions, camera or object

motionTwin comparison using Ts, Tb thresholdsApplies only to I frames (no DCTs for P, B frames)Faster but, many false positives

D values between successive frames

Furht. et.al 96

E.G.M. Petrakis Video Processing 30

Motion Vectors

Motion vectors are associated with P, B frames

The residual error between blocks is DCT encodedIf the error is large DCT encode the

original blocks and no motion vectors are stored in this case

Many blocks with no motion vectors indicate camera break Camera break: motion vectors M < Tb ~ 0

E.G.M. Petrakis Video Processing 31

Motion Vectors (cont.d)

Many false positives for static frames (frames with no motion)

Camera breaks: deep and narrow gaps in diagrams with number of vectors

Combine with D(i,i+f): camera break when high D(i,i+f) with M < Tb but large gap means no motion (static frames) and not transition

Difficult to detect gradual transitions

E.G.M. Petrakis Video Processing 32

Example

staticframes

shotboundary

M < Tb

Furht. et.al 96

E.G.M. Petrakis Video Processing 33

Hybrid Multiple Pass Approach

First step: DCT comparison on I frames to locate regions of potential interestlow spatial resolution (large f) : very fastadvantage: gradual transitions are likely to be

detected because of the large skip factordisadvantage: many false positives

Second step combining DCTs and motion vectors smaller skip factor at the vicinity of candidate boundaries

High processing speed in achieved

E.G.M. Petrakis Video Processing 34

Camera Motion - Object Motion

Detect specific patterns of motion vectors

Similar techniques with uncompressed video

Motion vectors are provided by MPEG P, B frames

Similar results

Furht. et.al 96

E.G.M. Petrakis Video Processing 35

Video Browsing

Select a key-frame from each shotFirst, middle, last, average frame of shots, I

frames for compressed video …Image retrieval based on key-frames

key-framesA.Smeaton,DCU

E.G.M. Petrakis Video Processing 36

Hierarchical Browsing

Problem: large number of key-framesSolution: organize key-frames

hierarchicallyvideo at the top, key-frames for scenes, shots

are lowerhierarchical video

browserA.Smeaton,DCU

E.G.M. Petrakis Video Processing 37

Furht et.al. 96

Hierarchical Browser

E.G.M. Petrakis Video Processing 38

Comments on Video Segmentation

Histograms are sufficient in most casesAudio could help (silence between shots)Only one pass through the entire videoComputational cost and delay can be highA pass at reduced spatial resolution

detects potential changes (comparisons every k frames)

Processing at the vicinity of changes to verify the results

E.G.M. Petrakis Video Processing 39

Further Reading Centre for Digital Video Processing, Dublin University

http://www.cdvp.dcu.ie B. Furht et.al. “Video and Image Processing in Multimedia

Systems”, Chapter 12-14, Kluwer, 1996 H.J.Zhang, A. Kankanhalli, S.W.Smoliar, “Automatic

Partitioning of Full-Motion Video”, Multimedia Systems, 1(1):10-28, 1993

V. Kobla, D. Doermann, C. Faloutsos “VideoTrails: Representing and Visualizing Structure in Video Sequences” ACM Multimedia 97, Nov. 1997

O. Marques, B. Furhrt, “Content Based Image and Video Retrieval”, Kluwer Academic Publishers, 2002

V. Kobla, D. S. Doermann, K-Ip (David) Lin, C. Faloutsos “Compressed domain video indexing techniques using DCT and motion vector information in MPEG video” Proc. of the SPIE Conf. on Storage and Retrieval for Image and Video Databases, Vol. 3022, Feb. 1997

E.G.M. Petrakis Video Processing 40

References S. Lefevre, J. Holler, N. Vincent, “

A review of Real-Time Segmentation of Uncompressed Video Sequences for Content-Based Search and Retrieval”, RFAI publication: Real Time Imaging

C. Doulaverakis, V. Vagionitis, M. Zervakis and E. Petrakis: "Adaptive Methods for Motion Characterization and Segmentation of MPEG Compressed Frame Sequence", Intern. Conference on Image Analysis and Recognition (ICIAR'2004), Porto, Portugal, Sept./Oct. 2004, Proc. Part I, Springer Verlag (LNCS 3211), pp. 310-317.