Simpsons @ work

Why temporal segmentation?

Terms

A Shot

• smallest piece of information

• elementary building block of a video

A Cut

• Easy

• Most common

• 95% of all transitions are cuts

A Fade-in

• Often used at the start of a chapter

• From Black Color to Image

• Linear intensity change

A Fade-out

• Often used at the end of a chapter

• From Image to Black Color

• Linear intensity change

A Dissolve

• Used to show that time has passed

• Often used as a subchapter

• From one Shot to another Shot

• Linear intensity change

Signals

Human vs. Computer

• Human: Knows what happens.• Computer: No cognitive understanding.

Segmentation has to rely on features or signals.

SAD

• Sum of absolute differences• Substract each pixel by each pixel • Very easy approach

SAD = -

¢ SAD (n)k =P width

i

P heightj jI magei ;j (n) ¡ I magei ;j (n+k)j¢ SAD (n)k =

P widthi

P heightj jI magei ;j (n) ¡ I magei ;j (n+k)j

RGB-Histogram

• Find and count all the different shades of red, green and blue and order them from dark to bright.

brightdark

amou

nt

Histogram Difference =

• For red,green and blue: Substract the histograms bin by bin and sum up the difference.

-256 bins

1 Frame Difference

No cut No cut No cut CUT No cut No cut No cut

Result:

How to find the cuts

• Find Cuts with a threshold• Works on SAD and Histogram Difference

frames

sign

al

cut cut cut cutcut cut cut cut

Problems with Motion

• SAD and Histogram are very sensitive to camera or object movement.

frames

sign

alcutcutcutcutcutcutcutcut

SAD:

cut

Problems with Flash

• SAD and Histogram are very sensitive to flash.

frames

sign

al

cutcutcutcutcutcutcutcutcutcut cutcutcutcutcut cut

Problems with dissolves & fades

• So far we have no features that would indicate a fade-in, fade-out or a dissolve

Improvment needed.

Sliding Window

So far: 1 Frame Difference

No cut No cut No cut CUT No cut No cut No cut

Result:

Now: n - Frame difference

No Cut CUT CUT CUT No cut No cut No cut

Result:cut cut cut cutcut cut cut

Motion & Flash

• Not sensitive to flash anymore

• Not sensitive to motion anymore

frames

sign

al

frames

sign

al

Dissolves & Fades

• Finally we are able to detect dissolves

frames

sign

al

fade

Problems with long dissolves

• With this feature we miss the long and slow dissolves.

frames

sign

al

dissolve start

dissolve end

Mean and Variance

Mean

• Mean is the arithmetic average of the histogram values.

• When the histogram changes, the mean moves to left or right.

meanred

meangreen

meanblue

Variance

• Variance indicates how much the histogram values are spread around the mean.

• When the histogram mean changes, the variance changes, too.

vs

small big

Linear Mean Change in Dissolves

Result:

frames

sign

al

Parabolic Variance Change in Dissolves

Result:

frames

sign

al

Process of Detection

• Finds slow dissolves

• Finds fade-in / fade-out

Search for candidates

Approximate mean

Approximatevariance

Calculate theerror made

If small enough:Add disolve

Evaluation

• ~ 250.000 frames of video material (ca. 3h)~ 1800 cuts, 90 dissolves / fades

• Genres: Sport, Cartoons, News, Music, Commericals, Movies, Documentary

• Focus on News ( ABC Lateline )

• 3 Categories: Match, Miss and False Positive

Results

Cuts Fades

Matched out of total

False Positive Matched out of total

False positive

News 290 | 295 5 27 | 29 2

Cartoons 298 | 313 6 12 | 13 4

Sport 130 | 160 3 15 | 17 6

Demo

Questions?