Considerations for Video Scene Selection

Margaret H. PinsonNTIA/ITS

www.its.bldrdoc.gov/n3/video/

Scene Complexity

Should evenly span a wide range of coding difficulty Major impact to quality

Why use an objective metric? Uneven distribution in available

scenes More easy-to-code scenes available

since they are easy-to-film

Complexity Algorithm #1

ITU-T P.910 section 6.3 “Two Criteria for Video Test Scene

Selection”, ITU-T Study Group 12, 1994 Spatial Information

SI = maxtime {stdspace [Sobel (Fn)]}

Temporal Information TI = maxtime {stdspace [Fn-Fn-1]}

Complexity Algorithm #2

“Perceptual Effects of Noise in Digitial Video Compression,” SMPTE 1998

o = log10 {meantime [SIn * TIn]} SIn = rmsspace [Sobel (Fn)] TIn = rmsspace [Fn-Fn-1]

Scene Complexity Criteria

One very difficult to code scene One very easy to code scene One high spatial detail scene Evenly span full range of complexity

Some low Some medium Some high

Wide Variety of Scene Content

Pro Relieves viewer boredom Prevents viewer overtraining Analysis results more general

Con More scene content required Less HRCs for given sized full-matrix

test Could use partial block design

Wider Quality Spread at Lower Bit Rates

2 6 10 14 18 OriginalBad

Poor

Fair

Good

Excellent

Bit-Rate (Mbps)

MO

S

Scene Content Selection Criteria

Avoid repetition of scene type e.g., only one basketball scene

Use each scene as few times as possible to allow desired HRC comparisons e.g., pair each scene with 4 HRCs

Divide experiment into 2 or more full matrixes (block design)

Scene Cuts Masking effect

Hides impairments Few frames before, ≈0.25 sec after

Philosophy No scene cuts Television ≈3 to 5 sec between scene cuts Opening credits rapid scene cuts

GOP structure interactions MPEG-2 GOP 15F for 30fps H.264 / MPEG-4 part 10 GOP 33F for 30fps

Scene Cuts Criteria

One scene with rapid scene cuts ≈½ source no scene cuts ≈½ source with scene cuts Avoid scene cuts at very beginning

or end of edited scene

Camera or Original Quality High quality camera recording

Many people don’t have Medium quality camera recording

Mild coding noise, mild analog noise Impacts HRC response

Low quality camera recording VQEG does not use Commonly available to end users

Impact of Scene Content

Despite viewer instructions “Please do not base your opinion on

the content of the scene or the quality of the acting.”

Ratings include opinion of content “Susie effect”

popular scene

Impact of Scene Content “Aspen” HDTV 50Mbps MOS=4.7

excellent (17 viewers) good (8 viewers) fair (0 viewer)

“ControlledBurn” 25 Mbps HDTV MOS=4.5

excellent (15 viewers) good (6 viewers) fair (3 viewer) coding artifacts

“TouchdownDay” 100 Mbps HDTV MOS=4.0

excellent (4 viewers) good (17 viewers) fair (3 viewers) professional cameraman, no scene cuts TouchdownDay

ControlledBurn

Aspen

Original Scene Content & Quality Criteria No objectionable or controversial

content No politics, violence, scantily clad people

Avoid scenes with technical issues Strong judder, awkward scene cuts, poor

focus, motion blur HDTV test – viewers more discriminating CIF test – viewers more tolerant

Include Animation

Coder/decoder response very different H.264 higher quality MPEG-2 lower quality

Animation overlay increasingly common

Scrolling text and edge noise Preferably, use at least one scene with

animation or animation overlay

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception

Fine detail Gravel, grass, hair, rug texture,

pinstripes Sensitive to blurring, but viewers may

not notice or care Sharp black/white edges Blurred background, foreground in

focus Night scene or dim lighting

At least one

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception

Ramped color with graduated change of color (e.g., sunset)

Water, fire & smoke contribute unusual shapes and shifting patterns

Picture jiggles or bounces significantly Hand-held camera

Flashing lights or other very fast events Emergency lights, camera flash, falling

snow, breaking glass

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception

High motion E.g., object moves 20+ pixels per

frame at VGA resolution Action is in a small portion of the

total picture Attention focus

Colorful scene Important

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception

Small amounts of analog noise Multiple objects moving in a

random, unpredictable manner

Include Close-up of People Popular Skin tones Knowledge of how people ought to

appear & move Internal reference

Internal Reference No Internal Reference

Close-up People Criteria

Filming Privacy issues Releases

Faces large enough to be easily seen Several scenes with close-up people

Close-up Crowd Scene

Contrast and Brightness One scene with low contrast

Soft edges One scene with high contrast

Hard edges One scene with low brightness

Dim One scene with high brightness

Mostly white

Obtaining Content

TV & movie content strictly controlled Purchase for own use

Typically cannot redistribute Hire someone to make content for you Shoot it yourself!

Royalty-free distribution More flexible usage No time expiration

2009, 4th Quarter Consumer Digital Video Library Repository of high quality video User forums For quality measurement algorithms

Subjectively rated databases For consumer video processing

Coding, corrective/reconstructive processing, enhancement, re-formatting

www.cdvl.org