ELIS – Multimedia Lab

Davy Van Deursen, Wim Van Lancker

Requirements for automatic evaluation of MF TCs

2

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

• Parsing of a media fragment• Retrieval of the media fragment scenarios• retrieve the full media resource• UA performs mapping and sends HTTP byte range request• UA sends HTTP time/track range request

• with or without redirect …• with or without setup information …

• Interpretation/rendering of the HTTP response• play/visualize the media fragment

Testing a MF-aware UA

3

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

• Automatic testing of parsing+retrieval• catch HTTP requests from the UA• information necessary in the TC

• media fragment URI + corresponding HTTP request(s)

• conclusion: should be feasible

• Automatic testing of interpretation/rendering• computer vision algorithms -> lot of work, not feasible• manual testing (inspired on for example SVG test cases)

• create an artificial a/v resource enabling the manual validation of the result (e.g., show a clock for temporal fragment evaluation)

• http://ninsuna.elis.ugent.be/content/TC_spatial_25fps_with_audio.mp4

Testing a MF-aware UA

4

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

• Parsing of HTTP Range header• Extraction of the requested media fragment• server-triggered redirect (e.g., for tracks)• calculate byte-time/track mapping and return bytes

corresponding to the found byte range

• Automatic testing of parsing and extraction• catch response • information necessary in the TC

• Range header and corresponding response headers• we assume that the server is able to provide the correct

bytes according to its calculated byte ranges

Testing a MF-aware server

5

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

• Automatic evaluation of the TCs demands detailed TC inputs• existing media resource• media fragment (e.g., t=10,20)• possible MF-specific HTTP request headers (i.e., Range

and Accept-Range-Redirect) combinations• possible MF-specific HTTP response headers (i.e.,

Content-Range-Mapping)

Conclusions

Server validation

UA validation

ELIS – Multimedia Lab

End2End examples

…

incompliant behavior

compliant behavior, should be specified in the TC

not sure if compliant or not

misuse of MF-specific headers

7

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

Valid temporal example

t=10UA

get m.ogv

m.ogvget m.ogv

t=10-

get m.ogvbytes=349-

get m.ogv…

get m.ogvt=10- / redirect

Server

206, t=9-

307, bytes=349-

…

…

200

307, m.ogv?t=10

Server

200

307, m.ogv?t=10

Cases to add: include-setup

8

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

UA-detectable ‘invalid’ temporal example

t=fooUA

get m.ogv

m.ogvget m.ogv

include-setup

get m.ogvbytes=0-22

get m.ogv…

Server

206, include-setup

…

200

Cases to add: redirect

9

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

UA-undetectable ‘invalid’ temporal example

UA

get m.ogv

m.ogv

get m.ogvt=20-;incl-setup

get m.ogv…

Server

???

Cases to add: redirect

t=20

length of m.ogv is 15s

get m.ogvt=20-

get m.ogvincl-setup Server 206, include-setup

…

200

10

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

Spatial example

xywh=2,2,20,20

UA

get m.ogv

m.ogvget m.ogv

xywh=2,2,20,20 Server

…

200

307, m.ogv?xywh=2,2,20,20

get m.ogv…

11

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

Valid track example

track=1UA

get m.ogv

m.ogvget m.ogv

track=1

get m.ogvbytes=…-…

get m.ogv…

get m.ogvtrack=1/redirect

Server

206, track=1

307, bytes=…-…

…

…

200

307, m.ogv?track=1

Server

200

307, m.ogv?track=1

Cases to add: include-setup

12

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

UA-undetectable ‘invalid’ track example

track=54UA

get m.ogv

m.ogv

get m.ogvinclude-setup

get m.ogvtrack=54

Server

206, include-setup

…

200

Server…

???

get m.ogvtrack=54;incl-setup

Server

…

206, include-setup

get m.ogvbytes=0-22

Cases to add: redirect

13

ELIS – Multimedia Lab

<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>

• TODOs• provide detailed, machine-readable TC descriptions

• and extend Corrib?

• write evaluation software • Questions

• which and how much media resources to use?• we can vary in container formats, codecs, codec profiles, codec settings (intra

frame period), …

• URI fragments vs. URI queries?• for the moment: only URI fragments• URI queries

• e.g., media.mp4?xywh=100,100,50,50 -> transcoding on server?

• Issues• what if the server chooses what it returns (not exactly the shortest

time range possible …)?

TODOs – questions – Issues