Towards a unique subjective experiment dataset for 3DTV – « 3DTV phase 1 »
Marcus Barkowsky
Overview
Purpose and scope Presentation of (additional) SRCs Presentation of (some) HRCs Timeline Examples of subjective test methodologies and
conditions Discussion
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 2 -
Purpose and scope
The scope of this database is to Provide a unique set of degraded (+reference)
sequences Free for distribution Free for publication Well documented Allowing comparisons!
Idea: Collecting a large number of “indices” for the same set of sequences in order to correlate the results and to discover the “ground truth”
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 3 -
Purpose and scope
This database allows for Cross - lab analysis: Run the same subjective experiment in
several labs Cross - methodology: Use different subjective test
methodologies Cross - environment: Use different environments for the
subjective experiment, e.g. displays, backgrounds Cross – scales: Asking observers to rate on various scales such
as discomfort, depth quality, preference to 2D, etc. in order to find the relation to “Overall Quality of Experience”
Comparing the effect of training of observers Evaluating additional inter-observer differences using for
example questionnaires, observer characterization tests (prism facility, 3D acuity, sensitivity to asymmetric conditions)
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 4 -
Out of this scope
The following points are out of the scope of this evaluation: Length of sequences
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 5 -
Presentation of additional SRC material database
NAMA3DS1:Nantes-Madrid 3D Stereoscopic Video Database 1 Content captured with Panasonic 3DA1 Most of the content captured uncompressed, otherwise
~20Mbit/s per view Camera parameters mostly available Depth map estimation available but not satisfying
Involved in the shooting: From UPM: Jesús Gutiérrez, Narciso García From IRCCyN/IVC: Matthieu Urvoy, Marcus Barkowsky,
Romain Cousseau, Yohann Pitrey, Yao Koudota, Junle Wang, Patrick Le Callet
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 6 -
Presentation of additional SRC material
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
Depth map
Filtered depth map
- 7 -
Presentation of (some) HRCs
HRCs currently planned for the database: Multiview Video Coding (MVC) MPEG-4 part 10 / H.264 (AVC) Transmission distortions Resolution reduction Framerate reduction Asymmetric conditions
Other ideas: Conversion S3D 2D+Z S3D Geometric distortions <input welcome!>
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 8 -
Selection of HRCs
How to select the PVSs for a unique subjective experiment ( ~ 160 sequences) from the large set of PVS? Spread the range of
Different content Different visual degradations (coding, blocking, blurring, color
distortions, geometric distortions, …) Different scale dimensions (video quality, discomfort, depth,
…) Suggestion: Perform a voting for PVSs in different 3D labs
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 9 -
Timeline
Preparation of SRC sequences 2 sets are already available
Identification of conditions in large HRC set End of January?
Distribution of PVSs Finish: End of February
Selection of PVSs Conference Call mid-March
Unique dataset Available end of March
Subjective experiments start Analyze the results, prepare a report Determine impact on Recommendations
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 10 -
Dimensions of subjective experiment
Ideas of different test methodologies and test setups Methodologies
ACR-HR DSCQS Paired Comparison (subset methods) SAMVIQ
Scales Quality of Experience Video Quality Naturalness Visual Discomfort
Environment Different screens Standardized / Home environment
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 11 -
Questions
Adding other SRC sequences NICT license open enough for free distribution?
Who would like to contribute more/different HRCs in that timeframe?
Interest in participation in the PVS selection? Who would be willing to run subjective experiments?
With well-prepared setup, e.g. 3D Blu-Ray? Controlling their own lab setup and experiment method?
13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1
- 12 -
Top Related