Next Generation Video Streaming System using Scalable High ...
Transcript of Next Generation Video Streaming System using Scalable High ...
1 © 2015 Gachon University. All rights reserved.
Next Generation Video Streaming System using
Scalable High Efficiency Video Coding (SHVC)
Aug. 24. 2015
KIISE SWCC 2015
Eun-Seok Ryu [email protected]
Department of Computer Engineering,
School of Information Technology
Gachon University
2 © 2015 Gachon University. All rights reserved.
Contents
+ Short Bio / 과거에 상상하던 멀티미디어 기술
1. Next Generation Video Coding Technologies• High Efficiency Video Coding (HEVC)
• Scalable HEVC (SHVC)
• Multi-view Video Coding (MVC) / 3D
2. Error Resilient Video Streaming• Unequal Error Protection using Raptor Codes
• Error Concealment using EC modes
• Selective Video Streaming
3. Video Streaming Applications• Video Streaming over Multiple Network Interfaces
• Power Aware HEVC Streaming with DASH
• Three-Screen TV using SVC Platform
• Telehealth and Haptic Video Streaming for the Individuals with Visual Impairments
3 © 2015 Gachon University. All rights reserved.
가천대학교 IT대학 컴퓨터공학과 조교수 류은석
• 주요경력 (연구분야: 멀티미디어 통신 시스템)
• 2015.03-현재 : 가천대학교 IT대학 컴퓨터공학과 (성남) 조교수
• 2014.03-2015.02: 삼성전자 (수원) 수석연구원 / 파트장•임베디드 비디오 처리 및 통신 연구/특허개발
• 2011.01-2014.02: InterDigital Labs (San Diego), Staff Engineer•JCT-VC HEVC/SHVC 비디오 표준화 연구
• 2008.09-2010.12: Georgia Inst. Tech. (Atlanta), Postdoctoral Research Fellow•Three-Screen TV를 위한 홈게이트웨이 연구
• 2008.03-2008.08: 고려대학교 정보통신기술연구소 (서울), 연구교수•Scalable Video Coding 기술을 이용한 다중망 비디오 전송기술 연구
• 학력
• 고려대학교 컴퓨터학과 이학사(1999) / 이학석사(2001) / 이학박사(2008) (지도교수유혁)
• 연구실적
• 논문: 19 Journal / 17 International Conf. / 22Korean Domestic Conf.
• 국제표준: 14 Video Standard Contribution
• 국제/국내 특허: 12+ US Patent Applications / 17 Korea Patent Applications
4 © 2015 Gachon University. All rights reserved.
과거에 상상하던 미래의 멀티미디어 기술
1912, Hugo Gernsback's 124C 41 A Romance of the Year 2660
1925, Telemedicine
1954, Teledoctor
1963, TV Glasses
Radio Doctor (1924)
5 © 2015 Gachon University. All rights reserved.
Part 1Next Generation Video Coding Technologies- High Efficiency Video Coding (HEVC)
- Scalable HEVC (SHVC)
- Multi-view Video Coding (MVC) / 3D
6 © 2015 Gachon University. All rights reserved.
The History of Video Coding Standards (1)
201420121988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2016
H.261
MPEG-1
H.263
MPEG-4Part 2
MPEG-2/H.262
JOINT
H.264/MPEG-4 Part 10 (AVC)
HEVCH.264/MPEG-4 AVC
SVC & MVC Ext.
HEVC SVC & MVC Ext.
MPEG-4 Version 2
H.263 + H.263 ++VideoConference
PMP
UHD,Mobile FHD
VC-2VC-1
(from WMV) (from Dirac)
AdobeFlash Video
(8K)
*김찬열, HEVC Overview, 삼성전자 DMC 연구소, Dec. 2014.
7 © 2015 Gachon University. All rights reserved.
The History of Video Coding Standards (2)
201420121988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2016
VC-1
VP6 VP7 VP8VP3
VC-2
WebMProject
Theora
WMV7
WMV8
WMV9
Dirac
Google acquired On2
Open Source
(Open Source)
(Open Source)
(Open Source)
AdobeFlash Video
VP9
*김찬열, HEVC Overview, 삼성전자 DMC 연구소, Dec. 2014.
8 © 2015 Gachon University. All rights reserved.
High Efficiency Video Coding (HEVC)
• Joint Standard (JCT-VC)
• ISO/IEC JTC1: ISO/IEC 23008-2 (MPEG-H Part 2)
• ITU-T: ITU-T Recommandation H.265
• Higher resolution / frame rate / color space
• UHD / 4K – 8K video
AVC HEVC
Picture > Slices > MB (Max 16x16)Macroblock / Block- Transform 4x4 / 8x8
Picture > Slices > CTUs (Max 64x64)CU / PU / TU (Coding/Prediction/Transform Unit)- Transform 4x4 ~ 32/32 (DCT+DST for Intra)
Intrapicture prediction- Up to 9 directional modes
Intrapicture prediction- Angular prediction 33 directional+2 modes
Variable Block Size Asymmetric Motion Partitioning (AMP)
Motion copy mode- Direct mode
Motion copy mode- Merge Mode /Advanced Motion Vector Prediction(AMVP)- Transmit MVD
Deblocking filter only Deblocking + Sample Adaptive Offset (SAO)
CAVLC / CABAC CABAC
Slices (FMO) Tiles / Wavefront
CTU CTU CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU
CTU CTU CTU CTU CTU CTU
Thread1
Thread2
Thread3
9 © 2015 Gachon University. All rights reserved.
High Efficiency Video Coding (HEVC)
Tools for Parallel Processing and Standard Extensions:
• Performance Comparison
• HEVC Extensions• SHVC / MV-HEVC / 3D-HEVC
• Transmission • MPEG DASH / MMT
• Industrial Broadcasting Standard• ATSC 3.0 (North America)
•Contents: UHD / Smart interactivity / Multiple contents on the screen•Network: High speed broadcasting network / Emerging broadband network / Hybrid delivery•Devices: Larger screen devices / Multimedia devices everywhere / Single service on the multiple screens
10 © 2015 Gachon University. All rights reserved.
Scalable Video Coding (SVC/SHVC)
Base Layer
(Temporal & SNR Scalability)
Spatial Enhancement Layer 1
(Temporal Scalability)
I frame
B frame
B frame
B frame
P frame
11 © 2015 Gachon University. All rights reserved.
Scalable Video Coding (SVC/SHVC) (2)
• Supports various device capabilities with single
bitstream
• Reduces total BW
SVC < AVC (around 20%, for 3 Spatial layers)
• No transcoding, No duplicated contents for every
target bitrates
Temporal
Decomposition
Transform/
Entropy coding
Motion
coding
2D spatial
upsampling
Intra prediction
for intra block
Temporal
Decomposition
Transform/
Entropy coding
Motion
coding
Intra prediction
for intra block
2D spatial
Down-
sampling
QP1
QP2 Mu
ltiplex
motion
info.
Video
input
Enhancement Layer Encoder
Base Layer Encoder
SVC
Bitstream
Base Layer
Enhancement Layer n
Enhancement Layer n’
HDTV
Laptop
Mobile
High quality,
Resolution, and
temporal video
Medium quality,
Resolution, and
temporal video
Low quality,
Resolution, and
temporal video
Enhancement Layer Encoder
2D spatial
Down-
sampling
12 © 2015 Gachon University. All rights reserved.
SVC versus SHVC
• Higher-layer decoding picture buffer (DPB) uses lower-layer’s
reconstructed picture
• Simple / easy to deploy
*US20140010294 A1 (InterDigital Patent Application /2014)
13 © 2015 Gachon University. All rights reserved.
Multi-view Video Coding (MVC) / 3D
• Close correlation between multiple views.
CC
TV
#2
Close
Correlation
Camera Server
Data
Processing /
Streaming
Server
WAN
LAN
MVC encoding
Picture #1 Picture #2 Picture #3
City Park
View 1
Time 1
View 2
Time 1
View 3
Time 1
View 4
Time 1
View 5
Time 1
View 1
Time 3
View 2
Time 3
View 3
Time 3
View 4
Time 3
View 5
Time 3
View 1
Time 2
View 2
Time 2
View 3
Time 2
View 4
Time 2
View 5
Time 2
View 1
Time 4
View 2
Time 4
View 3
Time 4
View 4
Time 4
View 5
Time 4
View 1
Time 5
View 2
Time 5
View 3
Time 5
View 4
Time 5
View 5
Time 5
View
Tim
e
14 © 2015 Gachon University. All rights reserved.
Part 2Error Resilient Video Streaming- Unequal Error Protection using Raptor Codes
- Error Concealment using EC modes
- Selective Video Streaming
15 © 2015 Gachon University. All rights reserved.
General Video Streaming System
• Video server and client
Video
Encoder
Error
Protection
Selective
Scheduler
Channel Prediction
Network
MANE
(smart router)
Edge server
Home
gateway
QoS Controller(EC mode selection)
signal
data
Video Client
(Decoder with EC)
data &
signal
Error Protection
- Forward Error Correction (FEC)
- Unequal Error Protection (UEP)
NetworkVideo Client
with decoder
Video Server
with encoder
Error Recovery
Error Concealment
- Interpolation / Picture copy / ...
Error Resilient Video Streaming
Channel monitoring / feedback
16 © 2015 Gachon University. All rights reserved.
2D Parity FEC / Interleaving
P1 P2 P3
P4 P5 P6
P7 P8 P9
XOR
XOR
C1 C2 C3
R1
R2
R3
Rep
air
Pack
ets
Repair Packets
Video Packet Interleaving
P1 P2 P3 P4 FEC0
P5 P6 P7 P8 FEC1
P9 P10 P11 P12 FEC2
P13 P14 P15 P16 FEC3
FEC4 FEC5 FEC6 FEC7
k 2t
Data Parity
n
17 © 2015 Gachon University. All rights reserved.
Raptor (FEC) Codes
• Need low computational complexity
• Show strong packet loss recovery feature
• No packet order coordination is needed
• Need low memory consumption
• Support unlimited encoding length
Source Symbols
(LDPC & Half )
Pre-coded Symbols
Raptor-encoded Symbols
( LT-coding )
Redundant symbols
Video
Encoder
QoS
Controller
Bitstream &
Priority info.
Damaged
bitstream
Available bandwidth
Raptor
Encoder
Bitstream &
FEC (n, k) Raptor
Decoder
Error pattern file
Bitstream with
repair symbols Video
Decoder
Statistic
logs
18 © 2015 Gachon University. All rights reserved.
Unequal Error Protection (UEP)
• Picture-level UEP
• In single layers•I / P / B pictures
•# of picture referencing
• In different layers•Base Layer / Enhancement layer
• Region of Interest (ROI) UEP
• Flexible Macroblock Ordering
(FMO) with Slice Group (SG)
Slice Group 1 Slice Group 0 Slice Group 2
Type 0
Type 3
Type 1
Type 4
Type 2
Type 5
SG 0
SG 1
SG 2
SG 3
SG 4
SG 0 SG 1
SG 0
SG 1
SG 2
SG 0
SG 1 SG 0
SG 1
SG 0
SG 1
19 © 2015 Gachon University. All rights reserved.
Picture-level UEP
• I / P / B pictures
• Amount of picture referencing (both picture-level and MB/PU level)
• Hierarchical B pictures located in the same temporal level
1. For the video frame priority, use the temporal level (or TID) of hierarchical predictionstructure.
2. However, pictures within the same temporal level can still have different priorities.Causes different levels of temporal error propagations and showes different video quality degradations.
3. frame_priority_idc field distinguishes priority of the frames within the same temporal level.
* Eun-Seok Ryu, “Prediction-Based Picture Prioritisation Method for Hierarchical B-structure of High Efficiency Video Coding”, IET Electronics Letters, Vol. 49, No. 20, pp. 1268-
1270, Sep. 26, 2013.
20 © 2015 Gachon University. All rights reserved.
Picture-level UEP: Experimental Results
Test settings
• RA in common test condition
• Drop 1 picture per intra period
• HM6.1
Observation
• Clear difference in performance
between packet loss at Position A
and at Position B
• upper: all TID = 0, lower: 4 level
TIDs
• Error Propagation Delta (Blue v.s.
Red): 2.8dB in Y-PSNR
• Results: gain 2.2dB to 7.5dB
*Results are from JCTVC-K0262.
21 © 2015 Gachon University. All rights reserved.
ROI UEP with SVC
EL2EL1BL
Encoded LayeredSVC Stream
H.264 SVC Encoder
Module forROI Scalability
EL2EL1BL
High Quality(ROI)
Medium Quality
EL1BL
ROI
Control # of Layers after Encoding
* Eun-Seok Ryu, Sung Won Han, “The Slice Group-Based SVC Rate Adaptation Using Channel Prediction Model”, In IEEE ComSoc Multimedia Communications Technical Committee
E-Letter (IEEE MMTC), Vol. 6, No. 5, pp. 39-41, May 2011.
22 © 2015 Gachon University. All rights reserved.
Error Concealment (EC)
• Picture Copy (PC)
• Temporal Direct (TD)
• Motion Copy (MC)
• Base Layer Skip (BLSkip; Motion & Residual upsampling)
• Reconstructed BL upsampling (RU)
• EC-mode Signaling (ECMS)
0
4
2
31
6
75
8
picture copy
picture copy
hierarchical B pictures
Picture Copy.
MV0
MV1
collocated block
MVc
direct mode block
RPL 1RPL 0 current B picture
Temporal Direct for B Pictures
23 © 2015 Gachon University. All rights reserved.
EC: ECMS Motivation
• Traditionally, Error Concealment (EC) methods are applied at the decoder side to
recover from packet loss and to reduce error propagation.
• There are various EC methods that decoder can perform.
• However, the decoder alone does not know which EC method is optimal because it
does not have the original picture.
• Thus, the optimal EC method is simulated in the encoder and is signaled.
24 © 2015 Gachon University. All rights reserved.
EC: ECMS Method
1) Encoder simulates various EC methods on a damaged picture and its error
propagation effects.
2) Determines the best EC method.
3) Signals the determined optimal EC mode to the video decoder.
* Eun-Seok Ryu, Joongheon Kim, “Error Concealment Mode Signaling Method for Robust Mobile Video Transmission”, International Journal of Electronics and Communication,
vol. 69, no. 7, pp. 1070-1073, July 2015.
25 © 2015 Gachon University. All rights reserved.
EC: ECMS Method: Experimental Results
• EC0: Picture copy from the first picture in RPL0.
• EC1: Picture copy from the first picture in RPL1.
• EC2: Picture copy from the ILP (upsampled base layer picture)
• EC3: Proposed EC mode signaling
• Condition: JCTVC SHVC 4.0, BL QP: 22, EL QP 20. Random Access with 1sec intraperiod. (common test condition)
• Related MMT CE proposal: M32346 (San Jose Meeting, Jan. 2014)
Average PSNR gain between EC modes and proposed method (2x spatial scalability).
Average PSNR gain between EC modes and proposed method (single layer coding).
Average Gain: 3.27dB for 2x
Coding Sequence Resolution frame rate frame lengthSingle layer coding (HEVC) Gain (Y-PSNR, dB)
EC0 EC1 Proposed Prop.-EC0 Prop.-EC1
Single Layer Coding (HEVC)
ParkScene 1920x1080 24 240 28.86 29.05 29.14 0.28 0.10
Han 1920x1080 30 300 32.34 32.19 32.48 0.14 0.29
KristenAndSara 1280x720 60 600 37.14 37.19 37.31 0.18 0.12
BQMall 832x480 60 600 22.13 22.14 22.17 0.04 0.03
Coding Sequence Resolution frame rate frame lengthResults (Y-PSNR, dB) Gain (Y-PSNR, dB)
EC0 EC1 EC2 Proposed Prop.-EC0 Prop.-EC1 Prop.-EC2
2x Spatial Scalability (SHVC)
ParkScene 1920x1080 24 240 28.87 29.07 34.18 34.38 5.50 5.31 0.20
Han 1920x1080 30 300 32.27 32.13 38.84 38.84 6.57 6.72 0.00
KristenAndSara 1280x720 60 600 37.17 37.23 24.67 37.35 0.18 0.12 12.68
BQMall 832x480 60 600 22.12 22.12 22.23 22.82 0.70 0.69 0.58
26 © 2015 Gachon University. All rights reserved.
Selective Video Streaming
• Rate Adaptation
• Quantization Parameter (QP) Control
• Transcoding
• Selective Video Streaming•Packet (Bitstream) Drop (Temporal / Spatial Subsampling)
•SVC Layer-switching for Rate Adaptation
27 © 2015 Gachon University. All rights reserved.
Selective Video Streaming (2)
• Layer-switching for SHVC-based DASH• We cannot adapt current bitrate during a segment (e.g. 10 sec.)
• Important factor for precise rate-adaptation
1.Number of layers (e.g. 256kbps, 512kbps, 1mbps)
2.Length of segment (e.g. 5-10 sec.)
• Frequent feedback from clients is a big burden to a VSP server
28 © 2015 Gachon University. All rights reserved.
Part 3Video Streaming Applications- Video Streaming over Multiple Network Interfaces
- Power Aware HEVC Streaming with DASH
- Three-Screen TV using SVC Platform
- Telehealth and Haptic Video Streaming for the Individuals with Visual
Impairments
29 © 2015 Gachon University. All rights reserved.
Video Streaming over Multiple Network Interfaces
• Application-level• Gain: 0.7dB ~ 1.6dB in PSNR
• Transport layer-level
• Link-level
Network
Media Server Client
Smart
Channel
Allocation
• Eun-Seok Ryu, Sung Won Han, “Priority-Based Selective H.264 SVC Streaming Over Erroneous Converged Networks”, Multimedia Tools and Applications, Vol. 68, No. 2, pp 337-353,Jan. 2014.
• Eun Seok Ryu, Hye-Soo Kim, Sungjun Park, Chuck Yoo, “Priority-Based Selective H.264 SVC Video Streaming Over Erroneous Multiple Networks”, Proceedings of The 2011 IEEEInternational Conference on Consumer Electronics (ICCE), pp. 337-338, Jan 2011.
• Eun-Seok Ryu, Jung-Hwan Lee, Hyuck Yoo, "Extracting and Transmitting Video Streams based on H.264 SVC in a Multi-Path Network", Journal of the Korean Institute of InformationScientists and Engineers, Vol. 35, No 6, pp. 510-520, Dec. 2008.
• Eun-Seok Ryu, Nikil Jayant, “Video Streaming over Multiple Network Interfaces by using H.264 SVC”, Poster Session of Georgia Tech Broadband Institute Advisory Board Meeting, Oct.2008.
• Eun-Seok Ryu, Jung-Hwan Lee, Chuck Yoo, “Content-aware and Network-adaptive Video Streaming over Multiple Network Interfaces”, Proceedings of the 2008 InternationalConference on Ubiquitous City Technology (ICUCT 2008), pp. 43-49, Feb. 2008.
• Eun-Seok Ryu, Jung-Hwan Lee, and Chuck Yoo. “Multi-Channel based Scalable Video Streaming for Evacuation Guidance System” , International Conference On ConvergenceInformation Technology, November. 2007.
30 © 2015 Gachon University. All rights reserved.
Power Aware HEVC Streaming with DASH
• Complexity-based content variables (different tools using computational complexity statistics)
• Power (battery) measurement
• Content selection using feedback message
MF
Fragment
Complexity
aware
encoder
Complexity
aware
MPD
Media file
generation
File
Packager
PREPARATION
Media
File
Origin
Storage CDN
DISTRIBUTION
MPD
File
HTTP
Cache
HTTP
Server
Decoder
Power
sensing
Receiver
CONSUMPTION
Power aware client
Power
aware
adaptation
control
Bandwidth
sensing
Request
Application
Complexity
statistics
and control
Quality
Bitrate
Low complexity mode
Medium complexity mode
High complexity mode
Low resolution Medium resolution High resolution
*Yuwen He, Markus Künstner, Srinivas Gudumasu, Eun-Seok Ryu, Yan Ye, Xiaoyu Xiu, “Power Aware HEVC Streaming for Mobile”, Visual Communications and Image Processing
(VCIP) 2013, Nov. 17. 2013.
31 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (1/9)
Introduction:
• SW-based gateway for 3STV• Low storage in server (20% lower)• Real-time SVC decoding and displaying (3-layered 1080p SVC @PC)• Error Resilient
Raptor FEC over
SVC Bitstream HDTV
Laptop
Mobile
Settop Box
Wi-Fi AP
Three Screen-TV using SVC Platform
VSP Server
Wi-Fi network
Home Gateway
(SVC Server)
VSP network
1 Spatial Layer : 640 x 360
2 Spatial Layer : 1280 x 720
3 Spatial Layer : 1920x1080Ethernet or
Wi-Fi network
SVC decodingRaptor FEC en/decodingVideo Content Multicast to Home
32 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (2/9)
Proposed Key Technologies:
• Adaptive overhead control of Raptor FEC
• Real-time BW adaptation using SVC layer-switching
Feedback-based SVC Layer Switching
MSGMSG
• Switches SVC layers according to client’s ABW
• Uses RSS I (Received Signal Strength Indicator)-based channel measurement model
• Pros : Light-weight rate control• Post processing in home gateway
• Eun-Seok Ryu, Nikil Jayant, “Home Gateway for Three-Screen TV Using H.264 SVC and Raptor FEC”, IEEE Transactions on Consumer Electronics (TCE), Vol. 57, No. 4, pp. 1652-1660, Dec. 27. 2011.
• Eun-Seok Ryu, Sung Won Han, “Two-Stage EWMA-based H.264 SVC Bandwidth Adaptation”, IET Electronics Letters, Vol. 48, No. 20, pp. 1271-1272, Sep. 27. 2012.
33 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (3/9)
Raptor FEC Overhead Adaptation:
Meaning : (Example)
Video Seq. : H.264 SVC (Layer 3) for HDTV
(25 frame/sec., 41.48dB)
- Orig. Bitrate : 3.03Mbps
- Rapt. Bitrate : 4.24Mbps (40% OH)
➢Avg. PSNR = 41.48 dB in 25% PLR
Control
OH
Network Condition (PLR)
Feedback
50
60
70
80
90
100
5% 10% 15% 20% 25% 30% 35% 40%
Err
or Reco
very
Packet Loss Rate (%)
0%
10%
20%
30%
40%
50%
Overhead
PLR (%)Overhead (%)
0% 10% 20% 30% 40% 50%
5% 97.18 99.9 100 100 100 100
10% 90.65 96.82 99.81 99.97 100 100
15% 84.84 89.48 99.64 99.90 99.87 100
20% 80.16 80.91 91.33 98.21 100 100
25% 75.91 76.75 78.70 93.64 99.84 100
30% 68.67 69.74 69.97 75.23 92.89 99.22
35% 64.12 65.45 64.29 65.39 69.61 94.35
40% 58.96 60.68 60.88 59.61 62.27 66.10
Experimental Results of Raptor Error Recovery
Refer the Table
Complexity : (Linear decoding complexity)➢Avg. En/decoding Time = 3.0 ms / frame
34 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (4/9)
Packetizing SVC NALs:
34
35 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (5/9)
System Architecture and Implementation:
Server Client
36 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (6/9)
Experimental Results 1:
Packet loss rate (PLR)
(a) 3% (b) 5% (c) 10% (d) 15%
SVC without FEC (forward error correction)
Proposed Method: SVC with Raptor FEC (with 20% overhead)
Packet loss rate (PLR)
(a) 3% (b) 5% (c) 10% (d) 15%
37 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (7/9)
Experimental Results 2:
Moving traces with mobile TV on building map.
Gain: Enhanced the video qualityfrom 2dB to 5dB in PSNR.
Home gateway in office
38 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (8/9)
Online Video Demo:
• H.264 SVC streaming for three-screen TV :
http://www.youtube.com/watch?v=qKdO01Nf-14
• H.264 SVC with Raptor FEC (Fountain code) :
http://www.youtube.com/watch?v=LIBoSjOlpuY
• H.264 SVC layer switching :
http://www.youtube.com/watch?v=gkCL4mG5VYA
39 © 2015 Gachon University. All rights reserved.
Three-Screen TV using SVC Platform (9/9)
Conclusion:
• Home Gateway for 3-screen TV• SW-based RT home gateway for 3STV
• Low storage in server (28% - 36% reduced bitrates )
• Low computational complexity (3-layered 1080p SVC @PC)
• Proposed method showed distinguished performance (2dB-5dB in PSNR)
• Home Gateway for 3DTV (on-going research)• Using JMVC reference tool
• Two factors for efficient coding•Interview redundancy : correlations between views•Psycho-visual redundancy : The suppression theory of human visual perception of 3D stereoscopic video allows sub-sampling of one of the views
39
40 © 2015 Gachon University. All rights reserved.
Telehealth: Robust Medical Video Transmission
• Introduction
• Telesurgery: enable the surgical operation with the
collaboration of local and remote surgeons by using
medical video streaming•On cruise ship in the ocean
•On ambulance in the battlefield for wounded soldiers
• Isolated static site
Telesurgery with 3D-medical image
Da-Vinci surgical robot system
41 © 2015 Gachon University. All rights reserved.
Haptic Video Streaming for the Individuals with Visual Impairments (with GT & GWU)
• Over 39 million individuals are legally blind.
• No video conferencing system for visually impaired person exists today.
• We propose a novel video conferencing system (VCS) for visually impaired.
• In addition to being able to hearing, the blind can now feel consecutive 3D images by
using a combination of Kinect (or 2D camera with depth-map generation) and Haptic
(or 3D Braille) device.
• Research results could be extended to the smart guidance system for the
blind and telesurgical system.
*Chung Hyuk Park, Eun-Seok Ryu, Ayanna M. Howard, “Telerobotic Haptic Exploration in Art Galleries and Museums for Individuals with Visual Impairments”, IEEE Transactions
on Haptics (accepted), 2015
42 © 2015 Gachon University. All rights reserved.
Conclusion
• Next Generation Video Streaming System Using Scalable High Efficient
Video Coding (SHVC)
• HEVC / Scalable HEVC (SHVC)
• Error Resilient Video Streaming•Error Protection with Raptor Codes
•Error Concealment with ECMS
•Selective Streaming / UEP
• Three-screen TV using Scalable Video Coding
Video
Encoder
Error
Protection
Selective
Scheduler
Channel Prediction
Network
MANE
(smart router)
Edge server
Home
gateway
QoS Controller(EC mode selection)
signal
data
Video Client
(Decoder with EC)
data &
signal
43 © 2015 Gachon University. All rights reserved.
Thank You !
Personal webpage: https://sites.google.com/site/hoperyu/
Research blog: http://r2d2n3po.tistory.com/
Questions > [email protected]