QoE Metric and Cross-Layer Optimization for Video …zli/new_home/pub/samsung.qoe-wireless...QoE...
Transcript of QoE Metric and Cross-Layer Optimization for Video …zli/new_home/pub/samsung.qoe-wireless...QoE...
QoE Metric and Cross-Layer Optimization for Video over Wireless Networks
1 Z. Li, 2012
Zhu Li
Outline
• Introduction• Research Motivations• A brief overview of my research
– Video QoE metric– Multi-Access Wireless Video Streaming– Wireless Video Broadcasting– P2P Video Networking
• In depth discussion
2 Z. Li, 2012
• In depth discussion – Multi-Access Video Networking,
» Temporal QoE metric and VLBR video adaptation, » NUM formulation and resource pricing scheme» Application in Multi-Access Video Networking» Simulation Results
– Source Channel Coding in Video Broadcasting (extra)
• Summary & Questions
About Me: http://users.eecs.northwestern.edu/~zli
• Bio:– Sr. Staff Researcher, FutureWei Technology, Bridgewater, NJ. – Asst Prof, HK Polytechnic Univ, and CTO, Mudi Technology,
2008.04~2010.10.– Senior, Senior Staff, and then Principal Staff Researcher, Multimedia
Research Lab, Motorola Labs, USA, 2000-08.– Software Engineer, CDMA Network Software Group, Motorola CIG, USA,
1998-2000. – PhD in Electrical & Computer Engineering, Northwestern University,
3 Z. Li, 2012
– PhD in Electrical & Computer Engineering, Northwestern University, USA, 2004.
– IEEE Senior Member, elected vice chair, IEEE Multimedia Communication Tech Committee , 2008~2010.
• Research Interests:– Video Coding and Adaptation, Optimization and Distributed Computing in
Video Networking with applications in mobile TV, wireless video on-demand streaming, and P2P video networking.
– Image/Video Analysis, Machine Learning and applications in Scalable large video repository search and mining problems.
2012
4 Z. Li, 2012
Devices
• Explosive growth of devices:– Billions of cell phones/PDAs– Billions of computers– Billions of TVs– Billions of Media Players
• Different Multimedia
5 Z. Li, 2012
• Different Multimedia Capabilities in:– display, – capture, – storage,– computing, – communication
Networks
• Better technology from equipment makers– Better wireless spectrum efficiency,
WiMAX/LTE– High speed DLS/Cable, 100x100– Fiber optical solutions, GPON
• More capacity from service providers
6 Z. Li, 2012
providers– More bandwidth, better coverage, – Convergence of data, voice and
media service from service providers
– Vertical integration of application and services
Content
• Explosive growth of digital media– Web, Email, Audio, Video, Game– News, Music, Movie, Talk show,
Game, 2nd Life.
• Rapid changes in the way contents are produced and consumed
7 Z. Li, 2012
consumed– Personal vs Commercial– Passive (TV) vs Interactive (Blog,
Game)– Centralized vs P2P
Key Challenges
• Application Needs:– Good Access, be able to get what you want, a storage and communication
problem– Mobility across devices and access sessions: anywhere, on any device,
not tied to a single device/location, get what they want, with good media quality (coding) and availability (communication/networking).
– Intelligence and Personalization: be able to utilize the “big data” collected from both networks and applications, and drive more intelligent and
8 Z. Li, 2012
from both networks and applications, and drive more intelligent and personalized applications
Technology Gap
Technology Gap ?– Video Communication:
» Rapidly widening gap between wireless capacity and explosive growth of mobile video traffic, 14 times in 2015 according to Verizon.
» Re-engineering the Internet for video traffic– Video Storage:
» Storage of multimedia content in cloud – explore various error-resilience and rate-distortion tradeoff characteristics to enable differential storage service.
– Video Computing:
9 Z. Li, 2012
– Video Computing:» Web scale multimedia analysis, indexing and retrieval, » Mobile search
Research and Biz Opportunities
Communication is Computing:– Computing and storage are cheaper than wireless spectrum. Use
computing and storage to mitigate the wireless gap.– Smarter networks: cognitive radio, cross layer design, employing richer
video QoE metric that enhances traffic elasticity and robustness– Smart caching and on-demand transcoding and adaptive streaming within
the network– Emerging standardization effort, eg. MPEG MMT.
10 Z. Li, 2012
Relevant Research Projects Highlights
11 Z. Li, 2012
Project Highlights – QoE Metrics and VLBR Video
• Video Summarization for VLBR video streaming– Developed a frame drop induced distortion metric– Offering a content aware way of temporal distortion adaptation
scheme that can drive the operating bit rate for QCIF video down to PCM voice rate range
– Dynamic Optimization in frame selection
35
36
VLBR Video Examples: R−Davg−Dmax−PSNR Performance
40
50
60summary frames
12 Z. Li, 2012
10 20 30 40 50 60 7029
30
31
32
33
34
35
R (kpbs)
PS
NR
(dB
)
Davg=31.9, Dmax=79.9D
avg=26.1, D
max=59.9
Davg
=49.4, Dmax
=99.1
10 20 30 40 50 60 70 80 90 100 110 1200
20
40
60
80
d(f k,
f k)
summary distortion
λ= 16.0e-4, D(S)=24.6, R(S)=80.8kb
0 20 40 60 80 100 1200
10
20
30
40
d(f k,
f k-1)
Project Highlights – Video Communication
• Intelligent Mobile TV– Supporting highly elastic and robust
QoS for a variety of mobile terminals with graceful visual quality degradation with channel conditions
– Practical frame-size and frame rate adaptation for wide codec support.
– PHY layer adaptation through DE-STC (diversity embedding space-time coding) to induce a set of embedded
13 Z. Li, 2012
coding) to induce a set of embedded channels that best suit the multicast group channel distribution
– APP layer source-channel coding optimization with layered video and digital fountain code
– Targeting in-band, or dual-band (WiFi + 3G) wireless infrastructure.
Project Highlights – Video Communication
• Next Generation Content Networks – Video accounts for > 70% of internet traffic now– P2P video networking, utility based on minimizing freezing in playback,
developed a gradient based solution. (collaboration with Profs. Chiang and Calderbank at Princeton).
– Optimization and distributed computing solutions in IPTV video networking, Primal-Dual decomposition, resource pricing schemes for multi-access and IPTV video networks, (RGC New Staff Grant, in collaboration with Prof. Mung Chiang’s EDGE lab at Princeton)
14 Z. Li, 2012
Prof. Mung Chiang’s EDGE lab at Princeton)– Video TCP: TCP re-engineering for content delivery networks. Reconsider
the congestion measure and pricing as well as source adaptation schemes in TCP to better suit for content intensive traffics.
– Caching and Network Coding schemes for video sharing in Mesh Networks (in collaboration with Prof Cao at HK PolyU).
Project Highlight - Mobile Search• Video Based Mobile Location Search:
– Location search by mobile video capture and query
– Video SIFT points indexing with appropriate scale and spatio-temporal quality metrics
– Fast search with multi-indexing of SIFT point sets. – See my recent ACM MM paper for more detail.
• Image/Video Based Mobile Product Search :– Query-by-Capture, when shopping in the malls,
15 Z. Li, 2012
– Query-by-Capture, when shopping in the malls, just took a picture/video of the product and search the online stores to compare prices.
– Novel Image Fingerprint indexing solution– Gone commercial with dangdang.com (amazon in
China) , can show a demo if interested.
Temporal QoE Metrics and Resource Pricing Control i n Multi-Access Wireless Video Networking
16 Z. Li, 2012
Mixed Voice/Video over CDMA Up Link
Radio tower
• Mixed QoS requirements for Video/Voice traffics
• Limited resource, video has to operate at VLBR
• Shared radio resource, and interference limited capacity, Ri=f(Pi;P-i),
17 Z. Li, 2012
Ri=f(Pi;P-i), • Diversity of channel gains and
source rate-distortion characteristics among users.
• How to optimize video adaptation and transmission to achieve better QoS and radio resource efficiency ?
A General Formulation
• Total utility maximization subject to a shared resou rce constraint,
– Where utility function Ui() is a concave differentiable function reflecting the
max,,,
..,)(max21
xxtsxUi
ii
iixxx n
≤∑∑L
18 Z. Li, 2012
– Where utility function Ui() is a concave differentiable function reflecting the quality-bit rate/resource trade-offs. (true for most video source’s PSNR-R function)
– Difficult to solve the primal problem by allocating {xi} directly, because of coupling of {xi} in constraint.
– Difficult to have all utility information for all mobile users, – Transform the problem for a distributed solution, utilizing computing
capability at mobiles
Distributed Solution of the Dual Problem
•Lagrangian relaxation:
•The dual problem:
•Decomposed into n separable video adaptation problems at mobiles :
]}))(([max{min max,,,0 21
xxxUi
iiixxx n
λλλ
+−∑≥ L
))((max xxxU λλ +−∑
)()(),,,,( max21 xxxUxxxLi
ii
iin −−= ∑∑ λλL
19 Z. Li, 2012
•And a base station resource pricing problem:
))((maxarg*
))((max
))((max
,,,
max,,,
21
21
iiix
i
iiii
xxx
iiii
xxx
xxUx
xxU
xxxU
i
n
n
λ
λ
λλ
−=⇔
−⇔
+−
∑
∑
L
L
∑≥i
ig )(min0
λλ
)()()( max** xxxUg iiii −−= λλ
BTS Mobile i
Announce resource price in iteration kkλ
Mobile optimization:
Protocol for Distributed Optimization
))((maxarg* ik
iix
i xxUxi
λ−=
Report back resource used xi* in iteration k
Increase price, if Otherwise decrease price
∑ >i
i xx max
kkk αλλ ±=+1 Mobile optimization:
))((maxarg* 1k xxUx +−= λ
20 Z. Li, 2012
))((maxarg* 1i
kii
xi xxUx
i
+−= λ
Distributed Optimization for Multiple Access Video Network
• Geometrical Interpretation on price:– From the Karush-Kuhn-
Tucker (KKT) condition:
– Allocations {x *} will have
*** ,,0 λ−=
∂∂
⇒=∂∂
i
i
i x
U
x
L
U1(x1)
U2(x2)
U3(x3)
U1(x1)
U2(x2)
U3(x3)
21 Z. Li, 2012
– Allocations {xi*} will have
the same marginal utility (slope) as -price.
– Optimal price must also be tight on all available resource.
U1(x1)
x1* x2
* x3*
U1(x1)
x1* x2
* x3*
Video Over Multiple Access Channel
• In solving real world problems with this distribute d pricing scheme:– Source coding: scalability, adaptability issues– Diversity in Channel state– Diversity in content– Collaboration in resource allocation, scheduling– Uplink problem: interference limited
22 Z. Li, 2012
– Downlink problem: power limited. – Computational complexity
CDMA Uplink with Mixed Voice/Video Traffic
• Consider a single cell CDMA uplink:– Pvoice – received power for a voice user– M – total voice users– Pvideo – total received power for all video users– Gvoice - modulation scheme related constant, BPSK = 1, QPSK = 2– W - bandwidth (Hz)– voice - voice QoS minimum SINR γ
23 Z. Li, 2012
voice
• Received Power Constraints:– QoS for voice users:
– Max allowable total received power for video users
,)1(0
voicevoicevideo
voice
voice
voice
PMPWn
P
R
WG γ≥−++
.1 0max WnPMR
WGP voice
voicevoice
voice −
−+=
γ
Problem Formulation
• Control video mobiles’ transmitting power to achiev e social optimality in total received utility (video quality ) :
– Optimization is over a sliding window of size T– Utility (PSNR, e.g) is a function of total rate in T
( ){ } ( )( ) ( ) ],0[,..,max max11 0
01
TtPtPtsdttRUN
jj
N
j
T
jjtP
Njj
∈∀≤
∑∑ ∫
==≥≤≤
P
24 Z. Li, 2012
– Utility (PSNR, e.g) is a function of total rate in T– Total N video users.
• How to solve ?– Spend resource that can give maximum return in quality
» Account for content diversity, each has different R-D curves» Account for channel state diversity,
– Distributed solution
Multiple Access for Dominanting Received Power Users
• Time-Division Multiplexing (TDM) is needed among vi deo users– Video users’ received power too strong for spectrum efficiency – Example: 4 video users’ achieve able total rates plot:
2000
2500
3000
Vid
eo u
ser’s
tota
l ach
ieva
ble
rate
(kb
ps)
GREEDYSIMCONST
25 Z. Li, 2012
– Therefore, we choose TDM among video users.
5 10 15 20 25 30 350
500
1000
1500
2000
Number of voice users
Vid
eo u
ser’s
tota
l ach
ieva
ble
rate
(kb
ps)
Problem Formulation with TDM Among Video Users
• Allocate transmission slots among video users to ac hieve social optimality in total received utility (video quality) :
{ } ( ) ,..,~
max11
01
∑∑==≥
≤≤≤
N
jj
N
jjj
tTttstU
Njj
( ) ( ).~jTDMjjj tRUtU =
26 Z. Li, 2012
– Total time slots {tj} length is T. – RTDM is the rate achieved using Pmax for a single video user, with current
voice traffic load.
Dual Decomposition : Pricing Solution
• The primal problem is difficult to solve.– The problem is convex, since we assume utility functions are convex.– Constraints are also convex– Strong duality exists.
• Dual Decomposition through Lagrangian Relaxation:– Lagrangian:
( ) ( ) ,~
,max 0
−−= ∑∑≥
N
j
N
jj TttUJ λλtt
27 Z. Li, 2012
– Mobile source adaptation surplus problem:
– Base station resource pricing problem:
( ) ( ) ,,max11
0
−−= ∑∑==
≥j
jj
jj TttUJ λλtt
( ) ( )( ).~maxarg jjjlj ttUt
jλλ −=
( )( ),,max 0 λλλ tJ≥
( ) ( ).~jTDMjjj tRUtU =
Video Source Adaptation with Resource Pricing
•The source surplus problem is to maximize pay off a s utility minus cost in resource
–Distributed to each video source, interact with other video users thru the price. –If scalable coded source, optimal bit extraction subject to a price on resource. Utility could be the PSNR quality of the video–For VLBR (e.g. 24~120kpbs), code video frames at very low PSNR is not
( ) ( )( ).~maxarg jjjlj ttUt
jλλ −=
28 Z. Li, 2012
–For VLBR (e.g. 24~120kpbs), code video frames at very low PSNR is not preferable. Use video summarization scheme instead.
( ) ( ) ( )jjjS
j StSDSj
λλ += minarg*
Temporal Video QoE Metric and Video Summarization
•What is video summary ?–A shorter version of the original video that preserves most information.
•Definitions:– n-frame video sequence:
– m-frame video summary:
– reconstruction by repeating last summary frame:
},,{110 −
=mlll fffS L
}',','{' 110 −= nS fffV L
},,{ 110 −= nfffV L
29 Z. Li, 2012
– reconstruction by repeating last summary frame:
– frame loss distortion:
– rate:
}',','{' 110 −= nS fffV L
)',()(1
0jj
n
j
ffdSD ∑−
==
+==
∑
∑∑ −
=
−
=−
=−
coding-inter,
coding-intra,
)()(1
1
1
01
0
1
0
m
t
ll
l
m
t
l
m
tl
t
t
t
t
rr
r
fbSR
Video Summary QoE metrics Examples
n=10, S={f0, f3, f5, f8 } , m=4, D(S)=0.6
1 2 3 4 5 6 7 8 9 100
0.2
0.4
0.6
0.8
1
1.2
1.4
d(f k
, fk-
1)
summary frames
f f f f f f f fffV=
30 Z. Li, 2012
1 2 3 4 5 6 7 8 9 100
0.5
1
1.5
2
d(f k
, fk')
summary distortion
1 2 3 4 5 6 7 8 9 10
f0 f0 f0 f3 f3 f5 f5 f8f8f5
f0 f1 f2 f3 f4 f5 f6 f9f8f7d(f0, f1)
d(f0, f2)
V=
VS’=
Frame Distortion for Summarization: What is a good d(fj, fk) ?
scale PCA.
352x240 video frame 11x8 image icon d-dimensional point
.d(fj,fk)
10
12
31 Z. Li, 2012
“foreman” seq in 2-d (1st and 2nd component) PCA space
1
101 201
301
X1
X2400
50 100 150 200 250 300 350 4000
2
4
6
8
10
frame number k
d(f
k, f’
k)
Surplus problems at mobiles•The adaptation problem:
– compute summary at mobile j, s.t. the following surplus function is maximized,
( ) ( ) ( )
( ) ( )TDM
jj
S
jjjS
j
R
SRSD
StSDS
j
j
λ
λλ
+=
+=
minarg
minarg*
32 Z. Li, 2012
– for the given voice traffic load, RTDM is known, R(Sj) is the bit rate for the resulting video summary.– exhaustive search is exponential in complexity, – the problem has some structure for which we will exploit for a Dynamic Programming solution.
Distortion State and Cost
• Summary Segment Distortion:
• Distortion State Dtk, for summaries with t frames ending with fk,
• Bit cost for Dtk,
∑−
=
++ =
11
1 ),(t
t
t
t
t
l
ljjl
ll ffdG
}{min2
2
1
1
2210
,,
nk
kl
ll
l
lll
kt GGGGD
tt
++++=−
−
LL
−1t
33 Z. Li, 2012
t
• The surplus problem:
∑−
=
=1
0
)(t
jl
kt j
fbR
}{min221 ,,
,
TDM
ktk
tlll
kt
R
RDJ
t
λλ +=−L
The surplus recursion at mobile– To simplify notation, let a new price on bit be,
–The recursion:
++−+++=
++
+++++=
+=
−
−
−−
−++
+
{min
)]}()(
)()([{min
}{min
11
1
1
1
121
121
100,,
11,,
,1
nknnll
kl
nk
kl
l
lll
kt
kt
lll
kt
t
t
tt
t
GGGGGG
fbfb
fbfbGGG
RDJ
λ
λλ
L
L
LL
L
TDMR
λλ =
34 Z. Li, 2012
+−
+−=
++−−
++++=
++++
++−+++=
−
−−
−
−−
−
−
−−
−−−
−
−−−
−
−−
codinginter if},{min
codingintra if},{min
)}(])([
)]()([{min
)}()]()()([
{min
1
11
1
11
1
,1
11
11
1
121
1
111
1
2
1
121
,,
,,
00,,
10
0,,
kl
kllt
l
kkllt
l
k
e
nk
kl
nl
lnl
l
lll
kl
nk
kl
nl
nl
ll
l
lll
t
tt
t
tt
t
ktl
tt
ttt
t
ttt
t
tt
reJ
reJ
fbGGG
fbfbGG
fbfbfbfb
GGGGGG
λ
λ
λ
λ
λλ
λ
λ
44 344 21
LL
L
L
L
L
A Viterbi Like Algorithm
3
3.5
4
4.5
5
summarization: λ = 1.50e-004, Kmax=5
J23=9.89
J24=10.11
J25=9.99
J33=10.00
J34=10.04
J35=9.89
J43=10.09
J44=10.15
J45=10.00
J54=10.24
J55=10.09 J6
5=10.24
fra
me
k
– DP solution for surplus maximization under a given price on resource
– Start with first frame– Compute the max surplus
incoming edge at each
35 Z. Li, 2012
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60
0.5
1
1.5
2
2.5
J10=9.99
J21=10.09
J22=9.99 J3
2=10.05
fra
me
k
epoch t
incoming edge at each node
– Backtracking for optimal solution.
Summarization Results
60
80summary distortion
λ= 16.0e-4, D(S)=24.6, R(S)=80.8kb
0 20 40 60 80 100 1200
10
20
30
40
50
60
d(f k,
f k-1)
summary frames
60
80summary distortion
λ=12.0e-4 D(S)=16.8, R(S)=107.2kb
0 20 40 60 80 100 1200
10
20
30
40
50
60
d(f k,
f k-1)
summary frames
36 Z. Li, 2012
10 20 30 40 50 60 70 80 90 100 110 1200
20
40
d(f k,
f k)
10 20 30 40 50 60 70 80 90 100 110 1200
20
40
d(f k,
f k)
=1.6e-5, PSNR=30dB, D(S)=24.6, R(S)=80.8kb
=1.2e-5, PSNR=30dB, D(S)=16.8, R(S)=107.2kb
λ λ
Video Summarization + Transcoding Scheme for VLBR Channels
• Optimally select a subset of frames to code at a hi gher PSNR quality
34
35
36
PS
NR
(dB
)VLBR Video Examples: R−Davg−Dmax−PSNR Performance
– “Foreman” sequence
– Bit rate range: 11.2kpbs ~46.5kbps
– PSNR: 29dB ~
37 Z. Li, 2012
10 20 30 40 50 60 7029
30
31
32
33
R (kpbs)
PS
NR
(dB
)
Davg=31.9, Dmax=79.9D
avg=26.1, D
max=59.9
Davg
=49.4, Dmax
=99.1
– PSNR: 29dB ~ 34.3dB
– R(S)
Demo !
Base Station Price Control Problem
• Base station solves for a price that maximizes total utility
– Achieved through a sub-gradient method, checking for constraint violation at each price iteration:
( )( ),,max0
λλλ
tJ≥
( )( ) .,0max1
1
−+= ∑
=
+ TStN
j
ijj
iii λαλλ
38 Z. Li, 2012
– The sub-gradient search converges if the step sizes:
– In practice, price iteration stops when total utility improvement ratio is below certain threshold.
– Also the time slot allocation need to be schedulable.
1 ∑
=j
0lim =→∞i
i α ∞→∑i
iα
Joint Packet Scheduling for video summary transmission
• Video packets are delay sensitive. – In TDM scheme, we have a GREEDY solution: sort packets by their
deadlines, transmit the nearest deadline ones.
0 10 20 30 40 50 60 70 80 900
5
P1(t)
Received Powers under GREEDY
0 10 20 30 40 50 60 70 80 900
5
P2(t)
39 Z. Li, 2012
– Pricing iteration is actually on schedulability (deadline violations)
0 10 20 30 40 50 60 70 80 900
0 10 20 30 40 50 60 70 80 900
5
P3(t)
0 10 20 30 40 50 60 70 80 900
5
P4(t)
Frame k
( ){ }}0,max{,0max1 iGREEDYii λβλλ ∆+=+
Simulation Results
• Simulation set up:– Channel (IS-95 alike):
Entity Symbol Value Bandwidth W 1.228MHz
Noise density n0 8.3*10-7 mW/Hz
Voice target SINR γvoice 6dB Voice modulation BPSK
Voice received power Pvoice 1mW Voice spreading gain Gvoice 128
Voice rate R 9.6kbps
40 Z. Li, 2012
Video Users:» 4 segments (90 frames each) from “foreman” and “mother-daughter”
sequences» Fixed PSNR: 27.8dB (foreman), and 31.0dB (mother-daughter)
Voice rate Rvoice 9.6kbps Video target SINR γvideo 6dB Video modulation QPSK
Price and Distortion Convergence
• Pricing iteration convergence at base station (left Fig), and summarization distortions at mobiles (right Fig):
0.05
0.06
0.07
0.08
0.09
0.1
Pric
e
15
20
25
Dis
tort
ion
Per
Fra
me
User 1User 2User 3User 4
41 Z. Li, 2012
1 2 3 4 5 60
0.01
0.02
0.03
0.04
Iterations
Pric
e
1 2 3 4 5 60
5
10
Iterations
Dis
tort
ion
Per
Fra
me
Simulation Results
• Resulting video summaries with pricing co-ordinatio n– D(S1)=3.09, D(S2)=6.42– D(S3)=0.76, D(S4)=0.81
10 20 30 40 50 60 70 80 9005
1015
D(S
1)
Summary Distortion
1015
2)
42 Z. Li, 2012
10 20 30 40 50 60 70 80 9005
10
D(S
2)
10 20 30 40 50 60 70 80 9005
1015
D(S
3)
10 20 30 40 50 60 70 80 9005
1015
D(S
4)
Frame k
Simulation Results – Compare with SIMCAST
• Resulting video summaries without pricing co-ordina tion– D(S1)=2.85, D(S2)=31.43– D(S3)=0.059, D(S4)=0.068
10 20 30 40 50 60 70 80 9005
1015
D(S
1) Summary Distortion under SIMCONST
1015
2)
43 Z. Li, 2012
10 20 30 40 50 60 70 80 9005
10
D(S
2)
10 20 30 40 50 60 70 80 9005
1015
D(S
3)
10 20 30 40 50 60 70 80 9005
1015
D(S
4)
Frame k
Performance Summary
• The Performance for CDMA uplink video networking:– In this work we proposed an efficient solution to support mixed voice and
VLBR video traffic that can help seamless migration from 2.5G to 3G and B3G systems
– The solution is distributed, with minimum communication overhead (prices, summary frames) between base station and mobiles
– The computational complexity for source adaptation is distributed among mobiles
44 Z. Li, 2012
mobiles– The solution works well in convergence
Summary of the Multi-Access Video Networking
• The Video Adaptation + Resource Pricing Framework– For multimedia networking problems that can be expressed as a sum
utility (concave) maximization and resource sum constraint(s), this resource pricing with local surplus maximization framwork works.
» E.g, the game traffic shaping work, T-MM, 2008.
– When we have multiple sum resource constraints, e.g, over links in the network, then we have TCP like solution. Eg ,out T-MM 2009 work on CAF
45 Z. Li, 2012
CAF– When constraints can’t be expressed as a simple sum, then need more
advanced solutions like auction, and other game theoretical approaches.
Light Computational Complexity Solution
• Video Cache Pruning Solution– Reduced complexity solution– Provision extra storage and computing resource at base station controller
– Mixed transcoding / packet pruning solution to balance computational complexity
– Offline or online frame drop QoE metric computing:
50
60
46 Z. Li, 2012
20 40 60 80 100 120 1400
10
20
30
40
50
dist
ortio
n
20 40 60 80 100 120 1400
10
20
30
40
50
60
dist
ortio
n
frame k
Light Computational Complexity Solution
• Frame drop distortion gradient priority algorithm– Provision extra storage and computing resource at base station controller
– Mixed transcoding / packet pruning solution to balance computational complexity
– Offline or online frame drop QoE metric computing:
sorted utility gradient
Algorithm 2. Video Queue Prunning
1. Compute Gradient for all n video queues of m frames in the
buffer
47 Z. Li, 2012
05
1015
2025
30
0
2
4
6
80
5
10
15
20
25
frame video seq
dis
tort
ion/
rate
2. FOR k=1:n
a. Find the minimum gradient frame drop for user k,
j
j
mjk r
dg
]..1[min∈
=
b. Select the minimum gradient among users, }{minarg* kk
gk =
c. Drop user k*’s min gradient frame (and associated
decoding dependent frames)
d. Check if the total reduction in rate meet the reque st or not.
Exit if okay.
3. END
Frame Pruning Demo
• Set up 10 mobile users playing back youtube video o f 200~500kbps range
• Simulating base station bottleneck at 10%~40% throu ghput reduction case
8
9
10
dienbienphu−2
dienbienphu−4
dienbienphu−5
48 Z. Li, 2012
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
1
2
3
4
5
6
7
rate reduction ratio
dis
tort
ion
foreman
i16
korolev−4
korolev−5
me109
Demo !
Related Journal Papers
• QoE metric and very low bit rate video:� Z. Li, A. K. Katsaggelos, G. Schuster and B. Gandhi, "Rate-Distortion Optimal Video
Summary Generation", IEEE Trans. on Image Processing, pp. 1550-1560, vol. 14, no. 10, October, 2005.
� Z. Li, G. Schuster, A. K. Katsaggelos, "MINMAX Optimal Video Summarization and Coding", special issue on Analysis & Understanding for Media Adaptation, IEEE Trans. on Circuits and System for Video Technology, pp. 1245-1256, vol. 15, no. 10, October, 2005.
• Wireless Video Networking:� Y. Yang, Z. Li, W. Shi, Y. Chen, and H. Xu, "Cross-Layer Optimization for State Update in
Mobile Gaming", IEEE Trans. on Multimedia, vol. 10(5), pp. 701-710, August, 2008.
49 Z. Li, 2012
Mobile Gaming", IEEE Trans. on Multimedia, vol. 10(5), pp. 701-710, August, 2008. � J. Huang, Z. Li, M. Chiang, and A. K. Katsaggelos, "Joint Source Adaptation and Resource
Allocation for Multi-User Wireless Video Streaming", IEEE Trans. on Circuits & System for Video Tech, vol. 18 (5), pp. 582-595, May, 2008.
� Z. Li, F. Zhai, and A. K. Katsaggelos, "Joint Video Summarization and Transmission Adaptation for Energy Efficient Wireless Streaming", EURASIP Journal on Advances in Signal Processing, special issue on Wireless Video, vol. 2008, May, 2008
� W. Ji, Z. Li, and Y.-Q. Chen, "Joint Source-Channel Coding and Optimization for Layered Video Broadcasting to Heterogeneous Devices", in press, IEEE Trans on Multimedia, 2012.
• Internet and P2P Video Networking:� Y. Li, Z. Li, M. Chiang and A. Robert Calderbank, "Content-Aware Distortion Fair Video
Streaming in Congested Networks", IEEE Trans. on Multimedia, 2009
Future Work
• In the future– Richer yet practical QoE metric for wireless video, characterizing both
signal quality and playback temporal quality.– Effective and flexible suite of signaling and protocols to address multi-
access wireless video for newer generation wireless network stacks like E-UTRA.
– Integration with modern streaming solutions like DASH, Silverlight.
• Collaborators:
50 Z. Li, 2012
• Collaborators:– Prof. Mung Chiang, Princeton University – Prof. Robert Calderbank, Princeton University– Prof. Jianwei Huang, Chinese University of Hong Kong
Questions on Video Networking Research
?…
51 Z. Li, 2012
…Thanks!