Parallel and Distributed Audio (Image/Video) Concealment Using Nonlocal Sparse Representation
description
Transcript of Parallel and Distributed Audio (Image/Video) Concealment Using Nonlocal Sparse Representation
Parallel and Distributed Audio (Image/Video) Concealment Using
Nonlocal Sparse Representation
Xin Li
LDCSEE, WVU
Multimedia: a combination of content forms
Neural Basis of Media Content
“Missing Data” Problem
… … … …
x
kx
1x
q
p
p
packet loss
Sequential Parallel
Skype MSN
Error Concealment in Image Communication
videoimage
Sequential Parallel
Why Parallel?
• Avoid choosing scanning order and its associated error propagation problem
• Modeling multimedia signal at a patch (e.g., speech frame, image block) level instead of data (e.g., speech sample, image pixel) level
• Parallelism has to work with distributed processing together (redundancy is exploited to fight against uncertainty)
Prior Model in the Patch Space
MRPx
xyz
zy
xy
zManifold constraint of sensorysignals: dimensionality of localsubspace is <<patch dimension p
Sparsity-based Prior Model
k
tttw
10 BB
Nonlinear Dimensionality ReductionBy Locally Linear Embedding (LLE) Roweis and Saul, Science’2000
0WD
},...,,1{ 1 kwwdiag W],...,,[ 10 kBBBD
Sparsifying transform
B0 B3 B4B1B2
Nonlocal Sparse Representation
0FD
Approximated solution(FFT/DCT)
0WD
Optimal sparsifyingtransform (KLT)
B0 BkB1 Pack into(n+1)-D Array D (n+1)D-FFT
Thresholding
…
(n+1)D-IFFTPack into
(n+1)-D Array D
B0 BkB1…^ ^ ^
n-D
n-D
Why is it Nonlocal?
B4
B2
B3
B0
B1
RPP
Local neighborsin range (photometricsimilarity)
nonlocal patchesin domain (geometricproximity)
Algorithm Flowchart
Projectiononto
observationconstraint
Projectionontoprior
constraint
Data clustering
in Rp
PDACalgorithm
Better estimate of local neighborhood in the patch space
Inner loop
Outer loop
Better estimate of missing data in the time domain
Experimental Results (I)
sequential-I (parametric)
sequential-II (nonparametric)
parallel (this work)
Experimental Results (II)
outer loopproceeds
Extension into Images
27.06dB 29.14dB 30.16dB
original damagedexemplar
basedGuleryuz’s
schemeThiswork
18.06dB 17.57dB
35.72dB 17.93dB 37.37dB
22.36dB
26.75dB 24.63dB 31.84dB
original damagedexemplar
basedGuleryuz’s
schemeThiswork
26.52dB 25.84dB 33.08dB
28.78dB 28.09dB 34.26dB
21.21dB 14.66dB
17.83dB 17.66dB
26.58dB
20.24dB
Parallel and Distributed Processing (PDP)
• Dominant form of Connectionism (models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units.)
• More widely known as neural networks since 1980s
Network Model Interpretation
Y
X
B
T
X1
B1X2
B2
B3
B4
I
Belief propagation in factor graph
observation
scene
patch
hidden