Improving analysis and performance of modern error-correction schemes: a physics approach...
-
Upload
christiana-daniel -
Category
Documents
-
view
214 -
download
1
Transcript of Improving analysis and performance of modern error-correction schemes: a physics approach...
Improving analysis and performance of modern error-correction schemes:
a physics approach
Phys.Rev.Lett. 93, 198702 (2004)IT workshop, San Antonio 10/2004CNLS workshop, Santa Fe 01/2005Phys.Rev.Lett. 95, 228701 (2005)arxiv.org/abs/cond-mat/0506037
arxiv.org/abs/cs.IT/0507031IT workshop, Allerton 09/2005
arxiv.org/abs/cs.IT/0601070arxiv.org/abs/cs.IT/0601113
Misha Chertkov (Theory Division, LANL)
Vladimir Chernyak (Department of Chemistry, Wayne State)Misha Stepanov (Theory Division, LANL)
Bane Vasic (Department of ECE, University of Arizona)
arxiv.org/abs/cond-mat/0601487arxiv.org/abs/cond-mat/0603189
Analyzing error-floor for LDPC codes Understanding Belief-Propagation:Loop Calculus
MC,VC
towards improving Belief Propagation
UoC,04/10/06
Menu:(first part)
• Analogous vs Digital &• Analogous Error-Correction &• Digital Error-Correction &• LDPC, Tanner graph, Parity Check &• Inference, Maximum-Likelihood, MAP &• MAP vs Belief Propagation (sum-product) & • BP is exact on the tree &• Error-correction Optimization &• Shannon-Transition &• Error-floor &
Introduction
• Instanton method – the idea &• Instanton-amoeba (efficient numerical method) &• Test code: (155,64,20) LDPC &• Instantons for the Gaussian channel (Results) &• BER: Monte-Carlo vs Instanton &
• Conclusions &• Path Forward &
Instanton: proof of principles test
Analogous vs digital
Analogous Digital
continuoushard to copy
discreteeasy to copy
0111100101
camera picturemusic on tape
typed text computer file
real number better/worse
integer number yes/no
menu
Error-correction for analogous
One iteration 4 iteration
16 iteration clean menu
menu
menu
L
L
N
N
Digital Error-Correction
Coding
Decoding
N > L R=L/N - code rate
)|()|( )(
1
)()()( ini
N
i
outi
inout xxpxxP
22exp)|(
222
syxs
yxp
channelwhite
Gaussian symmetricexam
ple
Nxxx ,,1
)()()( inoutin xxxnoise
menu
Low Density Parity Check Codes
menu
N=10variable nodes
M=N-L=5 checking nodes
Parity check matrix
0
0
0
0
0
10
9
8
7
6
5
4
3
2
1
v
v
v
v
v
v
v
v
v
v
H mod 2
Tanner graph
M
ii
ii x
1
1,
112
“spin” variables -
- set of constraintsM
Ni
,,1
,,1
(linear coding)
Parity check matrix(155,64,20) code
Tanner graph(155,64,20) code
menu
Inference
Given the detected (real) signal ---
To find the most probable (integer) pre-image ---
outx
inx
)|~(argmax ][~ outinallowedx xxPdecodingin
Maximum-Likelihood (ML) Decoding
menu
Decoding (optimal)
N
kkk
M
ii hhFhZ
11}{
exp1,)(exp)(
)( )(outxh “magnetic” field
log-likelihood
)|( )()( inout xxP
constraints
“free energy”
“partition function”
(symbol to symbol) Maximum-A-Posteriori (MAP) decoding (close to optimal)
)()( hmsignhoutput j
Efficient but Expensive:requires operationsL2
hhFhm
)()(
“magnetization”=a-posteriori log-likelihood
Stat Mech interpretation was suggested byN. Sourlas (Nature ‘89)
To notice – spin glass (replica) approach for random codes:e.g. Rujan ’93, Kanter, Saad ’99; Montanari, Sourlas ’00; Montanari ’01; Franz, Leone, Montanari, Ricci-Tersenghi ‘02 menu
Sub-optimal but efficient decoding
i
jii
j
jj
i
jii
j
jj
hm
h
tanhtanhtanh
tanhtanh
1
1 Belief Propagation (BP=sum-product) Gallager’63;Pearl ’88;MacKay ‘99
=solving Eqs. on the graph
it
i
i
ji
ti
j
jt
j
h
h
)(
)(1)1( tanhtanh
Iterative solution of BP= Message Passing (MP)
Q*m*N steps instead of Q - number of MP iterations
m - number of checking nodes contributing a variable node
L2
What about efficiency? Why BP is a good replacement for MAP?
* (no loops!)
menu
Tree -- no loops -- approximation
}{}{
1
}{
}{}{
1
}{
11}{
exp1,)(
exp1,)(
exp1,)(exp)(
kkk
iij
kkk
iij
N
kkk
M
ii
hhY
hhX
hhFhZ
j
j
i
jii
j
jj h tanhtanh 1
2/)/ln(
)()(2
1)exp(
)()(2
1)exp(
jjj
i
ji
i
jiiiii
j
jj
i
ji
i
jiiiii
j
jj
XY
YXYXhY
YXYXhX
MAP
BP
Belief Propagation is optimal (i.e. equivalentto Maximum-A-Posteriori decoding) on a tree (no loops)
Analogy: Bethe lattice (1937)
Gallager ’63; Pearl ’88; MacKay ’99Yedidia, Freeman, Weiss ‘01 menu
Bit Error Rate (BER)
)1|()( )()()(
outouti
outi xPxmxdB
measure of unsuccessful decoding
Probability of making an error in the bit “i”
{+1} is chosen for the initial code-word
probability density for givenmagnetic field/noise realization
(channel)
Digital error-correction scheme/optimizationDigital error-correction scheme/optimization
1. describe the channel/noise --- External2. suggest coding scheme3. suggest decoding scheme4. measure BER/FER5. If BER/FER is not satisfactory (small enough) goto 2
menu
From R. Urbanke, “Iterative coding systems”
SNR, s
BE
R, B
Shannon transition/limit
menu
Error floor (finite size & BP-approximate)
Error floor prediction for some regular (3,6) LDPC Codes using a 5-bit decoder. From T. Richardson “Error floor for LDPC codes”, 2003 Allerton conference Proccedings.
No-go zone for brute-force Monte-Carlo numerics.
Estimating very low BER is the major bottleneck
in coding theory/practice
menu
Our (current) objective:
For given (a) channel (b) coder (c) decoderto estimate BER by means ofanalytical and/or semi-analytical methods.
Hint:
BER is small and it is mainly formed at some very special“bad” configurations of the noise/”magnetic field”
Instanton approach is the right way to identifythe “bad” configurations and thus to estimate BER!
menu
Instanton Method
Laplace methodSaddle-point method
Steepest descent
1noise
2noise
...noise
errors
no errors
Error-surface (ES)
Point at the ESclosest to zero
menu
BER = d(noise) Weight(noise)instanton config.
of the noiseBER Weight
instanon config of the noise
Point at the ESclosest to zero
Parity check matrix(155,64,20) code
Tanner graph(155,64,20) code
menu
Found with numericalinstanton-amoeba scheme
instanton-amoeba menu
Instantons for (155,64,20) code: Gaussian channel
076.10210
4622 efl 203.10
79
8062 efl 298.10188
4422 efl
Phys. Rev. Lett -- Nov 25, 2005
menu
menu
We suggested amoeba-instanton method for efficient numerical evaluation of BER in the regime of high SNR (error floor). The main idea: error-floor is controlled
by only a few most damaging configurations of the noise (instantons).
Conclusions (for the first part – error floor analysis)
Results of the amoeba-instanton are successfully validatedagainst brut-force Monte-Carlo (in the regime of moderate
SNR)
menu
Path Forward Extend the amoeba-instanton test • to study the error-floor • to develop universal computational tool-box for the error-floor analysis Other codes Other decoding schemes (e.g. number of iterations)
Other channels (e.g. magnetic recording and fiber-optics specific)
Major challenge !!!! – to improve BP qualitativelyNew decoding ?!
New coding ?! Efficient (channel specific) LDPC optimization
Inter-symbol interference + noise (2d and 3d + error-correction)
Distributed coding, Network codingCombinatorial optimization menu
Understanding Belief Propagation
Questions:
• Why it works so well … even when it should not? -- BP is gauge fixing condition
• Can one constructs a full solution (MAP) from BP? -- yes one can!/loop series
Making use of the loop calculus/series
Improving BP – approximate algorithms • LDPC decoding• SATisfiability resolution• Data reconstruction• Clustering• etc
Answers:arxiv.org/abs/cond-mat/0601487arxiv.org/abs/cond-mat/0603189
first slide
Vertex Model
aXa
afZ
}{
Xa
aafZp 1)(
Partition function
Probability
},{ ia
otherwise
if ii
ii,0
,,,1)(
Reduction to bipartite graph (error-correction):
i i
ii
ii
ii q
hf exp1,exp1,)(
),(
),,(
),,,,,,(
1
13122
1814121
453423181412
baab
Ising variables on edges
}{edgesX
improving BP
Bethe Free Energy --- Variational Approach
Generalization ofYedidia, Freeman,Weiss ‘01
0bF
Constraints(introduce in minimizationthrough Lagrange multipliers)
Belief Propagation (Bethe-Peierls) equations
acacca
acacaaa
aaaaa
aa bbbbfbFacaa
lnlnln),(
self-energy entropy entropy correction
improving BP
)()()(:;,
1)()(:;,
1)(,)(0:;,
\\ccaaacac
acacaa
acacaa
bbbacca
bbacca
bbacca
accaca
aca
Loop series
abb --- beliefs (prob.) calculated within BP !
BetheFZ ln0 •BP is special, not only without loops!•Gauge invariant representation!
=C
improving BP
•integral representation• algebraic representation• gauge representation
Three alternative derivations:
Loop series (derivation #1)
“vertex”
“propagator”
ab --- gauge degrees of freedom (at our disposal !)
},,{' 2112
improving BP
a cb
cbbcaa
aaa ffZ
),(' 2
1)()(
coshsinhcoshsinh1
cosh
sinhcoshsinhcosh1*1
cbbcbc
cbcbcbbcbcbcbccbbc
V
V
cbbc
ababbaabaaaa
cbbc
aa
cb
fP
VPZ
)sinh(cosh)()(
cosh2),('
1
),(
Loop series (derivation #2)
' ),(
~ a cb
bca VPZ
.
""
**1
contribcolored
** … *
Expand the “vertex” (edge) term
Calculate resulting terms one-by-one
• Each node enters the product only once• Node is colored if it contains at least one colored edge
Gauge fixing condition:
To forbid “loose end contribution” for any node !!
ab --- gauge degrees of freedom (at our disposal !)
improving BP
Loop series (derivation #3)
fixing the gauge!!to kill loops
Belief Propagation !!
equations
Loop series has just been derived!!
improving BP
Future work
•Approximate algorithms --- leading loop, next after leading,.. --- apply to LDPC decoding --- different graphs, lattices• Generalization --- Ising Potts (longer alphabets) --- continuous alphabets (XY,Heisenberg,Quantum models) first slideimproving BP
Conclusions ( for the second part – Understanding/Improving BP)
• Loopy BP works well because BP is nothing but GAUGE FIXING condition
• Simple finite series --- LOOP SERIES --- for MAP is constructed in terms of BP solution
Instantons on the tree (semi-analytical)
menu
PRL 93, 198702 (2004) ITW 2004, San Antonio
m=2, l=3, n=3 m=3, l=5, n=2
Instanton-amoeba (efficient-numerical scheme)
)1|)(1()1|)(1(max~ **
efui uPuPB
0))((
)()(
*0
**
um
uluu
To minimize BER with respect to the unit vector !!
error-surface
unite vector in the noise space
Minimization method of our choice is simplex-minimization (amoeba)
)(
)1|)(1(maxarg
*
*
efef
uef
ull
uPu
menuinstanton-amoeba for Tanner code
Different noise models for different channels
)|()|( )(
1
)()()( ini
N
i
outi
inout xxpxxP
ii
ii x
xp
xp
sh
ssxp
)1|(
)1|(log
2
1
22exp)1|(
2
222
White
Gaussian
)(111 )()( uluxx outin Linear
)|()|( xypyxp Symmetric
simplifications
Laplacian
i
ii
i
i
ii xp
xp
sh
ssxp
0,1
02,1
2,1
)1|(
)1|(log
2
1
exp)2/()1|(
menu
Rational structure of instanton (computational tree analysis/explanation) min-sum
4 iterations
based on Wiberg ‘96
Phys.Rev.Lett. 95, 228701 (2005)
Minimize effective actionkeeping the condition
menu
Bit-Error-Rate: Gaussian channel
menu
Instantons for (155,64,20) code: Laplacian channel
6.7efl 0.8efl 0.8efl
menuIT workshop, Allerton 09/2005
Instantons as medians of pseudo-codewords
menuPRL -- Nov 25, 2005
Bit-Error-Rate: Laplacian channel
menu