Channel Coding Methods for Emerging Data Storageand Memory Systems

Opportunities to Innovate Beyond the Hamming Metric

L. Dolecek1 A. Jiang2

1Department of Electrical Engineering, UCLA

2Department of Computer Science, Texas A&M

ISIT, 2014

1 / 76

Welcome to the zettabyte age

!"#"$"%"&"'()*$

+$

,+$

-+$

.+$

,++/$,+0+$,+0/$,+,+$

12$-/$123$

2 / 76

Exciting times for data storage

Data storage

Innovations in data storage are the key enabler of the information revolution

Data avalanche

Rising NVM industry

Data center farms

0  

20  

40  

60  

2005  2010  2015  2020  

ZB

3 / 76

Exciting times for data storage

New technologies

Data storage

Innovations in data storage are the key enabler of the information revolution

Data avalanche

Personalized healthcare

Rising NVM industry

Smart cities

Augmented reality

Data center farms

Security

Scientific discoveries

0  

20  

40  

60  

2005  2010  2015  2020  

ZB

3 / 76

What is this tutorial about ?

Today we will learn about

Physical characteristics of emerging non-volatile memories,

Channel models associated with these technologies,

Various recent developments in coding for memories, and

Open problems and future directions in communication techniques formemories.

4 / 76

What is this tutorial about ?

Today we will learn about

Physical characteristics of emerging non-volatile memories,

Channel models associated with these technologies,

Various recent developments in coding for memories, and

Open problems and future directions in communication techniques formemories.

4 / 76

What is this tutorial about ?

Today we will learn about

Physical characteristics of emerging non-volatile memories,

Channel models associated with these technologies,

Various recent developments in coding for memories, and

Open problems and future directions in communication techniques formemories.

4 / 76

What is this tutorial about ?

Today we will learn about

Physical characteristics of emerging non-volatile memories,

Channel models associated with these technologies,

Various recent developments in coding for memories, and

Open problems and future directions in communication techniques formemories.

4 / 76

Outline

5 / 76

6 / 76

Basics of NVM operations

7 / 76

Flash technology

A cell is a floating-gate transistor.

Row of cells is a page.

Group of pages is a block.

A flash memory block is an array of 220 ∼ million cells.

!"#$%"&'()$*'+,-.*'/)0*%'1&")$-#2'()$*'+,-.*'/)0*%'

'''

3456$%)$*'

78'

9'

78'

('

8 / 76

Flash technology

A cell is a floating-gate transistor.

Row of cells is a page.

Group of pages is a block.

A flash memory block is an array of 220 ∼ million cells.

!"#$%"&'()$*'+,-.*'/)0*%'1&")$-#2'()$*'+,-.*'/)0*%'

'''

3456$%)$*'

78'

9'

78'

('

8 / 76

Flash technology

A cell is a floating-gate transistor.

Row of cells is a page.

Group of pages is a block.

A flash memory block is an array of 220 ∼ million cells.

!"#$%"&'()$*'+,-.*'/)0*%'1&")$-#2'()$*'+,-.*'/)0*%'

'''

3456$%)$*'

78'

9'

78'

('

8 / 76

Flash technology

A cell is a floating-gate transistor.

Row of cells is a page.

Group of pages is a block.

A flash memory block is an array of 220 ∼ million cells.

!"#$%"&'()$*'+,-.*'/)0*%'1&")$-#2'()$*'+,-.*'/)0*%'

'''

3456$%)$*'

78'

9'

78'

('

8 / 76

Flash technology

Cell value corresponds to the voltage induced by the number ofelectrons stored on the gate.

Single level cell (SLC)

stores 1 bit per cell; one separating level

Multiple level cell (MLC)

stores 2 or more bits per cell; multiple separating levels

_

_

_

___

__

__

___

__

_

_ _

_

_

_

_

___

__

__

___

__

9 / 76

Flash technology

Cell value corresponds to the voltage induced by the number ofelectrons stored on the gate.

Single level cell (SLC)

stores 1 bit per cell; one separating level

Multiple level cell (MLC)

stores 2 or more bits per cell; multiple separating levels

_

_

_

___

__

__

___

__

_

_

_

_

_

_

_

___

__

__

___

__

9 / 76

Flash technology

Cell value corresponds to the voltage induced by the number ofelectrons stored on the gate.

Single level cell (SLC)

stores 1 bit per cell; one separating level

Multiple level cell (MLC)

stores 2 or more bits per cell; multiple separating levels

_

_

_

___

__

__

___

__

_

_ _

_

_

_

_

___

__

__

___

__

9 / 76

Flash technology

Cell value corresponds to the voltage induced by the number ofelectrons stored on the gate.

Single level cell (SLC)

stores 1 bit per cell; one separating level

Multiple level cell (MLC)

stores 2 or more bits per cell; multiple separating levels

_

_

_

___

__

__

___

__

_

_ _

_

_

_

_

___

__

__

___

__

9 / 76

Flash programming – example 1

Flash block Initial state

1 1 1 11 1 1 11 1 1 11 1 1 1

10 / 76

Flash programming – example 1

Flash block Final state

1 1 2 11 1 1 12 2 1 13 3 2 1

11 / 76

Flash programming – example 2

Flash block Initial state

1 1 2 11 1 1 12 2 1 13 3 2 1

12 / 76

Flash programming – example 2

Flash block Final state

1 1 3 22 1 1 12 2 1 13 3 3 2

13 / 76

Flash programming – example 3

Flash block Initial state

1 1 2 11 1 1 12 2 1 13 3 2 1

14 / 76

Flash programming – example 3

Flash block Intermediate state

1 1 1 11 1 1 11 1 1 11 1 1 1

15 / 76

Flash programming – example 3

Flash block Final state

1 1 2 11 1 1 12 2 1 13 1 2 1

16 / 76

Flash endurance

Flash block Programmed state

1 3 4 34 2 3 41 1 3 24 3 2 4

17 / 76

Flash endurance

Flash block Retrieved state

1 3 4 33 2 3 31 1 2 23 3 2 3

Charge leakage over time results in level drop!

18 / 76

Flash endurance

Frequent block erases cause so-called “wearout” due to charge loss onthe gates.

Flash memory lifetime is commonly expressed in terms of the numberof program and erase (P/E) cycles allowed before memory is deemedunusable.

Lifetime:

SLC: ≈ 106 P/E cyclesMLC: ≈ 103 − 105 P/E cycles

With frequent writes 2GB TLC lasts less than 3 months!

19 / 76

Intercell coupling

Flash block Original state

1 3 4 34 2 3 41 1 3 21 1 1 3

20 / 76

Intercell coupling

Flash block Programmed state

1 3 4 34 2 3 41 1 3 23 1 3 3

21 / 76

Intercell coupling

Flash block Retrieved state

1 3 4 34 2 3 41 1 3 23 3 3 3

Change in charge in one cell affects voltage threshold of a neighboringcell.

During write/read, stressed (victim) cell appears weakly programmed.

22 / 76

To summarize...

Key flash features

Write on the page level, erase on the block level

Wearout limits usable lifetime

Inter-symbol interference

!"#$%"&'()$*'+,-.*'/)0*%'1&")$-#2'()$*'+,-.*'/)0*%'

'''

3456$%)$*'

78'

9'

78'

('

23 / 76

24 / 76

Phase change memories (PCM)

Pros

Erases on the cell level

Faster access

More P/E cycles, longerlifetime

Cons

Not as dense

Thermal accumulation

Thermal cross talk

Higher processing cost

Each cell is in two fundamental states(with in-between states)

Amorphous – high resistance↓ (intermediate)↓ (intermediate)

Polycrystalline – low resistance

25 / 76

Phase change memories (PCM)

Pros

Erases on the cell level

Faster access

More P/E cycles, longerlifetime

Cons

Not as dense

Thermal accumulation

Thermal cross talk

Higher processing cost

Each cell is in two fundamental states(with in-between states)

Amorphous – high resistance↓ (intermediate)↓ (intermediate)

Polycrystalline – low resistance

25 / 76

Phase change memories (PCM)

Pros

Erases on the cell level

Faster access

More P/E cycles, longerlifetime

Cons

Not as dense

Thermal accumulation

Thermal cross talk

Higher processing cost

Each cell is in two fundamental states(with in-between states)

Amorphous – high resistance↓ (intermediate)↓ (intermediate)

Polycrystalline – low resistance

25 / 76

References

J. Cooke, “The inconvenient truths about NAND flash memories”,FS, 2007.

L. M. Grupp et al., “Beyond the datasheet: using test beds to probenon-volatile memories’ dark secrets”, GC, 2010.

L. M. Grupp et al., “Characterizing flash memory: anomalies,observations, and applications”, Micro, 2009.

J. Burr et al., “Phase change memory technology”, JVST, 2010.

X. Huang et al., “Optimization of achievable information rates andnumber of levels in multilevel flash memories”, ICN, 2013.

26 / 76

27 / 76

Part II: Error Correction Coding

28 / 76

Part II: Error Correction Coding

28 / 76

ECC is a must for new NVMs

Early designs:Hamming code, Reed-Solomon code

Widespread industry standard:BCH code

!""#

!""#

!""#

!""#

Micronix, “NAND Flash Technical Note”, 2014.

Spansion, “What Types of ECC Should Be Used on Flash Memory?”,2011.

Micron, “The Inconvenient Truth About NAND Flash”, 2007.

29 / 76

ECC is a must for new NVMs

Early designs:Hamming code, Reed-Solomon code

Widespread industry standard:BCH code

!""#

!""#

!""#

!""#

Micronix, “NAND Flash Technical Note”, 2014.

Spansion, “What Types of ECC Should Be Used on Flash Memory?”,2011.

Micron, “The Inconvenient Truth About NAND Flash”, 2007.

29 / 76

ECC is a must for new NVMs

Early designs:Hamming code, Reed-Solomon code

Widespread industry standard:BCH code

!""#

!""#

!""#

!""#

Micronix, “NAND Flash Technical Note”, 2014.

Spansion, “What Types of ECC Should Be Used on Flash Memory?”,2011.

Micron, “The Inconvenient Truth About NAND Flash”, 2007.

29 / 76

Flash chip data collection: a closer look

0 500 1000 1500 2000 2500 3000 3500 4000 4500 500010−5

10−4

10−3

10−2

P/E Cycles

Erro

r Rat

e

Error Rates for TLC Flash

LSBCSBMSBSymbol Error Rate

Student Version of MATLAB

LSB: least significant bitCSB: center significant bitMSB: most significant bit

!!!"

!!#"

!##"

!#!"

##!"

###"

#!#"

#!!"

Data collected in Swanson Lab, UCSD.

30 / 76

Error patterns in TLC and the need for new codes

Number of wrong bits per wrong symbol Percentage of errors

1 0.96172 0.03143 0.0069

Why not use standard codes ?

!""#

!""#

!""#

!""#

31 / 76

Error patterns in TLC and the need for new codes

Number of wrong bits per wrong symbol Percentage of errors

1 0.96172 0.03143 0.0069

Why not use standard codes ?

!""#

!""#

!""#

!""#

31 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

32 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

32 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

32 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#( !"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

Conventional codes are optimized for the Hamming metric.

1 Do not differentiate among different types of symbol errors

2 Result in unnecessary overhead (rate loss)

33 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#( !"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

Conventional codes are optimized for the Hamming metric.

1 Do not differentiate among different types of symbol errors

2 Result in unnecessary overhead (rate loss)

33 / 76

Symmetric vs. non-symmetric error spheres

!"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#( !"#$%"&#'( $&&"&'()"(*$(!"&&$!)$#(

Conventional codes are optimized for the Hamming metric.

1 Do not differentiate among different types of symbol errors

2 Result in unnecessary overhead (rate loss)

33 / 76

34 / 76

Variability-aware coding

Key idea

Control for both the number of cells to be corrected as well as the numberof bits per erroneous cell to be corrected.

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

35 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:

1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

Variability-aware coding

An example:

Suppose (000 110 010 101 000 111) is stored

Suppose (101 110 000 101 000 011) is retrieved

Error pattern characterization:1 There are 3 symbols in error

→ Code that corrects t symbols

2 There are 3 symbols in error, with at most 2 erroneous bits each

→ [t, `] code that corrects t symbols, each with at most ` bits in error

3 There are 3 symbols in error, with at most 2 erroneous bits each, andthere is 1 symbol with more than 1 bit in error

→ [t1, t2; `1, `2] code that corrects t1 + t2 symbols, each with at most `1bits in error, and at most t2 symbols with more than `2 bits in error

36 / 76

New code constructions

Theorem (Construction 2)

Let H1 be the parity check matrix of a [m, k1, `]2 code C1 (standard[n, k, e] notation).

Let H2 be the parity check matrix of a [n, k2, t]2m−k1 code C2 definedover the alphabet of size GF (2)m−k1 .

Then, HA = H2 ⊗ H1 is the parity check matrix of a [t; `]2m code oflength n.

k1 denotes the number of message bits, m denotes the number ofcoded bits and ` is the number of bit errors code C1 can correct.

J. K. Wolf, “On codes derivable from the tensor product of checkmatrices”, T-IT, 1965.

37 / 76

New code constructions

Theorem (Construction 2)

Let H1 be the parity check matrix of a [m, k1, `]2 code C1 (standard[n, k, e] notation).

Let H2 be the parity check matrix of a [n, k2, t]2m−k1 code C2 definedover the alphabet of size GF (2)m−k1 .

Then, HA = H2 ⊗ H1 is the parity check matrix of a [t; `]2m code oflength n.

k2 denotes the number of message symbols, n denotes the number ofcoded symbols and t is the number of errors code C2 can correct.

J. K. Wolf, “On codes derivable from the tensor product of checkmatrices”, T-IT, 1965.

37 / 76

New code constructions

Theorem (Construction 2)

Let H1 be the parity check matrix of a [m, k1, `]2 code C1 (standard[n, k, e] notation).

Let H2 be the parity check matrix of a [n, k2, t]2m−k1 code C2 definedover the alphabet of size GF (2)m−k1 .

Then, HA = H2 ⊗ H1 is the parity check matrix of a [t; `]2m code oflength n.

J. K. Wolf, “On codes derivable from the tensor product of checkmatrices”, T-IT, 1965.

37 / 76

New code constructions

Choose H =

[H1

H3

]as a parity check matrix of a [m, k1, `2]2 code

and H1 as a parity check matrix of a [m,m − r , `1]2 code

H1 is an r ×m matrix, H3 is an s ×m matrix.

Suppose H2 is a parity check matrix of a [n, k2, t1 + t2]2r code.

Suppose H4 is a parity check matrix of a [n, k3, t2]2s code.

Theorem (Construction 3)

Then, HB is the parity check matrix of a [t1, t2; `1, `2]2m code, where

HB =

(H2 ⊗ H1

H4 ⊗ H3

).

38 / 76

Performance evaluation for codes of length 4000 bits andrate 0.86

2800 3000 3200 3400 3600 3800 4000 4200 440010−7

10−6

10−5

10−4

P/E Cycles

Bit E

rror R

ate

Error Rates of Codes Applied to TLC Flash

Non−Binary BCH Code Over GF(8)Binary BCH CodeDifferent Binary BCH Code Applied to LSB, CSB, MSBGraded−Bit Error−Correcting Code

39 / 76

Performance evaluation for codes of length 4000 bits andrate 0.86

2800 3000 3200 3400 3600 3800 4000 4200 440010−7

10−6

10−5

10−4

P/E Cycles

Bit E

rror R

ate

Error Rates of Codes Applied to TLC Flash

Non−Binary BCH Code Over GF(8)Binary BCH CodeDifferent Binary BCH Code Applied to LSB, CSB, MSBGraded−Bit Error−Correcting Code

40%  Improvement  

39 / 76

Performance evaluation for codes of length 4000 bits andrate 0.86

2800 3000 3200 3400 3600 3800 4000 4200 440010−7

10−6

10−5

10−4

P/E Cycles

Bit E

rror R

ate

Error Rates of Codes Applied to TLC Flash

Non−Binary BCH Code Over GF(8)Binary BCH CodeDifferent Binary BCH Code Applied to LSB, CSB, MSBGraded−Bit Error−Correcting Code

20%  Improvement  

39 / 76

40 / 76

Exploiting level distributions

_

_

_

___

__

__

___

__

_

_ _

_

_

_

_

___

__

__

___

__

What high-performance codes are amenable for soft decoding ?

41 / 76

Exploiting level distributions

_

_

_

___

__

__

___

__

_

_ _

_

_

_

_

___

__

__

___

__

What high-performance codes are amenable for soft decoding ?

41 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

2 reads compare against two thresholds

0 1

p2 p1p3

T-T

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T and −T )

42 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

2 reads compare against two thresholds

0 1

p2 p1p3

T-T

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T and −T )

42 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

2 reads compare against two thresholds

0 1

p2 p1p3

T-T

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T and −T )

42 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

2 reads compare against two thresholds

0 1

p2 p1p3

T-T

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T and −T )

42 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

3 reads compare against three thresholds

0 1

p4 p1p2

T1-T1

p3

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T1, −T1 and 0)

43 / 76

LDPC for Flash: Extracting soft information

Idea: multiple word line reads

3 reads compare against three thresholds

0 1

p4 p1p2

T1-T1

p3

0 0

1 1

e

p1

p1

p2p3

p2

p3

00

10

p1

p1

p2p3

p2

p3

01

11

p4

p4

0

1

(a) Two reads (b) Three reads

Fig. 3. Equivalent discrete memoryless channel models for SLC with (a) two reads and (b) three reads with distinct word-line

voltages.

A. PAM-Gaussian Model

This subsection describes how to select word-line voltages to achieve MMI in the context of

a simple model of the flash cell read channel as PAM transmission with Gaussian noise. MMI

word-line voltage selection is presented using three examples: SLC with two reads, SLC with

three reads, and 4-level MLC with six reads.

For SLC flash memory, each cell can store one bit of information. Fig. 2 shows the model of

the threshold voltage distribution as a mixture of two identically distributed Gaussian random

variables. When either a “0” or “1” is written to the cell, the threshold voltage is modeled as a

Gaussian random variable with variance N0/2 and either mean −√Es (for “1” ) or mean +

√Es

(for “0” ).

For SLC with two reads using two different word line voltages, which should be symmetric

(shown as q and −q in Fig. 2), the threshold voltage is quantized according to three regions

shown in Fig. 2: the white region, the black region, and the union of the light and dark gray

regions (which essentially corresponds to an erasure). This quantization produces the effective

discrete memoryless channel (DMC) model as shown in Fig. 3 (a) with input X ∈ {0, 1} and

output Y ∈ {0, e, 1}.

Assuming X is equally likely to be 0 or 1, the MI I(X; Y ) between the input X and output

Maximize mutual information of the induced channel to determine thebest thresholds (here T1, −T1 and 0)

43 / 76

Performance with multi read

Figure: Performance comparison for 0.9-rate LDPC and BCH codes of lengthn = 9100.

0 0.2 0.4 0.6 0.8 1

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2

Threshold q

Mut

ual I

nfor

mat

ion

SNR 10 dBSNR 12 dBSNR 14 dBSNR 16 dBSNR 18 dB

Fig. 9. MI vs. q for various SNRs for the Gaussian model of MLC(four-level) Flash with the erasure regions in Fig. 8 of size 2q and cen-tered on the natural hard-decoding thresholds for Gaussians with means{µ1, µ2, µ3, µ4} = {!3, !1, 1, 3}.

0 5 10 15 200.8

1

1.2

1.4

1.6

1.8

2

SNR (dB)

Mut

ual I

nfor

mat

ion

constrained to a single qunconstrained

Fig. 10. MI vs. SNR for thresholds for MLC with six reads where optimiza-tion is either constrained by a single parameter q or fully unconstrained.

studied in Fig. 9 cause a significant reduction in MMI ascompared to unconstrained thresholds. Fig. 10 compares theperformance of the constrained optimization, which has asingle parameter q, and the full unconstrained optimization.As shown in the figure, the benefit of fully unconstrainedoptimization is insignificant.Fig. 11 shows performance of unconstrained MMI quan-

tization on the Gaussian channel model of Fig. 8 for threeand six reads for Codes 1 and 2. With four levels, three readsare required for hard decoding. For MLC (four-level) Flash,using six reads recovers more than half of the gap betweenhard decoding (three reads) and full soft-precision decoding.This is similar to the performance improvement seen for SLC(two-level) Flash when increasing from one read to two reads.Note that in Fig. 11, the trade-off between performance

with soft decoding and performance with hard decoding iseven more pronounced. Code 1 is clearly superior with softdecoding but demonstrates a noticeable error floor whendecoded with three or six reads.LDPC error floors due to absorbing sets can be sensitive

0.010.003 0.060.006 0.020.008 0.030.004 0.015

10−4

10−3

10−2

10−1

100

Channel Bit Error Probability

Fram

e Er

ror R

ate

Frame Error Rate vs. Raw Bit Error Rate (MLC Gaussian Model)

Hard BCHHard (3 reads)Code 1Hard (3 reads)Code 26 reads MMICode 16 reads MMICode 2Soft Code 2Soft Code 1Hard (3 reads)Shannon Limit6 readsShannon Limit MMISoft Shannon Limit

Fig. 11. FER vs. channel bit error probability simulation results using theGaussian channel model for 4-level MLC comparing LDPC Codes 1 and 2with varying levels of soft information and a BCH code with hard decoding.All codes have rate 0.9021.

Fig. 12. A (4,2) absorbing set. Variable nodes are shown as black circles.Satisfied check nodes are shown as white squares. Unsatisfied check nodesare shown as black squares. Note that each of the four variable nodes hasdegree three. This absorbing set is avoided by precluding degree-3 nodes.

to the quantization precision, occurring at low precision butnot at high precision [27], [28]. Code 1 has small absorbingsets including the (4, 2), (5, 1), and (5, 2) absorbing sets. Asshown in Fig. 12 for the (4,2) absorbing set, these absorbingsets can all be avoided by precluding degree-three variablenodes. Code 2 avoids these absorbing sets because it has nodegree-3 variable nodes. As shown in Fig. 11, Code 2 avoidsthe error floor problems of Code 1.

B. A More Realistic Model

We can extend the MMI analysis of Section III-B to anymodel for the Flash memory read channel. Consider againthe 4-level 6-read MLC as a 4-input 7-output DMC. Insteadof assuming Gaussian noise distributions as shown in Fig. 8,Fig. 13 shows the four conditional threshold-voltage proba-bility density functions generated according to the six-monthretention model of [16] and the six word-line voltages thatmaximize MI for this noise model. While the conditional noisefor each transmitted (or written) threshold voltage is similar tothat of a Gaussian, the variance of the conditional distributionsvaries greatly across the four possible threshold voltages. Notethat the lowest threshold voltage has by far the largest variance.

Caution:

Optimal code design in the error floor region depends on the chosenquantization.AWGN-optimized LDPC codes may not be the best for the quantized(and asymmetric) Flash channel !

44 / 76

Performance with multi read

Figure: Performance comparison for 0.9-rate LDPC and BCH codes of lengthn = 9100.

0 0.2 0.4 0.6 0.8 1

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2

Threshold q

Mut

ual I

nfor

mat

ion

SNR 10 dBSNR 12 dBSNR 14 dBSNR 16 dBSNR 18 dB

Fig. 9. MI vs. q for various SNRs for the Gaussian model of MLC(four-level) Flash with the erasure regions in Fig. 8 of size 2q and cen-tered on the natural hard-decoding thresholds for Gaussians with means{µ1, µ2, µ3, µ4} = {!3, !1, 1, 3}.

0 5 10 15 200.8

1

1.2

1.4

1.6

1.8

2

SNR (dB)

Mut

ual I

nfor

mat

ion

constrained to a single qunconstrained

Fig. 10. MI vs. SNR for thresholds for MLC with six reads where optimiza-tion is either constrained by a single parameter q or fully unconstrained.

studied in Fig. 9 cause a significant reduction in MMI ascompared to unconstrained thresholds. Fig. 10 compares theperformance of the constrained optimization, which has asingle parameter q, and the full unconstrained optimization.As shown in the figure, the benefit of fully unconstrainedoptimization is insignificant.Fig. 11 shows performance of unconstrained MMI quan-

tization on the Gaussian channel model of Fig. 8 for threeand six reads for Codes 1 and 2. With four levels, three readsare required for hard decoding. For MLC (four-level) Flash,using six reads recovers more than half of the gap betweenhard decoding (three reads) and full soft-precision decoding.This is similar to the performance improvement seen for SLC(two-level) Flash when increasing from one read to two reads.Note that in Fig. 11, the trade-off between performance

with soft decoding and performance with hard decoding iseven more pronounced. Code 1 is clearly superior with softdecoding but demonstrates a noticeable error floor whendecoded with three or six reads.LDPC error floors due to absorbing sets can be sensitive

0.010.003 0.060.006 0.020.008 0.030.004 0.015

10−4

10−3

10−2

10−1

100

Channel Bit Error Probability

Fram

e Er

ror R

ate

Frame Error Rate vs. Raw Bit Error Rate (MLC Gaussian Model)

Hard BCHHard (3 reads)Code 1Hard (3 reads)Code 26 reads MMICode 16 reads MMICode 2Soft Code 2Soft Code 1Hard (3 reads)Shannon Limit6 readsShannon Limit MMISoft Shannon Limit

Fig. 11. FER vs. channel bit error probability simulation results using theGaussian channel model for 4-level MLC comparing LDPC Codes 1 and 2with varying levels of soft information and a BCH code with hard decoding.All codes have rate 0.9021.

Fig. 12. A (4,2) absorbing set. Variable nodes are shown as black circles.Satisfied check nodes are shown as white squares. Unsatisfied check nodesare shown as black squares. Note that each of the four variable nodes hasdegree three. This absorbing set is avoided by precluding degree-3 nodes.

to the quantization precision, occurring at low precision butnot at high precision [27], [28]. Code 1 has small absorbingsets including the (4, 2), (5, 1), and (5, 2) absorbing sets. Asshown in Fig. 12 for the (4,2) absorbing set, these absorbingsets can all be avoided by precluding degree-three variablenodes. Code 2 avoids these absorbing sets because it has nodegree-3 variable nodes. As shown in Fig. 11, Code 2 avoidsthe error floor problems of Code 1.

B. A More Realistic Model

We can extend the MMI analysis of Section III-B to anymodel for the Flash memory read channel. Consider againthe 4-level 6-read MLC as a 4-input 7-output DMC. Insteadof assuming Gaussian noise distributions as shown in Fig. 8,Fig. 13 shows the four conditional threshold-voltage proba-bility density functions generated according to the six-monthretention model of [16] and the six word-line voltages thatmaximize MI for this noise model. While the conditional noisefor each transmitted (or written) threshold voltage is similar tothat of a Gaussian, the variance of the conditional distributionsvaries greatly across the four possible threshold voltages. Notethat the lowest threshold voltage has by far the largest variance.

Caution:

Optimal code design in the error floor region depends on the chosenquantization.AWGN-optimized LDPC codes may not be the best for the quantized(and asymmetric) Flash channel !

44 / 76

45 / 76

Motivation: Write latency

Recall Flash programming

In practice incremental steppulse programming is used,a.k.a. guess and verify.

Latency increases with numberof levels.

If one allows for “dirty writes”, it suffices to correct errors in only onedirection.

46 / 76

Motivation: Write latency

Recall Flash programming

In practice incremental steppulse programming is used,a.k.a. guess and verify.

Latency increases with numberof levels.

If one allows for “dirty writes”, it suffices to correct errors in only onedirection.

46 / 76

Motivation: Write latency

Recall Flash programming

In practice incremental steppulse programming is used,a.k.a. guess and verify.

Latency increases with numberof levels.

If one allows for “dirty writes”, it suffices to correct errors in only onedirection.

46 / 76

Coding for asymmetric, limited-magnitude errors

If one allows for “dirty writes”, it suffices to correct errors in only onedirection and that are of limited magnitude.

Definition (ALM Code)

Let C1 be a code over the alphabet Q1. The code C over the alphabet Q(with |Q| > |Q1| = q1 = `+ 1) is defined asC = {x = (x1, x2, . . . , xn) ∈ Qn | x mod q1 ∈ C1}.

Theorem

Code C corrects t asymmetric errors of limited magnitude ` if code C1corrects t symmetric errors.

New construction inherits encoding/decoding complexity of theunderlying (symmetric) ECC.

Connections with additive number theory and asymmetric ECC(Varshamov 1970s).

47 / 76

Coding for asymmetric, limited-magnitude errors

If one allows for “dirty writes”, it suffices to correct errors in only onedirection and that are of limited magnitude.

Definition (ALM Code)

Let C1 be a code over the alphabet Q1. The code C over the alphabet Q(with |Q| > |Q1| = q1 = `+ 1) is defined asC = {x = (x1, x2, . . . , xn) ∈ Qn | x mod q1 ∈ C1}.

Theorem

Code C corrects t asymmetric errors of limited magnitude ` if code C1corrects t symmetric errors.

New construction inherits encoding/decoding complexity of theunderlying (symmetric) ECC.

Connections with additive number theory and asymmetric ECC(Varshamov 1970s).

47 / 76

Further Reading (1/2)

Y. Cassuto, M. Schwartz, V. Bohossian, and J. Bruck, “Codes forasymmetric limited magnitude errors with application to multilevelflash memories”, T-IT, 2010.

S. Yari, T. Kløve, and B. Bose, “Some codes correcting unbalancederrors of limited magnitude for flash memories”, T-IT, 2013.

S. Bizaglo and T. Etzion, “Tilings with n-dimensional chairs and theirapplications to asymmetric codes”, T-IT,2013.

H. Zhou, A. Jiang, and J. Bruck, “Nonuniform codes for correctingasymmetric errors in data storage”, T-IT, 2013.

M. Blaum, “Codes for detecting and correcting unidirectional errors”,1993.

48 / 76

Further Reading (2/2)

R. Gabrys, E. Yaakobi, and L. Dolecek, “Graded bit error correctingcodes with applications to flash memory”, T-IT, 2013.

P. Vontobel and R. Roth, “Coding for combined block-symbol errorcorrection”, T-IT, 2014.

J. Wang et al., “Enhanced precision through multiple reads for LDPCdecoding in flash memories”, JSAC, 2014.

J. Wang, L. Dolecek, and R. D. Wesel, “The cycle consistency matrixapproach to LDPC absorbing sets in separable circulant-based codes”,T-IT, 2013.

K. Haymaker and C. Kelley, “Structured bit-interleaved LDPC codesfor MLC flash memory”, JSAC, 2014.

A. Jiang, H. Li, and J. Bruck, “On the capacity and programming offlash memories”, T-IT, 2012.

49 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

Research problems on coding for spatio-temporal variabilityin NVMs

1 Investigation of algebraic codes for both transient andpermanent errors

2 Investigation of graph-based binary and non-binary codesand their decoders for flash (LDPC, spatially coupledcodes etc.)

3 Constructions with tighter bounds, i.e., beyondasymptotic optimality

4 Noise-adaptive coding

5 Coding for the target average performance

6 Deeper connections with number theoretic andcombinatorial methods

7 Performance – complexity tradeoff and evaluations formore realistic channels

50 / 76

51 / 76

Motivation: Coding can defer costly erases

Recall write and block-erase operations

What if we can maximize the amount of data written before the blockerase is necessary?

52 / 76

WOM Coding

53 / 76

WOM Coding

53 / 76

Write-once memory (WOM)

Cells are irreversibly programmed from “0” to “1”

Original applications of WOM were punch cards and optical disks

WOM coding allows for increased storage utilization

54 / 76

The mother of all WOM codes

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

55 / 76

Encoding example 1

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 10 → Encode: 101.

56 / 76

Encoding example 1

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 10 → Encode: 101.

56 / 76

Encoding example 1

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 10 → Encode: 101.

56 / 76

Encoding example 1

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 10 → Encode: 101.

57 / 76

Encoding example 2

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 01 → Encode: 001. No change.

58 / 76

Encoding example 2

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 01 → Encode: 001. No change.

58 / 76

Encoding example 2

Rivest-Shamir WOM code:

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Write-1: 01 → Encode: 001.

Write-2: 01 → Encode: 001. No change.

58 / 76

WOM model

!"#$%&'#"%(& )(*+,-&'#"%(& ./"#-&'#"%(&

Definition (WOM constraint)

The memory state is modeled as a vector yj of length n where j is thecurrent write (or generation). Each element y j

i , 1 ≤ i ≤ n, takes values inthe set {0, 1, . . . q − 1}. On write j , the encoder writes one of Mj

messages to the memory by updating yj−1 to yj while satisfying theWOM-constraint yj ≥ yj−1.

59 / 76

WOM model

Definition (Sum rate)

If Mj codewords can be represented at generationj , then generation j has rate 1

n log(Mj). The sumrate is the sum of rates across generations.

Rivest-Shamir code has rate log(M1 + M2)/n = log(4 + 4)/3 = 1.33.

Capacity is the maximum achievable sum rate.

60 / 76

WOM model

Definition (Sum rate)

If Mj codewords can be represented at generationj , then generation j has rate 1

n log(Mj). The sumrate is the sum of rates across generations.

Rivest-Shamir code has rate log(M1 + M2)/n = log(4 + 4)/3 = 1.33.

Capacity is the maximum achievable sum rate.

60 / 76

WOM model

Definition (Sum rate)

If Mj codewords can be represented at generationj , then generation j has rate 1

n log(Mj). The sumrate is the sum of rates across generations.

Rivest-Shamir code has rate log(M1 + M2)/n = log(4 + 4)/3 = 1.33.

Capacity is the maximum achievable sum rate.

60 / 76

Capacity as a function of field order size

0

1

2

3

4

5

6

7

8

0 5 10 15 20

Capaci

ty(n

orm

alize

d)

Number of generations t

Unrestricted sum-rate for q = 2, 4, 8, 10

q = 2Uncoded

61 / 76

Capacity as a function of field order size

0

1

2

3

4

5

6

7

8

0 5 10 15 20

Capaci

ty(n

orm

alize

d)

Number of generations t

Unrestricted sum-rate for q = 2, 4, 8, 10

q = 10q = 8q = 4q = 2

Uncoded

61 / 76

Equal rate vs. unequal rate per write

0

1

2

3

4

5

0 5 10 15 20

Capaci

ty(b

its)

Number of generations t

Capacity of binary WOM codes

Unrestricted-rateFixed-rate

62 / 76

Equal rate vs. unequal rate per write

0

1

2

3

4

5

0 5 10 15 20

Capaci

ty(b

its)

Number of generations t

Capacity of binary WOM codes

3.46

3.35

Unrestricted-rateFixed-rate

62 / 76

A construction of binary WOM codes

We want to construct a binary three-write WOM over 2n cells.

Let C3 be a ternary two-write WOM of length n.

First write

Pick a first-generation codeword u in C3.Map length-n u into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.

Second write

Pick a second-generation codeword v in C3.Map length-n v into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.Program each pair only once.

Third write

Write length-n binary string in the unused cells (one per pair).

63 / 76

A construction of binary WOM codes

We want to construct a binary three-write WOM over 2n cells.

Let C3 be a ternary two-write WOM of length n.

First write

Pick a first-generation codeword u in C3.Map length-n u into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.

Second write

Pick a second-generation codeword v in C3.Map length-n v into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.Program each pair only once.

Third write

Write length-n binary string in the unused cells (one per pair).

63 / 76

A construction of binary WOM codes

We want to construct a binary three-write WOM over 2n cells.

Let C3 be a ternary two-write WOM of length n.

First write

Pick a first-generation codeword u in C3.Map length-n u into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.

Second write

Pick a second-generation codeword v in C3.Map length-n v into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.Program each pair only once.

Third write

Write length-n binary string in the unused cells (one per pair).

63 / 76

A construction of binary WOM codes

We want to construct a binary three-write WOM over 2n cells.

Let C3 be a ternary two-write WOM of length n.

First write

Pick a first-generation codeword u in C3.Map length-n u into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.

Second write

Pick a second-generation codeword v in C3.Map length-n v into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.Program each pair only once.

Third write

Write length-n binary string in the unused cells (one per pair).

63 / 76

A construction of binary WOM codes

We want to construct a binary three-write WOM over 2n cells.

Let C3 be a ternary two-write WOM of length n.

First write

Pick a first-generation codeword u in C3.Map length-n u into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.

Second write

Pick a second-generation codeword v in C3.Map length-n v into length-2n binary string using Φ(0) = 00,Φ(1) = 10, Φ(2) = 01.Program each pair only once.

Third write

Write length-n binary string in the unused cells (one per pair).

63 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write:

first and second partition only

Pick a message m in Z n3 .

Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write:

second and third partition only

Pick a message s in Z n3 .

Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write:

first and second partition only

Pick a message m in Z n3 .

Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write:

second and third partition only

Pick a message s in Z n3 .

Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write:

first and second partition only

Pick a message m in Z n3 .

Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write:

second and third partition only

Pick a message s in Z n3 .

Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write:

first and second partition only

Pick a message m in Z n3 .

Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write:

second and third partition only

Pick a message s in Z n3 .

Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write: first and second partition onlyPick a message m in Z n

3 .Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write:

second and third partition only

Pick a message s in Z n3 .

Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

A construction of non-binary WOM codes

We want to construct a non-binary two-write WOM with q levels,with 3|(q + 1).

Partition levels in groups of L = (q + 1)/3

first partition is {0, 1, . . . , L− 1}second partition is {L− 1, L, . . . , 2L− 1}third partition is {2L− 1, 2L, . . . , 3L− 1}

Let C2 be a binary two-write WOM code of length n.

First write: first and second partition onlyPick a message m in Z n

3 .Pick a first-generation codeword u in C2.For each i , 1 ≤ i ≤ n: If ui = 0 write mi . If ui = 1 write mi + L.

Second write: second and third partition onlyPick a message s in Z n

3 .Pick a second-generation codeword v in C2.For each i , 1 ≤ i ≤ n: If vi = 0 write si + (L− 1). If vi = 1 writesi + (2L− 1).

64 / 76

An example using Rivest and Shamir write-twice code forq = 2 and n = 8

Rivest-Shamir WOM code

Information First Generation Second Generation

00 000 11101 001 11010 010 10111 100 011

Non-binary WOM code

Write no. Information RS code + info Encoded values

1 (0,1),(0,1,2) (001),(012) (0,1,5)

2 (0,0),(2,1,2) (111),(212) (7,6,7)

65 / 76

Sum rate of short WOM codes

1

1.5

2

2.5

3

3.5

2 3 4 5

Capaci

ty(n

orm

alize

d)

Number of generations t

Unrestricted sum-rate for q = 2, 4, 8, 10

q = 10q = 8q = 4q = 2

Uncoded

66 / 76

Sum rate of short WOM codes

66 / 76

Sum rate of short WOM codes

66 / 76

Sum rate of short WOM codes

66 / 76

Sum rate of short WOM codes

66 / 76

Sum rate of short WOM codes

66 / 76

Facets of WOM coding

Capacity approaching constructionsA. Shpilka, “Capacity achieving multiwrite WOM codes”, T-IT, 2014.

D. Burshtein and A. Strugatski, “Polar write once memory codes”,T-IT 2013.

WOM with error correction/detection capabilities

Q. Huang, S. Lin, and K. A. S. Abdel Ghaffar, “Error correcting codesfor flash coding”, T-IT, 2011.

E. Yaakobi, P. H. Siegel, A. Vardy, and J. K. Wolf, “Multiple errorcorrecting WOM codes”, T-IT, 2012.

A. Jiang, Y. Li, E. En Gad, M. Langberg and J. Bruck, “Joint rewritingand error correction in write-once memories”, ISIT, 2013.

ExtensionsN. Bitouze, A. Graell i Amat, and E. Rosnes, “Using short synchronousWOM codes to make WOM codes decodable”, TCOM, 2014.L. Wang, M. Qin, E. Yaakobi, Y. H. Kim, and P. H. Siegel, “WOMwith retained messages”, ISIT, 2012.

Y. Cassuto and E. Yaakobi, “Short q-ary WOM codes with hot/coldwrite differentiation”, ISIT, 2012.

67 / 76

68 / 76

Motivation: Noisy writes in PCM

The write process in PCM is inherently inexact

The data can be written iteratively, by checking the content betweensuccessive writes

Correctly written cells need not be further updated (cells in PCM canbe accessed individually)

The write process terminates when all cells are sufficiently wellprogrammed

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

69 / 76

Motivation: Noisy writes in PCM

The write process in PCM is inherently inexact

The data can be written iteratively, by checking the content betweensuccessive writes

Correctly written cells need not be further updated (cells in PCM canbe accessed individually)

The write process terminates when all cells are sufficiently wellprogrammed

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

69 / 76

Motivation: Noisy writes in PCM

The write process in PCM is inherently inexact

The data can be written iteratively, by checking the content betweensuccessive writes

Correctly written cells need not be further updated (cells in PCM canbe accessed individually)

The write process terminates when all cells are sufficiently wellprogrammed

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

69 / 76

Motivation: Noisy writes in PCM

The write process in PCM is inherently inexact

The data can be written iteratively, by checking the content betweensuccessive writes

Correctly written cells need not be further updated (cells in PCM canbe accessed individually)

The write process terminates when all cells are sufficiently wellprogrammed

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

69 / 76

Motivation: Noisy writes in PCM

The write process in PCM is inherently inexact

The data can be written iteratively, by checking the content betweensuccessive writes

Correctly written cells need not be further updated (cells in PCM canbe accessed individually)

The write process terminates when all cells are sufficiently wellprogrammed

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

69 / 76

Rewritable channel for PCM

!"#$%&'(#)*&+,-..*/&

!"#$*/*$$&(*-0&+,-..*/&

&&

1(#)*&+".)("//*(&2.34)&5*$$-6*&

7)"(*0&5*$$-6*&

Average rewriting cost is κ

Theorem

Capacity is upper-bounded by log(Γκ)

Parameter Γ depends on noise characteristics.

Exact expressions known in certain cases

Several extensions available

70 / 76

Further Reading – Early Results

R. L. Rivest and A. Shamir, “How to reuse a “write-once” memory”,I&C, 1982.

J. K. Wolf, A. D. Wyner, J. Ziv, and J. Korner, “Coding forwrite-once memory”, AT&T, 1984.

C. Heegard, “On the capacity of permanent memory”, T-IT, 1985.

G. D. Cohen, P. Godlewski, and F. Merkx, “Linear binary code forwrite-once memories”, T-IT, 1986.

F. Fu and A. J. Han Vinck, “On the capacity of generalizedwrite-once memory with state transitions described by an arbitrarydirected acyclic graph”, T-IT, 1999.

71 / 76

Further Reading – Recent Results

E. Yaakobi, S. Kayser, P. Siegel, A. Vardy, and J. K. Wolf, “Codes forwrite-once memories”, T-IT, 2012.

A. Bhatia, M. Qin, A. Iyengar, B. Kurkoski, and P. Siegel,“Lattice-based WOM codes for multilevel flash memories”, JSAC,2014.

R. Gabrys and L. Dolecek, “Constructions of nonbinary WOM codesfor multilevel flash memories”, preprint, 2014.

R. Gabrys, E. Yaakobi, L. Dolecek, P. Siegel, A. Vardy, and J. K. Wolf,“Non-binary WOM codes for multilevel flash memories”, ITW, 2011.

L. A. Lastras-Montano, M. Franceschini, T. Mittelholzer, and M.Sharma, “Rewritable storage channels”, ISITA, 2008.

R. Venkataramanan, S. Tatikonda, L. Lastras-Montano, M.Franceschini, “Rewritable storage channels with hidden state”, JSAC,2014.

72 / 76

Research problems on WOM and rewriting

1 Coding for rewritable channels with read feedback

2 Extensions to other related models (WAM, floating codesetc).

3 Connections with interference channels (e.g., dirty papercoding)

4 Establishment of the complete capacity region/beyondzero-error capacity.

5 Development of new tools for high-rate codes spanningmore than a couple of cells (e.g., lattices, posets)

73 / 76

74 / 76

Further topics – beyond coding

Statistical and signal processing methods for voltage thresholdmodeling and characterization

Parameter estimation and hypothesis testingStudy of time-varying stochastic processes

Communication methods for Flash channels

Design of optimal detectors under ICIUse of equalization and signal shaping

Capacity calculations

Analysis of quantized channelsNon asymptotic performance analysis

Probabilistic methods and complexity

Implementation complexity evaluation and prototypingAverage vs. worst case performance analysis and design

Applications of interference alignment and interference channels

75 / 76

Further topics – beyond coding

Statistical and signal processing methods for voltage thresholdmodeling and characterization

Parameter estimation and hypothesis testingStudy of time-varying stochastic processes

Communication methods for Flash channels

Design of optimal detectors under ICIUse of equalization and signal shaping

Capacity calculations

Analysis of quantized channelsNon asymptotic performance analysis

Probabilistic methods and complexity

Implementation complexity evaluation and prototypingAverage vs. worst case performance analysis and design

Applications of interference alignment and interference channels

75 / 76

Further topics – beyond coding

Statistical and signal processing methods for voltage thresholdmodeling and characterization

Parameter estimation and hypothesis testingStudy of time-varying stochastic processes

Communication methods for Flash channels

Design of optimal detectors under ICIUse of equalization and signal shaping

Capacity calculations

Analysis of quantized channelsNon asymptotic performance analysis

Probabilistic methods and complexity

Implementation complexity evaluation and prototypingAverage vs. worst case performance analysis and design

Applications of interference alignment and interference channels

75 / 76

Further topics – beyond coding

Statistical and signal processing methods for voltage thresholdmodeling and characterization

Parameter estimation and hypothesis testingStudy of time-varying stochastic processes

Communication methods for Flash channels

Design of optimal detectors under ICIUse of equalization and signal shaping

Capacity calculations

Analysis of quantized channelsNon asymptotic performance analysis

Probabilistic methods and complexity

Implementation complexity evaluation and prototypingAverage vs. worst case performance analysis and design

Applications of interference alignment and interference channels

75 / 76

Further topics – beyond coding

Statistical and signal processing methods for voltage thresholdmodeling and characterization

Parameter estimation and hypothesis testingStudy of time-varying stochastic processes

Communication methods for Flash channels

Design of optimal detectors under ICIUse of equalization and signal shaping

Capacity calculations

Analysis of quantized channelsNon asymptotic performance analysis

Probabilistic methods and complexity

Implementation complexity evaluation and prototypingAverage vs. worst case performance analysis and design

Applications of interference alignment and interference channels

75 / 76

TIME FOR A BREAK!

76 / 76