Backing off from infinity:

Andrea Goldsmith

fundamental communication limits in non-asymptotic regimes

Thanks to collaborators Chen, Eldar, Grover, Mirghaderi, Weissman

Information Theory and AsymptopiaCapacity with asymptotically small error achieved by

asymptotically long codes.Defining capacity in terms of asymptotically small error and

infinite delay is brilliant!Has also been limitingCause of unconsummated union between networks and information

theory

Optimal compression based on properties of asymptotically long sequencesLeads to optimality of separation

Other forms of asymptopiaInfinite SNR, energy, sampling, precision, feedback, …

Why back off?

Theory not informing practice

Theory vs. practice

Theory PracticeInfinite blocklength codesInfinite SNRInfinite energyInfinite feedbackInfinite sampling rates

Infinite (free) processingInfinite precision ADCs

Uncoded to LDPC-7dB in LTEFinite battery life1 bit ARQ50-500 Msps 200 MFLOPs-1B FLOPs8-16 bits

What else lives in asymptopia?

Backing off from: infinite blocklengthRecent developments on finite blocklength

Channel codes (Capacity C for n)Source codes (entropy H or rate distortion R(D))

[Ingber, Kochman’11; Kostina, Verdu’11]

Separation not Optimal Separation not Optimal[Wang et. Al’11; Kostina, Verdu’12]

Grand Challenges Workshop: CTW MauiFrom the perspective of the cellular industry, the Shannon bounds

evaluated by Slepian are within .5 dB for a packet size of 30 bits or more for the real AWGN channel at 0.5 bits/sym, for BLER = 1e-4. In this perhaps narrow context there is not much uncertainty for performance evaluations.

For cellular and general wireless channels, finite blocklength bounds for practical fading models are needed and there is very little work along those lines.

Even for the AWGN channel the computational effort of evaluating the Shannon bounds is formidable.

This indicates a need for accurate approximations, such as those recently developed based on the idea of channel dispersion.

Diversity vs. Multiplexing TradeoffUse antennas for multiplexing or diversity

Diversity/Multiplexing tradeoffs (Zheng/Tse)Error Prone Low Pe

r)r)(N(N(r)d rt*

rSNRlog

R(SNR)lim SNR

dSNRlog

P log e

)(lim SNRSNR

Whatis

Infinite?

Backing off from: infinite SNRHigh SNR Myth: Use some spatial dimensions for

multiplexing and others for diversity

*Transmit Diversity vs. Spatial Multiplexing in Modern MIMO Systems”, Lozano/Jindal

Reality: Use all spatial dimensions for one or the other*Diversity is wasteful of spatial dimensions with HARQAdapt modulation/coding to channel SNR

Diversity-Multiplexing-ARQ TradeoffSuppose we allow ARQ with incremental

redundancy

ARQ is a form of diversity [Caire/El Gamal 2005]

0

2

4

6

8

10

12

14

16

0 1 2 3 4

ARQ WindowSize L=1

L=2 L=3

L=4

d

r

Joint Source/Channel CodingUse antennas for multiplexing:

Use antennas for diversity High-RateQuantizer

ST CodeHigh Rate Decoder

Error Prone

Low Pe

Low-RateQuantizer

ST CodeHigh

DiversityDecoder

How should antennas be used: Depends on end-to-end metric

Joint Source-Channel coding w/MIMOkRu

Index Assignme

nt

s bitsp(i)

Channel Encoder

s bitsi

MIMO Channel

Channel Decoder

Inverse Index Assignment p(j)

s bitsj

s bits

Increased rate heredecreases source distortion

But permits less diversity

here

Resulting in more errors

SourceEncoder

SourceDecoder

And maybe higher total distortion

A joint design is needed

vj

Antenna Assignment vs. SNR

Relaying in wireless networks

Intermediate nodes (relays) in a route help to forward the packet to its final destination.

Decode-and-forward (store-and-forward) most common:Packet decoded, then re-encoded for transmissionRemoves noise at the expense of complexity

Amplify-and-forward: relay just amplifies received packetAlso amplifies noise: works poorly for long routes; low SNR.

Compress-and-forward: relay compresses received packetUsed when Source-relay link good, relay-destination link weak

SourceRelay Destination

Capacity of the relay channel unknown: only have bounds

Cooperation in Wireless Networks

Relaying is a simple form of cooperationMany more complex ways to cooperate:

Virtual MIMO , generalized relaying, interference forwarding, and one-shot/iterative conferencing

Many theoretical and practice issues: Overhead, forming groups, dynamics, full-duplex, synch,

…

Generalized Relaying and Interference Forwarding

Can forward message and/or interference Relay can forward all or part of the messages

Much room for innovation Relay can forward interference

To help subtract it out

TX1

TX2

relay

RX2

RX1X1

X2

Y3=X1+X2+Z3

Y4=X1+X2+X3+Z4

Y5=X1+X2+X3+Z5

X3= f(Y3) Analog network coding

Beneficial to forward bothinterference and message

In fact, it can achieve capacity

S DPs

P1

P2

P3

P4

• For large powers Ps, P1, P2, …, analog network coding (AF) approaches capacity

: Asymptopia?

Maric/Goldsmith’12

Interference AlignmentAddresses the number of interference-free signaling

dimensions in an interference channel

Based on our orthogonal analysis earlier, it would appear that resources need to be divided evenly, so only 2BT/N dimensions available

Jafar and Cadambe showed that by aligning interference, 2BT/2 dimensions are available

Everyone gets half the cake!Except at finite SNRs

Backing off from: infinite SNRHigh SNR Myth: Decode-and-forward equivalent to amplify-

forward, which is optimal at high SNR*Noise amplification drawback of AF diminishes at high SNRAmplify-forward achieves full degrees of freedom in MIMO systems

(Borade/Zheng/Gallager’07)At high-SNR, Amplify-forward is within a constant gap from the capacity upper bound as

the received powers increase (Maric/Goldsmith’07)

Reality: optimal relaying unknown at most SNRs:Amplify-forward highly suboptimal outside high SNR per-node regime, which is not

always the high power or high channel gain regimeAmplify-forward has unbounded gap from capacity in the high channel gain regime

(Avestimehr/Diggavi/Tse’11)

Relay strategy should depend on the worst link

Decode-forward used in practice

Capacity and FeedbackCapacity under feedback largely unknown

Channels with memoryFinite rate and/or noisy feedbackMultiuser channels Multihop networks

ARQ is ubiquitious in practiceWorks well on finite-rate noisy feedback channelsReduces end-to-end delay

Why hasn’t theory met practice when it comes to feedback?

PtP Memoryless Channels: Perfect Feedback

• Shannon• Feedback does not increase capacity of DMCs

• Schalkwijk-Kailath Scheme for AWGN channels–Low-complexity linear recursive scheme –Achieves capacity–Double exponential decay in error probability

Encoder DecoderWW WW

+

Backing off from: Perfect Feedback

+Channel Encoder Decoder

Feedback Module

• [Shannon 59]: No Feedback

• [Pinsker, Gallager et al.]: Perfect feedback • Infinite rate/no noise

• [Kim et. al. 07/10]: Feedback with AWGN

• [Polyaskiy et. al. 10]: Noiseless feedback reduces the minimum energy per bit when nR is fixed and n

• Objective:

Choose and to maximize the decay rate of error probability

Gaussian Channel with Rate-Limited Feedback

+Channel Encoder Decoder

Feedback Module

• Constraints

Feedback is rate-limited ; no noise

A super-exponential error probability is achievable if and only if

• : The error exponent is finite but higher than no-feedback error exponent

• : Double exponential error probability

• : L-fold exponential error probability

m-bit Encoder

m-bit Decoder

m-bit Encoder

m-bit Decoder

Forward Channel

Feedback

Channel

If , sendTermination Alarm

Otherwise, resend with energy

Send back with energy

If Termination Alarm is received, report as the decoded message

Feedback under Energy/Delay Constraint

• Constraints Objective: Choose to

minimize the overall probability of error

Depends on the error probability model ε()

• Exponential Error Model: ε(x)=βe-αx

Applicable when Tx energy dominatesFeedback gain is high if total energy is large

enoughNo feedback gain for energy budgets below

a threshold

Feedback Gain under Energy/Delay Constraint

• Super-Exponential Error Model: ε(x)=βe-αx2

- Applicable when Tx and coding energy are comparable- No feedback gain for energy budgets above a threshold

Backing off from: perfect feedback • Memoryless point-to-point channels:

• Capacity unchanged with perfect feedback• Simple linear scheme reduces error exponent (Schalkwijk-Kailath: double

exponential)• Feedback reduces energy consumption

• Capacity of feedback channels largely unknown• Unknown for general channels with memory and perfect feedback• Unknown under finite rate and/or noisy feedback• Unknown in general for multiuser channels • Unknown in general for multihop networks

• ARQ is ubiquitious in practice• Assumes channel errors• Works well on finite-rate noisy feedback channels• Reduces end-to-end delay

No feedback

Feedback

Output feedbackChannel information (CSI)AcknowledgementsSomething else?

Noisy/Compressed

How to use feedback in wireless networks?

Interesting applications to neuroscience

For a given sampling mechanism (i.e. a “new” channel)What is the optimal input signal?What is the tradeoff between capacity and sampling rate?What known sampling methods lead to highest capacity?

What is the optimal sampling mechanism? Among all possible (known and unknown) sampling schemes

h(t)

SamplingMechanism

(rate fs)

New Channel

Backing off from: infinite sampling

Capacity under Sampling w/Prefilter

Theorem: Channel capacity

h(t)

)(th

)(t

)(tx )(ts

snTt

][ny

“Folded” SNR filtered by S(f) Determined by

waterfilling:suppresses aliasing

Capacity not monotonic in fsConsider a “sparse” channel

Capacity not monotonic in fs!

Single-branchsampling fails to exploit channel structure

Filter Bank Sampling

Theorem: Capacity of the sampled channel using a bank of m filters with aggregate rate fs

h(t)

)(th

)(t

)(tx

)(1 ts

)(tsi

)(tsm

)( smTnt

)( smTnt

)( smTnt

][1 ny

][nyi

][nym

Similar to MIMO; no combining!

Equivalent MIMO Channel Model

h(t)

)(th

)(t

)(tx

)(1 ts

)(tsi

)(tsm

)( smTnt

)( smTnt

)( smTnt

][1 ny

][nyi

][nym

( fX

( skffX

( skffX

)( skffH

)( fH

)( skffH

)( skffN

)( fN

)( skffN

( fYi

( fY1

( fYm

( fS1

( sm kffS

( fSi

( si kffS

( skffS 1

( fSm

( sm kffS

( si kffS

( skffS 1

Theorem 3: The channel capacity of the sampled channel using a bank of m filters with aggregate rate is

For each f

Water-filling over singularvalues

MIMO – DecouplingPre-whitening

Selects the m branches with m highest SNRExample (Bank of 2 branches)

highest SNR

2nd highest SNR

low SNR

( skffX 2

( fX

( skffX

( skffX

)( skffH

)( fH

)( skffH

)( skffN

)( fN

)( skffN

)( skffS

)( fS

)( skffS

)2( skffH

)2( skffN )2( skffS

Joint Optimization of Input and Filter Bank

low SNR

( fY1

( fY2

Capacity monotonic in fs

Can we do better?

Sampling with Modulator+Filter (1 or more)

h(t)

)(th

)(t

)(ts ][nyTheorem:

Bank of Modulator+FilterSingle Branch Filter Bank

Theorem Optimal among all time-preserving nonuniform

sampling techniques of rate fs

zzzzzzzzzz

)(ts ][nyzzzzzzzzzz

)(1 ts

)(tsi

)(tsm

)( smTnt

)( smTnt

)( smTnt

][1 ny

][nyi

][nym

equals

Backing off from: Infinite processing power

Is Shannon-capacity still a good metric for system design?

Our approach

Power consumption via a network graphpower consumed in nodes and wires

Extends early work of El Gamal et. al.’84 and Thompson’80

Fundamental area-time-performance tradeoffs

For encoding/decoding “good” codes,

Stay away from capacity!Close to capacity we have

Large chip-area More time More power

Area occupied by wires Encoding/decoding clock cycles

Total power diverges to infinity!

Regular LDPCs closer to bound than capacity-approaching LDPCs!Need novel code designs with short wires, good performance

ConclusionsInformation theory asympotia has provided much insight and

decades of sublime delight to researchers

Backing off from infinity required for some problems to gain insight and fundamental bounds

New mathematical tools and new ways of applying conventional tools needed for these problems

Many interesting applications in finance, biology, neuroscience, …