Physical Limits of ComputingPhysical Limits of ComputingA Brief IntroductionA Brief Introduction

Dr. Michael P. FrankDr. Michael P. Frankmpf@cise.ufl.edumpf@cise.ufl.edu

Dept. of Computer & Information Science & EngineeringDept. of Computer & Information Science & Engineering(Affil. Dept. of Electrical & Computer Engineering)(Affil. Dept. of Electrical & Computer Engineering)

University of Florida, Gainesville, FloridaUniversity of Florida, Gainesville, Florida

Presented at:2004 Computing Beyond Silicon Summer School (Week 4)

California Institute of TechnologyPasadena, California, July 6-8, 2004

AbstractAbstract• Physical and computational systems share a number of Physical and computational systems share a number of

common characteristics.common characteristics. They are both special cases of the more general concept of They are both special cases of the more general concept of

dynamical systemsdynamical systems..• In fact, we can even show that there is a fundamental In fact, we can even show that there is a fundamental

underlying unity between the physical and underlying unity between the physical and computational domains.computational domains. E.g.E.g., in this talk, we will survey some ways to understand a , in this talk, we will survey some ways to understand a

variety of key physical concepts in computational terms.variety of key physical concepts in computational terms.• Due to this underlying unity, physical systems have Due to this underlying unity, physical systems have

firm limits on their computational capabilities.firm limits on their computational capabilities. Since a computation embedded within a physical system Since a computation embedded within a physical system

clearly cannot exceed the raw computational capabilities of clearly cannot exceed the raw computational capabilities of the physical system itself.the physical system itself.

• We review some of the known limits.We review some of the known limits. On information capacity, processing rate, and On information capacity, processing rate, and

communication bandwidth.communication bandwidth.

Physics as ComputationPhysics as Computation• Preview:Preview: Most/all physical quantities can be validly Most/all physical quantities can be validly

reinterpreted in terms of information and computation.reinterpreted in terms of information and computation. Physical Physical entropyentropy isis

• Incompressible informationIncompressible information.. Physical Physical actionaction isis

• Total amount of (quantum-physical) computation.Total amount of (quantum-physical) computation. Physical Physical energyenergy isis

• The rate of physical computation.The rate of physical computation. The various different forms of energy correspond to physical The various different forms of energy correspond to physical

computation that is occupied doing different kinds of things.computation that is occupied doing different kinds of things. Physical Physical temperaturetemperature is (proportional to) is (proportional to)

• Physical rate of computing per bit of information capacity.Physical rate of computing per bit of information capacity. The “clock speed” for physical computation.The “clock speed” for physical computation.

Physical Physical momentummomentum isis• Amount of “motional” computation per unit distance translated…Amount of “motional” computation per unit distance translated…

there are others for angular momentum, velocity, there are others for angular momentum, velocity, etc.etc. … …• These identities can be made rigorous! These identities can be made rigorous!

We will sketch the arguments later if there is time…We will sketch the arguments later if there is time…

Fundamental Physics Implies Fundamental Physics Implies Various Firm Limits on ComputingVarious Firm Limits on Computing

Speed-of-LightLimit

Thoroughly Confirmed

Physical Theories

UncertaintyPrinciple

Definitionof Energy

Reversibility

2nd Law ofThermodynamics

Adiabatic Theorem

Gravity

Theory ofRelativity

QuantumTheory

ImpliedUniversal Facts

Affected Quantities in Information Processing

Communications Latency

Information Capacity

Information Bandwidth

Memory Access Times

Processing Rate

Energy Loss per Operation

Some limits…Some limits…• Communications latency…Communications latency…

Over distance Over distance dd is at least is at least tt = = dd//cc..• Despite “spooky” non-local-seeming quantum statistics.Despite “spooky” non-local-seeming quantum statistics.

• Information capacity…Information capacity… For systems of given size & energy is finite.For systems of given size & energy is finite.

• Obtained by counting numbers of distinct quantum states.Obtained by counting numbers of distinct quantum states.

• Information bandwidth…Information bandwidth… Limited for flows of given power and cross-sectional area.Limited for flows of given power and cross-sectional area.

• Obtained from capacity and propagation velocity limitsObtained from capacity and propagation velocity limits

• Memory access times…Memory access times… Limited by information density & velocity…Limited by information density & velocity…

• Processing rate…Processing rate… Limited by accessible energy, indirectly by size.Limited by accessible energy, indirectly by size. Also limited by power constraints & energy efficiency.Also limited by power constraints & energy efficiency.

• Energy efficiency…Energy efficiency… Limited by Landauer bound for irreversible computing,Limited by Landauer bound for irreversible computing, No technology-independent limits for reversible computing yet known.No technology-independent limits for reversible computing yet known.

Entropy and InformationEntropy and Information• The following definitions of the entropy content of a The following definitions of the entropy content of a

given physical system can all be shown to be given physical system can all be shown to be essentially equivalent…essentially equivalent… Expected logarithm of the state Expected logarithm of the state improbabilityimprobability 1/ 1/pp..

• Given a probability distribution over system states.Given a probability distribution over system states. Expected size of the smallest compressed description of the Expected size of the smallest compressed description of the

system’s state.system’s state.• Using the best available description language & compressor.Using the best available description language & compressor.

Expected amount of information in the state that cannot be Expected amount of information in the state that cannot be reversibly reversibly decomputed.decomputed.

• Using the best available mechanism.Using the best available mechanism. Expected amount of a system’s information capacity that is Expected amount of a system’s information capacity that is

in usein use• It cannot be used to store newly-computed information for later It cannot be used to store newly-computed information for later

retrieval.retrieval.

Action and Amount of ComputationAction and Amount of Computation• In quantum mechanics, In quantum mechanics,

States are represented as complex-valued vectors States are represented as complex-valued vectors vv,, & temporal transformations are represented by unitary operators & temporal transformations are represented by unitary operators

(generalized rotations) (generalized rotations) UU on the vector space. on the vector space.• The The UU’s may be parameterized as ’s may be parameterized as eeiiHHθθ ((HH hermitian, hermitian, θθ real) real)

• We can characterize theWe can characterize the magnitude magnitude of a given vector rotation of a given vector rotation UvUv = e = eiiHHθθvv by by The The areaarea swept out in the complex plane by the normalized vector swept out in the complex plane by the normalized vector

components as components as θθ is swept from 0 to a given value is swept from 0 to a given value..• Important conjecture: This quantity is basis-independent!Important conjecture: This quantity is basis-independent!

• We can characterize the action performed by a given unitary We can characterize the action performed by a given unitary transform operating on a set of possible transform operating on a set of possible vv’s as the ’s as the maximummaximum rotation magnitude over the rotation magnitude over the vv’s.’s. Or, if we have a probability distribution over initial vectors, we can Or, if we have a probability distribution over initial vectors, we can

define an define an expected actionexpected action accordingly. accordingly.• The connection with computation is provided by showing that it The connection with computation is provided by showing that it

takes a minimum area (action) of takes a minimum area (action) of ππ/4/4 to flip a bit. to flip a bit. I.e.I.e., minimum angle of , minimum angle of ππ/2/2 to rotate to an orthogonal vector. to rotate to an orthogonal vector.

• It takes a minimum action of It takes a minimum action of ππ/2/2 (annihilate/create pair) to (annihilate/create pair) to movemove a state forward by 1 position along an unbounded chain.a state forward by 1 position along an unbounded chain. The total action of a transform gives the total number of such operations.The total action of a transform gives the total number of such operations.

Energy and Rate of ComputationEnergy and Rate of Computation

• The The energyenergy of an eigenvector of of an eigenvector of HH is the is the corresponding eigenvalue.corresponding eigenvalue. The The averageaverage energy of a general quantum state energy of a general quantum state

follows directly from the eigenstate probabilities.follows directly from the eigenstate probabilities.• The average energy is exactly the rate at which The average energy is exactly the rate at which

complex-plane area is swept out (action complex-plane area is swept out (action accumulated).accumulated). In the energy basis, and also in other bases.In the energy basis, and also in other bases.

• Thus, if action is Thus, if action is amount of computationamount of computation, then , then energy is energy is rate of computationrate of computation..

Generalized TemperatureGeneralized Temperature• The concept of temperature can be generalized to apply even to The concept of temperature can be generalized to apply even to

non-equilibrium systems.non-equilibrium systems. Where entropy is less than the maximum.Where entropy is less than the maximum.

• Example:Example: Consider an ideal Fermi gas. Consider an ideal Fermi gas. Heat capacity/fermion is Heat capacity/fermion is CC = π = π22kk22TT/2/2μμ..

• μμ = Fermi energy; = Fermi energy; kk = log e = log e; ; TT = temperature = temperature Equilibrium temperature turns out to be:Equilibrium temperature turns out to be:

• TT = (2/ = (2/ππkk)()(EExxμμ))1/21/2, thus , thus CC = = ππkk((EExx//μμ))1/21/2 where: where:EExx = = EE − − EE00, avg. energy excess/fermion rel. to , avg. energy excess/fermion rel. to TT=0=0

Equilibirum (max) entropy/fermion is: Equilibirum (max) entropy/fermion is: • SSmax max = ∫d= ∫dSS = ∫d′ = ∫d′Q/TQ/T = ∫d = ∫dEExx/T/T = = ππkk((EExx//μμ))1/21/2 = = CC• Consider this to be the total information content Consider this to be the total information content

SSmaxmax = = IItottot = = SS + + XX (entropy plus extropy). (entropy plus extropy). We thus have: We thus have: TT = 2( = 2(EExx//IItottot))

• The temperature is simply 2× the excess energy per unit of total information The temperature is simply 2× the excess energy per unit of total information content.content.

• Note that the expression Note that the expression EExx//IItottot is well-defined even for non- is well-defined even for non-equilibrium states, where the entropy is equilibrium states, where the entropy is SS < < SSmaxmax = = IItottot.. Thus, we can validly ascribe a (generalized) temperature to such states.Thus, we can validly ascribe a (generalized) temperature to such states.

Generalized Temperature Generalized Temperature as “Clock Speed”as “Clock Speed”

• Consider systems such as the Fermi gas, where Consider systems such as the Fermi gas, where TT = = cEcE//II.. Where Where cc is a constant of integration. is a constant of integration. EE is excess energy above the ground state. is excess energy above the ground state. II is total physical info. content is total physical info. content

• For such systems, we can say that the generalized temperature For such systems, we can say that the generalized temperature gives a measure of the energy content, per bit of physical gives a measure of the energy content, per bit of physical information content.information content. EEbb = = cc-1-1TTbb = = cc-1-1kkBBT T ln 2ln 2

• Since energy (we saw) gives the rate of computing, the Since energy (we saw) gives the rate of computing, the temperature therefore gives the rate of computing per bit.temperature therefore gives the rate of computing per bit. In other words, the clock frequency!In other words, the clock frequency!

• For our case For our case cc=2, room temperature corresponds to a max. =2, room temperature corresponds to a max. frequency of: frequency of: ffmaxmax = 2 = 2cc-1-1TTb/b/hh = = kkBB(300 K)(ln 2)/(300 K)(ln 2)/h h = ~4.3 THz= ~4.3 THz Comparable to freq. of room-Comparable to freq. of room-TT IR photons IR photons

• A computational subsystem that is at a generalized temperature A computational subsystem that is at a generalized temperature equal to room temperature can never update its digital state at a equal to room temperature can never update its digital state at a higher frequency than this!higher frequency than this!

Information LimitsInformation Limits

Some Quantities of InterestSome Quantities of Interest

• We would like to know if there are limits on:We would like to know if there are limits on: Infropy densityInfropy density

• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost. Infropy fluxInfropy flux

• = Bits per unit area per unit time= Bits per unit area per unit time• Affects cross-sectional bandwidth, data I/O rates, rates of Affects cross-sectional bandwidth, data I/O rates, rates of

standard-information input & effective entropy removalstandard-information input & effective entropy removal Rate of computationRate of computation

• = Number of distinguishable-state changes per unit time= Number of distinguishable-state changes per unit time• Affects rate of information processing achievable in individual Affects rate of information processing achievable in individual

devicesdevices

Bit Density: No classical limitBit Density: No classical limit• In classical (continuum) physics, even a In classical (continuum) physics, even a singlesingle

particle has a real-valued position+momentumparticle has a real-valued position+momentum All such states are considered physically distinctAll such states are considered physically distinct Each position & momentum coordinate in general requires Each position & momentum coordinate in general requires

an an infiniteinfinite string of digits to specify: string of digits to specify:• xx = 4.592181950149194019240194209490124… meters = 4.592181950149194019240194209490124… meters• pp = 2.393492340940140914291029091230103… kg m/s = 2.393492340940140914291029091230103… kg m/s

Even the smallest system contains an infinite amount of Even the smallest system contains an infinite amount of information! information! No limit to bit density. No limit to bit density.

This picture is the basis for various This picture is the basis for various analog computinganalog computing models studied by some theoreticians.models studied by some theoreticians.

• Wee problem: Wee problem: Classical physics is dead wrong!Classical physics is dead wrong!

The Quantum “Continuum”The Quantum “Continuum”• In QM, still In QM, still uncountably many uncountably many describabledescribable states states

(mathematically possible wavefunctions)(mathematically possible wavefunctions) Can theoretically take infinite info. to describeCan theoretically take infinite info. to describe

• But, not all this info has physical relevance!But, not all this info has physical relevance! States are only physically States are only physically distinguishabledistinguishable when their state when their state

vectors are vectors are orthogonalorthogonal.. States that are only indistinguishably different can only States that are only indistinguishably different can only

lead to indistinguishably different consequences (resulting lead to indistinguishably different consequences (resulting states)states)

• due to linearity of quantum physicsdue to linearity of quantum physics There is There is no physical consequenceno physical consequence from presuming an from presuming an

infinite # of bits in one’s wavefunctioninfinite # of bits in one’s wavefunction

Quantum Particle-in-a-BoxQuantum Particle-in-a-Box

• Uncountably manyUncountably manycontinuouscontinuouswavefunctions?wavefunctions?

• No, can expressNo, can expresswave as a vectorwave as a vectorover countablyover countablymany orthogonalmany orthogonalnormal modesnormal modes.. Fourier transformFourier transform

• High-frequencyHigh-frequencymodes have highermodes have higherenergy (energy (E=hfE=hf); ); energy limits implyenergy limits implythey are unlikely.they are unlikely.

Ways of Counting StatesWays of Counting StatesEntire field of quantum statistical mechanics is about this, Entire field of quantum statistical mechanics is about this,

but here are some simple ways:but here are some simple ways:• For a system w. a constant # of particles:For a system w. a constant # of particles:

# of states = numerical volume of position-momentum # of states = numerical volume of position-momentum configuration space (phase space)configuration space (phase space)

• in units where in units where hh=1.=1.• Approached in macroscopic limit.Approached in macroscopic limit.

Unfortunately, # of particles not usually constant!Unfortunately, # of particles not usually constant!

• Quantum field theory bounds:Quantum field theory bounds: Smith-Lloyd bound. Still ignores gravity.Smith-Lloyd bound. Still ignores gravity.

• General relativistic bounds:General relativistic bounds: Bekenstein bound, holographic bound.Bekenstein bound, holographic bound.

Smith-Lloyd BoundSmith-Lloyd Bound

• Based on counting field modes.Based on counting field modes.• SS = entropy, = entropy, MM = mass, = mass, VV = volume = volume• qq = number of distinct particle types = number of distinct particle types• Lloyd’s bound is tighter by a factor of Lloyd’s bound is tighter by a factor of • Note:Note:

Entropy density scales with 3/4 power of mass-energy densityEntropy density scales with 3/4 power of mass-energy density• E.g.E.g., Increasing entropy density by a factor of 1,000 requires , Increasing entropy density by a factor of 1,000 requires

increasing energy density by 10,000×.increasing energy density by 10,000×.

4/3

4/1

4/1

603

16

2

V

Mcq

V

S

22

Smith ‘95Lloyd ‘00

Examples w. Smith-Lloyd BoundExamples w. Smith-Lloyd Bound

• For systems at the density of water (1 g/cmFor systems at the density of water (1 g/cm33), ), composed only of photons:composed only of photons: Smith’s example: 1 mSmith’s example: 1 m33 box holds 6×10 box holds 6×103434 bits bits

• = 60 kb/Å= 60 kb/Å33

Lloyd’s example: 1 liter “ultimate laptop”, 2×10Lloyd’s example: 1 liter “ultimate laptop”, 2×103131 b b• = 21 kb/Å= 21 kb/Å33

• Cool, but what’s wrong with this picture?Cool, but what’s wrong with this picture? Example requires very high temperature+pressure!Example requires very high temperature+pressure!

• Temperature around 1/2 billion Kelvins!!Temperature around 1/2 billion Kelvins!!• Photonic pressure on the order of 10Photonic pressure on the order of 101616 psi!! psi!!

““Like a miniature piece of the big bang.” -LloydLike a miniature piece of the big bang.” -Lloyd Probably not feasible to implement any time soon!Probably not feasible to implement any time soon!

More Normal TemperaturesMore Normal Temperatures

• Let’s pick a more reasonable temperature: Let’s pick a more reasonable temperature: 1356 K (melting point of copper):1356 K (melting point of copper): Entropy density of light only 0.74 bits/Entropy density of light only 0.74 bits/mm33!!

• Less than the bit density in a DRAM today!Less than the bit density in a DRAM today! Bit size comparable to wavelength of optical-frequency Bit size comparable to wavelength of optical-frequency

light emitted by melting copperlight emitted by melting copper

• Lesson: Lesson: Photons are not a viable information storage Photons are not a viable information storage medium at ordinary temperatures.medium at ordinary temperatures. Not dense enough.Not dense enough.

• CPUs that do logic with optical photons can’t have CPUs that do logic with optical photons can’t have logic devices packed very densely.logic devices packed very densely.

Entropy Density of SolidsEntropy Density of Solids

• Can easily calculate from standard empirical Can easily calculate from standard empirical thermochemical data.thermochemical data.

• Obtain entropy by integrating heat capacity ÷ Obtain entropy by integrating heat capacity ÷ temperature, as temperature increases…temperature, as temperature increases… Example result, for copper:Example result, for copper:

• Has one of the highest entropy densities among pure elements at Has one of the highest entropy densities among pure elements at atmospheric pressureatmospheric pressure

• @ room temperature: 6 bits/atom, 0.5 b/Å@ room temperature: 6 bits/atom, 0.5 b/Å33

• At boiling point: 1.5 b/ÅAt boiling point: 1.5 b/Å33

Cesium has one of the highest #bits/atom at room Cesium has one of the highest #bits/atom at room temperature, about 15. -But only 0.13 b/Åtemperature, about 15. -But only 0.13 b/Å33

Lithium has a high #bits/mass, 0.7 bits/amu.Lithium has a high #bits/mass, 0.7 bits/amu.

Related toconductivity?

1012×denser

thanits light!

Some Quantities of InterestSome Quantities of Interest

• We would like to know if there are limits on:We would like to know if there are limits on: Infropy densityInfropy density

• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost. Infropy fluxInfropy flux

• = Bits per unit area per unit time= Bits per unit area per unit time• Affects cross-sectional bandwidth, data I/O rates, rates of Affects cross-sectional bandwidth, data I/O rates, rates of

standard-information input & effective entropy removalstandard-information input & effective entropy removal Rate of computationRate of computation

• = Number of distinguishable-state changes per unit time= Number of distinguishable-state changes per unit time• Affects rate of information processing achievable in individual Affects rate of information processing achievable in individual

devicesdevices

Smith-Lloyd BoundSmith-Lloyd Bound

• Based on counting orthogonal field modes.Based on counting orthogonal field modes.• SS = entropy, = entropy, MM = mass, = mass, VV = volume = volume• qq = number of distinct particle types = number of distinct particle types• Lloyd’s bound is tighter by a factor of Lloyd’s bound is tighter by a factor of • Note:Note:

Entropy density scales with 3/4 power of mass-energy densityEntropy density scales with 3/4 power of mass-energy density• E.g.E.g., Increasing entropy density by a factor of 1,000 requires , Increasing entropy density by a factor of 1,000 requires

increasing energy density by 10,000×.increasing energy density by 10,000×.

nat603

16

2

4/3

4/1

4/1

V

Mcq

V

S

22

Smith ‘95Lloyd ‘00

Whence this scaling relation?Whence this scaling relation?

• Note that in the field theory limit, Note that in the field theory limit, SS EE3/43/4.. Where does this come from?Where does this come from?

• Consider a typical freq. in field spectrumConsider a typical freq. in field spectrum Note that the minimum size of aNote that the minimum size of a

given wavelet is ~its wavelength given wavelet is ~its wavelength ..

• # of distinguishable wave-packet location states in a # of distinguishable wave-packet location states in a given volume given volume 1/ 1/33

Each such state carries a little entropyEach such state carries a little entropy• occupation number of that state (# of photons in it)occupation number of that state (# of photons in it)

1/1/33 particles each energy particles each energy 1/1/, , 1/1/4 4 energyenergy• SS1/1/33 EE1/1/44 SSEE3/43/4

Whence the distribution?Whence the distribution?

• Could the use of more particles (with less energy Could the use of more particles (with less energy per particle) yield greater entropy?per particle) yield greater entropy? What frequency spectrum (power level or particle What frequency spectrum (power level or particle

number density as a function of frequency) gives the number density as a function of frequency) gives the largest # states?largest # states?

Note Note a minimum particle energy due to box size a minimum particle energy due to box size

• No. The Smith-Lloyd bound is based on the No. The Smith-Lloyd bound is based on the blackbodyblackbody radiation spectrum. radiation spectrum. We know this spectrum has the maximum infropy among We know this spectrum has the maximum infropy among

abstract states, b/c it’s the equilibrium state.abstract states, b/c it’s the equilibrium state.• Empirically verified in hot ovens, Empirically verified in hot ovens, etc.etc.

General-Relativistic BoundsGeneral-Relativistic Bounds

• The Smith-Lloyd bound does not take into account The Smith-Lloyd bound does not take into account the effect of gravity.the effect of gravity.

• Earlier bound from Bekenstein: Derives a limit on Earlier bound from Bekenstein: Derives a limit on entropy from black-hole physics:entropy from black-hole physics:

SS < 2 < 2ER ER / / ccEE = total energy = total energyRR = radius of system = radius of system

• Limit only attained by black holes!Limit only attained by black holes! Black holes have 1/4 nat entropy per square Planck length Black holes have 1/4 nat entropy per square Planck length

of surface (event horizon) area.of surface (event horizon) area.• Minimum size of a nat: 2 Planck lengths, squareMinimum size of a nat: 2 Planck lengths, square

4×1039 b/Å3

average ent. dens.of a 1-m radius

black hole!(MassSaturn)

The Holographic BoundThe Holographic Bound

• Based on Bekenstein black-hole bound.Based on Bekenstein black-hole bound.• The maximum entropy within The maximum entropy within anyany surface of area surface of area AA

(independent of energy!) is(independent of energy!) is AA/(2/(2LLPP))22

LLPP is Planck length (see lecture on units) is Planck length (see lecture on units)

• Implies any 3D object (of any size) could be Implies any 3D object (of any size) could be completely defined via a flat (2D) “hologram” on its completely defined via a flat (2D) “hologram” on its surface having Planck-scale resolution.surface having Planck-scale resolution. Bound is only really achieved by a black hole with event Bound is only really achieved by a black hole with event

horizon=that surface.horizon=that surface.

Do Black Holes Destroy Do Black Holes Destroy Information?Information?

• Currently, it seems that no one completely understands how Currently, it seems that no one completely understands how information is preserved during black hole accretion for later information is preserved during black hole accretion for later re-emission as Hawking radiation.re-emission as Hawking radiation. Via infinite time dialation at surface?Via infinite time dialation at surface?

• Some researchers (Some researchers (e.g.e.g. Hawking) claimed that black Hawking) claimed that black holes must be doing something irreversible in their holes must be doing something irreversible in their interior (destroying information).interior (destroying information). The arguments for this seem not very rigorous...The arguments for this seem not very rigorous...

• The issue is not completely resolved, but I have many The issue is not completely resolved, but I have many papers on it if you’re interested.papers on it if you’re interested. Incidentally, Hawking recently conceded a bet on this.Incidentally, Hawking recently conceded a bet on this.

Implications of Density LimitsImplications of Density Limits

• Minimum device sizeMinimum device size thus minimum communication latency (as per thus minimum communication latency (as per

earlier).earlier). Minimum device cost, given a minimum cost of Minimum device cost, given a minimum cost of

matter/energy.matter/energy.

• Implications for communications bandwidth Implications for communications bandwidth limits (coming up)limits (coming up)

Communication LimitsCommunication Limits

• LatencyLatency (propagation-time delay) limit from (propagation-time delay) limit from earlier, due to speed of light.earlier, due to speed of light. Teaches us scalable interconnection technologiesTeaches us scalable interconnection technologies

• BandwidthBandwidth (infropy rate) limits: (infropy rate) limits: Classical information-theory limit (Shannon)Classical information-theory limit (Shannon)

• Limit, per-channel, given signal bandwidth & SNR.Limit, per-channel, given signal bandwidth & SNR. Limits based on field theory (Smith/Lloyd)Limits based on field theory (Smith/Lloyd)

• Limit given only area and power.Limit given only area and power.• Applies to I/O, cross-sectional bandwidths in parallel Applies to I/O, cross-sectional bandwidths in parallel

machines, and entropy removal rates.machines, and entropy removal rates.

Hartley-Shannon LawHartley-Shannon Law• The maximum information rate (capacity) of a The maximum information rate (capacity) of a

single wave-based communication channel is:single wave-based communication channel is:CC = = BB log (1+ log (1+SS//NN))

BB = bandwidth of channel in frequency units = bandwidth of channel in frequency units SS = signal power level = signal power level NN = noise power level = noise power level

• Law not sufficiently powerful for our purposes!Law not sufficiently powerful for our purposes! Does not tell us Does not tell us how manyhow many effective channels are effective channels are

possible, given available power and/or area.possible, given available power and/or area. Does not give us Does not give us anyany limit if we are allowed to increase limit if we are allowed to increase

bandwidth or decrease noise arbitrarily.bandwidth or decrease noise arbitrarily.

Density & FluxDensity & Flux• Note that any time you have:Note that any time you have:

a limit a limit on density (per volume) of something on density (per volume) of something & a limit & a limit vv on its propagation velocity on its propagation velocity

• this automatically implies:this automatically implies: a limit a limit FF = = vv on the on the fluxflux

• by which I mean rate per time per areaby which I mean rate per time per area

• Note also we always have a limit Note also we always have a limit cc on velocity! on velocity! At speeds near At speeds near cc must account for relativistic effects must account for relativistic effects Slower velocities also relevant:Slower velocities also relevant:

• electron saturation velocity in various materialselectron saturation velocity in various materials• velocity of air or liquid coolant in a cooling systemvelocity of air or liquid coolant in a cooling system

• Thus density limit Thus density limit implies flux limit implies flux limit FF==cc

v

Cross-section

Relativistic EffectsRelativistic Effects• For normal matter (bound massive-particle states) For normal matter (bound massive-particle states)

moving at a velocity moving at a velocity vv near near cc:: Entropy density increases by factor Entropy density increases by factor = (1 = (1((v/cv/c))22))11

• Due to relativistic length contractionDue to relativistic length contraction But, energy density increases by factor But, energy density increases by factor 22

• Both length contraction & mass amplificationBoth length contraction & mass amplification entropy density scales up only w. square root (1/2 power) entropy density scales up only w. square root (1/2 power)

of energy density from high velocityof energy density from high velocity

• Note that light travels at Note that light travels at cc already, already,• & its entropy density scales with energy density to the & its entropy density scales with energy density to the

3/4 power. 3/4 power. Light wins as Light wins as vvcc.. If you want to maximize entropy/energy fluxIf you want to maximize entropy/energy flux

Entropy Flux Using LightEntropy Flux Using Light

• FFSS = = entropy fluxentropy flux

• FFEE = = energy fluxenergy flux

SBSB = Stefan-Boltzmann constant, = Stefan-Boltzmann constant, 22kkBB44/60/60cc2233

• Derived from same field-theory arguments as the Derived from same field-theory arguments as the density bound.density bound.

• Again, blackbody spectrum optimizes entropy flux Again, blackbody spectrum optimizes entropy flux given energy fluxgiven energy flux It is the equilibrium spectrumIt is the equilibrium spectrum

434134

ESBS FF

Smith ‘95

Entropy Flux ExamplesEntropy Flux Examples• Consider a 10cm-wide, flat, square wireless Consider a 10cm-wide, flat, square wireless

tablet with a 10 W power supply.tablet with a 10 W power supply. What’s it’s maximum rate of bit transmission?What’s it’s maximum rate of bit transmission?

• Independent of spectrum used, noise floor, Independent of spectrum used, noise floor, etc.etc.

• Answer: Answer: Energy flux 10 W/(2·(10 cm)Energy flux 10 W/(2·(10 cm)22) (use both sides)) (use both sides) Smith’s formula gives 2.2×10Smith’s formula gives 2.2×102121 bps bps

• What’s the rate What’s the rate per square nanometerper square nanometer surface? surface? Only 109 kbps! (ISDN speed, in a 100 GHz CPU?)Only 109 kbps! (ISDN speed, in a 100 GHz CPU?) 100 Gbps/nm100 Gbps/nm22 nearly 1 GW power! nearly 1 GW power!

Light is not infropically dense enough for high-BW comms. between densely packed nanometer-scale devices at reasonable power levels!!!

Entropy Flux w. Atomic MatterEntropy Flux w. Atomic Matter

• Consider liquid copper (~1.5 b/ÅConsider liquid copper (~1.5 b/Å33) moving along at a ) moving along at a leisurely 10 cm/s…leisurely 10 cm/s… BW=1.5x10BW=1.5x102727 bps through the 10-cm wide square! bps through the 10-cm wide square!

• A million times higher BW than with 10W light!A million times higher BW than with 10W light! 150 Gbps/nm150 Gbps/nm22 entropy flux! entropy flux!

• Plenty for nano-scale devices to talk to their neighborsPlenty for nano-scale devices to talk to their neighbors Most of this entropy is in the conduction electrons...Most of this entropy is in the conduction electrons...

• Less conductive materials have much less entropyLess conductive materials have much less entropy

• Lesson:Lesson: For maximum bandwidth density at realistic power levels, For maximum bandwidth density at realistic power levels,

encode information using states of matter (electrons) rather encode information using states of matter (electrons) rather than states of radiation (light).than states of radiation (light).

Exercise: Kinetic energy flux?

Some Quantities of InterestSome Quantities of Interest

• We would like to know if there are limits on:We would like to know if there are limits on: Infropy densityInfropy density

• = Bits per unit volume= Bits per unit volume• Affects physical size and thus propagation delayAffects physical size and thus propagation delay

across memories and processors. Also affects cost.across memories and processors. Also affects cost. Infropy fluxInfropy flux

• = Bits per unit area per unit time= Bits per unit area per unit time• Affects cross-sectional bandwidth, data I/O rates, rates of Affects cross-sectional bandwidth, data I/O rates, rates of

standard-information input & effective entropy removalstandard-information input & effective entropy removal Rate of computationRate of computation

• = Number of distinguishable-state changes per unit time= Number of distinguishable-state changes per unit time• Affects rate of information processing achievable in individual Affects rate of information processing achievable in individual

devicesdevices

Speed LimitsSpeed Limits

The Margolus-Levitin BoundThe Margolus-Levitin Bound

• The maximum rate The maximum rate at which a system can transition at which a system can transition

between distinguishable (orthogonal) states is:between distinguishable (orthogonal) states is: 4( 4(EE EE00)/)/hh

where:where:• EE = average energy (expectation value of energy over all states, = average energy (expectation value of energy over all states,

weighted by their probability)weighted by their probability)• EE00 = energy of lowest-energy or = energy of lowest-energy or groundground state of system state of system• hh = Planck’s constant (converts energy to frequency) = Planck’s constant (converts energy to frequency)

• Implication for computing:Implication for computing: A circuit node can’t switch between 2 logic states faster A circuit node can’t switch between 2 logic states faster

than this frequency determined by its energy.than this frequency determined by its energy.

Example of Frequency Bound Example of Frequency Bound

• Consider Lloyd’s 1 liter, 1 kg “ultimate laptop”Consider Lloyd’s 1 liter, 1 kg “ultimate laptop” Total gravitating mass-energy Total gravitating mass-energy EE of 9 of 910101616 J J Gives a limit of 5Gives a limit of 510105050 bit-operations per second! bit-operations per second! If laptop contains 2If laptop contains 210103131 bits (photonic maximum), bits (photonic maximum),

• each bit can change state at a frequency of 2.5each bit can change state at a frequency of 2.510101919 Hz (25 Hz (25 EHz)EHz)

12 billion times higher-frequency than today’s 2 GHz Intel 12 billion times higher-frequency than today’s 2 GHz Intel processorsprocessors

250 million times higher-frequency than today’s 100 GHz 250 million times higher-frequency than today’s 100 GHz superconducting logicsuperconducting logic

• But, the Margolus-Levitin limit may be far from But, the Margolus-Levitin limit may be far from achievable!achievable!

More Realistic EstimatesMore Realistic Estimates• Most of the energy in complex stable structures is not Most of the energy in complex stable structures is not

accessibleaccessible for computational purposes... for computational purposes... Tied up in the rest masses of atomic nuclei,Tied up in the rest masses of atomic nuclei,

• form anchor points for electron orbitalsform anchor points for electron orbitals mass & energy of “core” atomic electrons,mass & energy of “core” atomic electrons,

• fill up low-energy states not involved in bonding, fill up low-energy states not involved in bonding, & of electrons involved in atomic bonds& of electrons involved in atomic bonds

• needed to hold the structure togetherneeded to hold the structure together

• Conjecture:Conjecture: Can obtain tighter valid quantum Can obtain tighter valid quantum bounds on infropy densities & state-transition rates bounds on infropy densities & state-transition rates by considering only the by considering only the accessibleaccessible energy. energy. Energy whose state-infropy is manipulable.Energy whose state-infropy is manipulable.

More Realistic ExamplesMore Realistic Examples

• Suppose the following system is accessible:Suppose the following system is accessible:1 electron confined to a (10 nm)1 electron confined to a (10 nm)33 volume, at an volume, at an average potential of 10 V above ground state.average potential of 10 V above ground state. Accessible energy: 10 eVAccessible energy: 10 eV Accessible-energy density: 10 eV/(10 nm)Accessible-energy density: 10 eV/(10 nm)33

Maximum entropy in Smith bound: 1.4 bits?Maximum entropy in Smith bound: 1.4 bits?• Not clear whether bound is applicable to this case.Not clear whether bound is applicable to this case.

Maximum rate of change: 9.7 PHzMaximum rate of change: 9.7 PHz• 5 million × typical frequencies in today’s CPUs5 million × typical frequencies in today’s CPUs• 100,000 × frequencies in today’s superconducting logics100,000 × frequencies in today’s superconducting logics

Summary of Fundamental LimitsSummary of Fundamental Limits