Tight Lower Bounds for the Distinct Elements

Problem

David WoodruffMIT

dpwood@mit.edu

Joint work with Piotr Indyk

The Problem

• Stream of elements a1, …, an each in {1, …, m}

• Want F0 = # of distinct elements

• Elements in adversarial order• Algorithms given one pass over stream• Goal: Minimum-space algorithm

0113734 …

A Trivial Algorithm

…0113734

• Keep m-bit characteristic vector v of stream

• j in stream $ vj = 1 • F0 = wt(10011011) = 5• Space = m

00000000 10011011

Can we do better?

Negative Results

• Any algorithm computing F0 exactly must use (m) space [AMS96]

• Any deterministic alg. that outputs x with |F0 – x| < F0 must use (m) space [AMS96]

• What about randomized approximation algorithms?

Rand. Approx. Algorithms for F0

• O(log log m/2 + log m log 1/) alg. outputs x with Pr[| F0 – x| < F0 ] > ¾ [BJKST02]

• Lots of hashing tricks

Is this optimal?

• Previous lower bounds• (log m) [AMS96]• (1/) [Bar-Yossef]

• Open Problem of [BJKST02]: GAP: 1/ << 1/2

Idea Behind Lower Bounds

x 2 {0,1}m

y 2 {0,1}m

Stream s(x) Stream s(y)

(1 § ) F0 algorithm A

(1 § ) F0 algorithm A

Internal state of A

• Compute (1 § ) F0(s(x) ± s(y)) w.p. > ¾• Idea: If can decide f(x,y) w.p. > ¾, space used by A at least f’s rand. 1-way comm. complexity

S

Alice Bob

Randomized 1-way comm. complexity

• Boolean function f: X £ Y ! {0,1}

• Alice has x 2 X, Bob y 2 Y. Bob wants f(x,y)

• Only 1 message sent: must be from Alice to Bob

• Comm. cost of protocol = expected length of longest message sent over all inputs.

-error randomized 1-way comm. complexity of f, R(f), is comm. cost of optimal protocol computing f w.p. ¸ 1-

• How do we lower bound R(f)?

The VC Dimension [KNR] • F = {f : X ! {0,1}} family of Boolean functions• f 2 F is length-|X | “bit string”

• For S µ X, shatter coefficient SC(fS) of S is |{f |S}f 2 F| = # distinct bit strings when F restricted to S

• SC(F, p) = maxS 2 X, |S| = p SC(fS)

• If SC(fS) = 2|S|, S shattered by F

• VC Dimension of F, VCD(F), = size of largest S shattered by F

Shatter Coefficient Theorem

• Notation: For f: X £ Y ! {0,1}, define: fX = { fx(y) : Y ! {0,1} | x 2 X },

where fx(y) = f(x,y)

• Theorem [BJKS]: For every f: X £ Y ! {0,1}, every p ¸ VCD( fX ),

R1/4(f) = (log(SC(fX, p)))

The (1/) Lower Bound [Bar-Yossef]

• Alice has x 2R {0,1}m, wt(x) = m/2 • Bob has y 2R {0,1}m, wt(y) = m and:

• Either wt(x Æ y) = 0 OR wt(x Æ y) = m f(x,y) = 0 f(x,y) = 1

• R1/4(f) = (VCD(fX)) = (1/) [Bar-Yossef]• s(x), s(y) any streams w/char. vectors x, y

• f(x,y) = 1 ! F0(s(x) ± s(y)) = m/2• f(x,y) = 0 ! F0(s(x) ± s(y)) = m/2 + m• (1+’)m/2 < (1 - ’)(m/2 + m) for ’ = ()

• Hence, can decide f ! F0 alg. uses (1/) space

Our Results

• Remainder of talk: (1/2) lower bound for = (m-1/(9+k)) for any k > 0.

! O(log log m/2 + log m log 1/) upper bound almost optimal

IDEA: Reduce from protocol for computing dot product

The Promise Problem

• X = {x 2 [0,1]t, ||x|| = 1 and 9 y 2 Y s.t. (x,y) 2 }• We lower bound R1/4(f) via SC(fX, t)

• t = (1/2), Y = basis of unit vectors of Rt

x 2 [0,1]t

||x|| = 1y 2 Y

Alice Bob

Promise Problem : hx,yi = 0 hx,yi = 2/t1/2

f(x,y) = 0 OR f(x,y) = 1

Bounding SC(fX, t)

• Theorem: SC(fX, t/4) = 2(t)

• Proof:

1. 8 T ½ {Y} s.t. |T| = t/4, put xT = (2/t1/2) ¢ e 2 T e

2. Define X1 ½ X as X1 = {xT | T ½ {Y}, |T| = t/4}

3. Claim: 8 s 2 {0,1}t w/ wt(x) = t/4, s 2 truth tab. of fX1

4. Proof:

1. Let s 2 {0,1}t with 1s in positions i1, …, it/42. Put T = {ei1, …, eit/4}. 8 e 2 T, he, xTi = 2/t1/2 = 2

3. 8 e 2 Y - T, h e, xT i = 0

5. There are 2(t) such s.

Bounding R1/4(f)

• Corollary:

• ReductionReduction: we need protocol computing f with communication = space used by any (1 § ) F0 approx. alg.

Reduction• Recall:

• hx,yi = 0 if f(x,y) = 0 • hx,yi = 2/t1/2 if f(x,y) = 1

• Goal:Goal: Reduce “separation” of hx,yi to separation of F0(s(x) ± s(y)) for streams s(x),s(y) Alice/Bob can derive from x,y

• Use relation: ||y-x||2 = ||y||2 + ||x||2 – 2hx, yi• f(x,y) = 0 ! ||y-x|| = 21/2

• f(x,y) = 1 ! ||y-x|| < 21/2 (1- 1/t1/2) = 21/2 (1 - ())

Overview of Reductionx 2 [0,1]t

||x|| = 1y 2 E

1. Low-distortion embedding : l2t ! l1poly(t)

2. Rational Approximation

(x) (y)

3. Scale rationals to integers s

4. Convert integer coords to unary to get {0,1} vectors x’,y’x’ y’

F0(s(x’) ± s(y’)) can decide f(x,y) w.p. ¸ 3/4

F0 Alg

F0(s(x’) ± s(y’))

F0 AlgState

s(x’) s(y’)

Embedding l2t into l1poly(t)

• A (1+)-distortion embedding : l2t ! l1d is mapping s.t. 8 p,q 2 l2t,

• Theorem [FLM77]: 8 9 a (1+ )-distortion embedding : l2t ! l1d with:

Embedding l2t into l1d

x 2 [0,1]t

||x|| = 1y 2 E

Low-distortion embedding

: l2t ! l1d

(x) (y)

• Using Theorem [FLM77], Alice/Bob get (x), (y) 2 Rd with d = O(t ¢ (log 1/) / 2):

• specified later

Rational Approximation

• z = z(t): N ! N; assume z ¸ d

• Approximate each coord. of output of embedding by integer multiple of 1/z

Scaling

• Alice (resp. Bob) multiplies each coord. of (resp. ) by z

• Obtains s( ) (resp. s( )

• Claim: coords. are integers in range [-2z, 2z]

• Proof: 1. | | · |(¢)| + d/z · 22. |s( )| = z| |

Converting to Unary

• For i=1 to d• j à s( )i

• Replace s( )i with 12z+j02z-j

• Bob does same for s( )

• x’, y’ denote new length 4dz bitstrings

• wt(x’) = |s( )|, wt(y’) = |s( )| (x’,y’) = |s( ) – s( )|

Reducing (x’,y’) to F0

• Alice (Bob) chooses stream ax’ (ay’) with char. vector x’ (y’).

• Lemma: If 1 < wt(x’), wt(y’) < 2, then:

1 + (x’,y’)/2 < F0(ax’ ± ay’) < 2 + (x’,y’)/2

Follows from fact: F0(ax’ ± ay’) = wt(x’ Ç y’)

Reducing (x’,y’) to F0

• Use lemma to show:

• Set = (), z = (1/5 log 1/) so that two cases distinguished by (1 § ()) F0 alg

Conclusions

• ax’, ay’ must be in universe of size ¸ 4zd = (log (1/)/9)

• Reduction only valid if 4zd · m (1/2) bound for = (m-1/(9+k)) 8 k > 0.

• Recently lower bound improved to:• (1/2) for ¸ m-1/2, which is optimal• Find set of vectors directly in Hamming space via involved prob. method argument