A 3-Query PCP over integersa.k.a

Solving Sparse Linear Systems

Prasad Raghavendra

Venkatesan Guruswami

Linear Equations

Given a system of linear equations over reals,Find a solution.

.Easy,

Use Gaussian elimination.

Noise?

Given a set of linear equations for which there is a solution satisfying 99% of the equations,

What is the best solution that can be efficiently found?

Can we atleast satisfy 1% of the equations?

10 years ago [Håstad STOC97, JACM 01]

For any prime p, ε > 0, given a set of linear equations modulo p , it is NP-hard to distinguish between:

• (1 – ε) – fraction of the equations can be satisfied.• 1/p + ε – fraction of the equations can be satisfied.

All equations are of the form Xi + Xj = Xk + c (mod p)

X1 + X2 = X3 + 10 (mod p)X1 + X3 = X5 + 17 (mod p)X9 + X4 = X3 + 23 (mod p)X11 + X2 = X31 + 1 (mod p)X1 + X2 = X7 - 1 (mod p)X1 + X3 = X8 + p-10 (mod p)……..……..X9 + X7 = X3 + p/2 (mod p)X5 + X2 = X7 + 10 (mod p)

It is a 3-Query Probabilistically Checkable Proof system for NP

Just have to read values of 3 variables to check an equation.

Håstad’s 3-Query PCP[STOC97, JACM 01]

Can be verified by 3 queries

Reals?

NP-hard[Guruswami-Raghavendra 06, Feldman-Gopalan-Khot-Ponnuswami 06]For any ε,δ > 0, Given a set of linear equations over reals, it is NP-hard to distinguish between the following two cases:• There is a solution that satisfies 1 – ε fraction of the equations.• No solution satisfies more than δ fraction of the equations.

Unlike Hastad’s result, equations are not sparse

Sparse Equations?

• Solving sparse systems of equations important for many applications.

• In the spirit of PCP theorem..

• Sparse equations have important connections to PCPs, linearity testing, Unique Games conjecture.

Sparse Equations over Reals

For any ε,δ > 0, Given a set of sparse linear equations, it is NP-hard to distinguish between:

• (1 – ε) – fraction of the equations can be satisfied.• δ – fraction of the equations can be satisfied. X1 + X2 = X3 + 10

X1 + X3 = X5 + 17…X9 + X4 = X3 + 23 X2 + X6 = X7 + 27

Some fixed constant accuracy, say ±1

Label Cover Problem

U, V : set of verticesE : set of edges{1,2… R} : set of labels πe: constraint on edge e

An assignment A satisfies an edge e = (u,v) E if

πe (A(u)) = A(v)

123..R

123..Rπe

U V

u

v

Find an assignment A that satisfies maximum number of edges

3

π e (3)=2

7

5

2

3

3

1

4

1

2

5

6

Label Cover with Long Codes

πe

7

5

2

3

3

1

4

1

2

5

6

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

Write Long Codes of the Labels instead of the labels itself

Long Code

A long code over a finite field F is a function:

Gi : F X F … X F XF F

Gi(x1 , x2, … xn ) = xi

• n different long codes.• Long code over Fp represented by a table of pn values.

• Linear Function.

Extending Hastad’s result to integers

πe

7

5

2

3

3

1

4

1

2

5

6

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0 X1

X2

G2 (x1 , x2 ) = x2

A Long Code over integers is an infinite object.

Use long code over integers

Just Truncate the long code!

Core Problem

4 4 4 4 43 3 3 1 32 2 1 2 21 1 1 1 10 0 0 0 0 X1

X2

0 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 4 X1

X2

G2 (x1 , x2 ) = x2 G1 (x1 , x2 ) = x1

?=

Given two supposed long codes, query 3 locations and test if they are close to some long code

If test succeeds, must decode a small set of possible labels

Proof Obstacles

• Linearity Testing

• Decoding Labels

Linearity Testing

Given a function from an group G1 to group G2 (both abelian)

A : G1 -> G2

Pick x,y uniformly at random from G1

Test if

A(x) + A(y) = A(x+y)

[Blum-Luby-Rubinfeld] With G1 = {0,1}n ,G2 = {0,1}, if A is δ- far from linear function, then the test rejects with probability at least δ

Derandomized Linearity Testing

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

For sufficiently large primes p,

Linearity testing on truncated long code

= Testing modulo p

This should imply a derandomized linearity test.Total randomness used independent of the prime.

p

Proof Obstacles

• Linearity Testing

• Decoding Labels

Fourier Analysis

• One Fourier coefficient corresponding to every linear function

Pω(x) = ω•x for ω =(ω1 , ω2 ,… ωR ) in FpR

Â(ω) measures similarity with Pω(x) = ω•x

Â(ω) = E[ A(x)e-iω•x ]

Hastad’s Decoding

Pick a large Fourier coefficient Â(ω) of the long code, randomly pick a nonzero coordinate ωi

Decode to label iNot too many large Fourier Coefficients

Parseval’s Identity

Obstacle

The distribution is not uniform, so a Fourier coefficient that appears is

There could be exponentially many large Fourier coefficients!

P(x)

A(x)

P(x)A(x)

Large values in Fourier spectrum are

clustered.

Function Fourier Transform

Decoding Labels

Pick a large Fourier coefficient AP(ω) , randomly pick one of its large coordinate ωi

Assign label i to the vertex

•All large Fourier coefficients in the same cluster, will yield the same label with high probability.

• There are very few clusters, so there are very few possible choices

Conclusion

• Sparse linear equations over real numbers are hard to solve even with little noise.

• In a weak sense, complete Derandomization of linearity testing is possible.

• Two variable linear equations over reals?

Thank You

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

πAB

Randomly pick a vector x = (x1,x2,.. xR)

Definex o π = (xπ(1), xπ(2) , xπ(3) ….xπ(R))

Test if a(x o π) = b(x)

Testing an EdgeI will just assign 0

to everything!

Testing an Edge

Randomly pick a vector x = (x1,x2,.. xR)

Definex o π = (xπ(1), xπ(2) , xπ(3) ….xπ(R))

A(x o π) = B(x)

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

πAB

For a long code a,

a(x + 1 ) = a(x) + 1

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0 X1

X2

G2 (x1 , x2 ) = x2

A(x o π – t11) + t1 = B(x – t21) + t2

I will give something that does not look

linear at all

Testing an Edge

Randomly pick a vector x = (x1,x2,.. xR)

Definex o π = (xπ(1), xπ(2) , xπ(3) ….xπ(R))

Randomly pick y = (y1,y2,.. yR)

Test ifa(x o π + y) – a(y) = b(x) + c

Long Code is a linear function!

a(x o π + y ) – a(y) = a(x o π)

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

πAB A(x) = (x1 + x2 + ...xR)/R

(ε,δ) – concentrated distribution

All Fourier coefficients of P that are 2πδ away from origin are bounded by ε

4πδ ε

Examples• Epsilon Biased Spaces over [0,1]n are (ε, ½) – concentrated.

• Epsilon Biased Spaces over Fp are (ε, 1/p) – concentrated. [BenSasson-Sudan-Vadhan-Widgerson] use Epsilon biased spaces to derandomize low degree tests(including linearity)

•Any sufficiently slowly decaying probability distribution over integers.

Hardness of Label Cover

There exists γ > 0 such thatGiven a label cover instance Г =(U,V,E,R,π), it is

NP-hard to distinguish between :• Г is completely satisfiable• No assignment satisfies more than 1/Rγ

fraction of the edges.

[Raz 98]

Testing an Edge

For a function π : [1,2,.. R] -> [1,2..R]

A vector x = (x1,x2,.. xR) Definex o π = (xπ(1), xπ(2) , xπ(3) ….xπ(R))

a(x o π) = b(x)

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

πeab

1

2

1

2

πe

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0

X1

X2

A (x1 , x2 ) = x2

0 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 4

X1

X2

B (x1 , x2 ) = x1

a(2, 4) = b(4,2)

A Linear Equation on Long code symbols

Hastad’s 3 Query PCP

Randomly pick a vector xDefine

x o π = (xπ(1), xπ(2) , xπ(3) ….xπ(R))

Randomly pick yPerturb each coordinate of x o π + y independently with

probability ε. To perturb just change the value to anything else in Fp

Test ifa(x o π + y+μ) – a(y) = b(x) + c

Long codes/Dictator functions are stable against noise in the coordinates

Arithmetization

Define A(x) = ωa(x) = e2πia(x)/p

B(y) = ωb(y) = e2πib(y)/p

Then : a(x o π + y+μ) – a(y) - b(x) = 0if and only if

1/p ∑ (A(y)B(x) A(x o π + y+μ) )j = 1

Soundness Argument

• As Linearity is tested, a(x) must have some similarity to a linear function. There have to be large Fourier coefficients Â(ω)

• As we force a(x + 1) = a(x) + 1 the function a(x) is not similar to constant function. Thus, there are some nonzero ω with large Â(ω)

• There are large Â(ω) with ω having few non-zero labels.

Obstacles

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0 X1

X2

G2 (x1 , x2 ) = x2

Truncated region no more a group.

For a constant fraction of x and y, (x+y) is outside the region.

There could be exponentially many (in the dimension of space) large Fourier coefficients.

Modified TestP, P’ be decaying and (ε,δ)- concentrated distributions

Pick x from distribution PPick y from distribution P’

Perturb each coordinate of x o π + y independently with

probability ε. To perturb, just change the value by a random number < M

Test ifA(x o π + y+μ) – A(y) = B(x) + c

P’

P

P much more flatter than P’

Fourier Analysis (continued)

Inverse Fourier Transform

Not too many large Fourier Coefficients

Parseval’s Identity

Time Domain Fourier Domain

P(x) = 1

Properties

• For a long code a,

a(x + 1 ) = a(x) + 1

• Long codes/Dictator functions are stable against noise in the coordinates.

4 4 4 4 43 3 3 3 32 2 2 2 21 1 1 1 10 0 0 0 0 X1

X2

0 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 40 1 2 3 4 X1

X2

G2 (x1 , x2 ) = x2

G1 (x1 , x2 ) = x1

Two dimensional long codes over F5