Sum of SquaresA lens on average case complexity

Tensor PCA (for 4-tensors)

Tensor PCAgiven: a 4-tensor !: # $ → ℝ,

! = ( ⋅ *⊗$ +-!.,0,1,ℓ = *.*0*1*ℓ +-.,0,1,ℓ where

unknown signal * ∈ ℝ4, * = 1 and -.,0,1,ℓ ∼ 778 9:;<<7:#goal: recover the signal * ∈ ℝ4

( : controls the signal-to-noise ratio.

Tensor PCA (for 4-tensors)

TensorPCAgiven: a 4-tensor !: # $ → ℝ,

! = ( ⋅ *⊗$ +-where

unknown signal * ∈ℝ/, * = 1 and -2,3,4,ℓ ∼ 778 9:;<<7:#goal: recover * ∈ ℝ/

( ∶ <7>#:?#@7<A B:C7@D@EF

;C:C7@#:?D@E

F?A*7CG

IMPOSSIBLE

COMPUTATIONALLY

EASY

Sum

of S

quar

es D

egre

eDifficult to reason about

Tensor PCA (for 4-tensors)

! ∶ #$%&'(&)$#* +',$)-)./

0,',$)&'(-).

/(*1$,2

IMPO

SSIBLE

COM

PUTATIONALLY

EASY

Sum

of S

quar

es D

egre

eSum of squares:

• subsumes several well-known algorithmic techniques

• conjectured to be optimal in worst case settings.

• a sum of square lower bound almost always implies no known algorithm.

Signal strength vs Degree: Tensor PCA

IMP

OS

SIB

LE

! = #Θ %&'&

n%

De

gre

e o

f S

oS

pro

of

'Upper bound:[R-Rao-Schramm 16, Bhattiprolu-Guruswami-Lee 16]

Lower bound: [Hopkins-Kothari-Potechin-R-Schramm-S 17]

Tensor PCA in any dimensions,

Random CSPs

[R-Rao-Schramm 16, Kothari-Mori-O’Donnell-

Witmer]

Wish List

• Community Detection in stochastic block models.• Properties of random graphs: independent set, chromatic number,

vertex expansion …• Spiked Wigner models• Sparse PCA• Compressed Sensing• Mixtures models such as mixtures of Gaussians……………

Goal: Characterize optimal algorithms and thresholds for inference problems, and demonstrate the computational phase transitions predicted via statistical physics techniques.

SoS Degree Upper Boundfor TensorPCA

:[R-Rao-Schramm 16, Bhattiprolu-Guruswami-Lee 16]

TensorPCAgiven: a 4-tensor !: # $ → ℝ,

! = ( ⋅ *⊗$ +-!.,0,1,ℓ = *.*0*1*ℓ +-.,0,1,ℓ where

unknown signal * ∈ ℝ4, * = 1 and -.,0,1,ℓ ∼ 778 9:;<<7:#goal: recover * ∈ ℝ4

Natural approach to recover * approximately,

Maximize ! * = ∑.,0,1,ℓ !.,0,1,ℓ*.*0*1*ℓ

(Injective tensor norm)For a 4-tensor ! ∶ n $ → ℝ,

! .40 = maxC:‖C‖EF

G.,0,1,ℓ

!.,0,1,ℓ *.*0*1*ℓ = ⟨!, *⊗$⟩

Tensor Norm Certification

Certification Problem:given: a 4-tensor ! ∶ n $ → ℝ, !',),*,ℓ ∼ --. /0122-03goal: efficiently compute an upper bound 4(!) for ‖!‖'8) = max=:| = |@A⟨!, C

⊗$⟩

(Injective tensor norm)For a 4-tensor ! ∶ n $ → ℝ,

! '8) = max=:‖=‖@A F',),*,ℓ

!',),*,ℓ C'C)C*Cℓ = ⟨!, C⊗$⟩

Lies at the heart of random constraint satisfaction problems like random 3-SAT

By standard !-net argument," #$% ≤ ' (

with probability > 0.99for a random tensor ".

However, given a random tensor ", efficient algorithms are only known to certify upper bounds of O(n).

Typical TensorsGoal: upper bound for max

0 12⟨4, 6⊗8⟩

Spectral Upper Bound

Let ! be a symmetric "⊗$-dimensional tensor.

%&,& %&,(%(,&

%&,)

%),& %),)

⋯

⋯⋮

⋮⋱A =

Let / be the "(×"( matrix flattening of !.

Goal: upper bound for max4 5&

⟨!, 7⊗$⟩

Spectral Upper Bound

For any unit vector ! ∈ ℝ$,

&',' &',(

&(,'

&',$

&$,' &$,$

⋯

⋯⋮

⋮⋱A =

⟨/, !⊗1⟩ = ! ⊗ ! 34 ! ⊗ !

≤ max9∈ℝ:;

<34<<3<

= ‖4‖ = Θ ? w.h.p.

Recall that w.h.p., ‖/‖@AB ≤ C( ?).

∑/HIJℓ!H!I!J!ℓ

Goal: upper bound for maxL M'

⟨/, !⊗1⟩

Proof can be rewritten as degree 4 sum of squares

Random ?(×?(Gaussian matrix

Tensoring & Symmetrization

! "#$ = max) *+

,⊗., 0,⊗. 1+2

= max) *+

(,⊗.1)56⊗1 ,⊗.1+1

≤ 6⊗1+1

Symmetric under

permutations of dimensions {1, … , 2<}

= ‖6‖

Tensoring & Symmetrization

! "#$ = max) *+

,⊗., 0,⊗. 1+2

= max) *+

(,⊗.1)56 1 ,⊗.1+1

(avg of 7⊗1 over row & column symmetries)

≤ 6 1+1

M 1 = :;,< = ∘ 7⊗1 ∘ ?

Degree 4k SoS proof

Spectral Bound

! " #$ " ≤ &' ()

Theorem [R-Rao-Schramm, Bhattiprolu-Venkatesan]

!(+) -$-+-.-/, 1$1+1.1/= 1

4!+(6 -$-+1$1+ ⋅ 6 -.-/1.1/ + 6 -$-+1$1. ⋅ 6 -.-/1+1/ + … . . )

!(") = random (+"×(+" matrix whose entries have variance $"!

A Simple Spectral Algorithm

1. Compute a matrix ! " whose entries are polynomials of input:

! " ≔ $%,' (⊗"%,'

2. Compute the spectrum of !(") :

Output ! " ,- . ≥ 0 123

Runtime: 45 "eigenvalue of 4"×4" matrix

Theorem: [Hopkins-Kothari-Potechin-R-Schramm-S]For all problems satisfying a certain robust-inference property,

If SoS meta algorithm succeeds then a simpler spectral algorithmsucceeds.

SoS Degree Lower Bounds

Schema for a Lower boundPlanted Distribution

Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:

Fix degree 5. Suppose SoS algorithm recovers x, then

it solves a convex optimization problem <=``degree d SoS SDP relaxation”

whose solution yields %

Convex optimization problem >?“degree d SoS SDP relaxation”

is feasible & x is essentially only solution

Suppose degree d-SoS recovers x,

Schema for a Lower boundPlanted Distribution

Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:

Convex optimization problem ;<“degree d SoS SDP relaxation”

is feasible& x is essentially only solution

Suppose degree d-SoS recovers x,

Null Distribution (no signal)! = )

)/,0,1,ℓ ∼ 445 6789947:

Convex optimization problem ;<“degree d SoS SDP relaxation”

is infeasible

Contradict this!

Schema for a Lower boundPlanted Distribution

Given: ! = # ⋅ %⊗' +)unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:

Null Distribution (no signal)! = )

)/,0,1,ℓ ∼ 445 6789947:

Lower bound strategy:Exhibit a feasible solution to the convex optimization

problem ;< <=> < ?@? ?AB C=DEFEGH@I for inputs from the null distribution.

Constructions by hand, technical & difficult to generalize • Random CSPs [Grigoriev, Schoenebeck, Tulsiani, Barak-Kothari, Kothari-Mori-O’Donnell-Witmer]

• Planted Clique [Meka-Potechin-Wigderson, Hopkins-Kothari-Potechin-R-Schramm, Montanari-Deshpande]

Pseudocalibration [Barak-Hopkins-Kothari-Kelner-Moitra-Potechin]

Planted Distribution! = # ⋅ %⊗' +)

unknown signal % ∈ ℝ,, % = 1 and )/,0,1,ℓ ∼ 445 6789947:

Null Distribution (no signal)! = )

)/,0,1,ℓ ∼ 445 6789947:

-- a general candidate construction of SoS SDP solutions for null distributions

Signal % : a solution to ;< solutions for null distribution

=: ?@9ABCA5 DA:9?B ! → 94F:7G %

ΠI ∘ = = projection of F onto low-degree polynomials in T

ΠI ∘ =(!) -- candidate SoS SDP solution

Pros: Systematic and general, recovers several existing constructions in a unified manner.Cons: Construction is still conjectural in general, proof of feasibility is extremely technical and difficult to generalize.

A bold conjecture:For broad families of statistical estimation problems,

low-degree (degree D) SoS semidefinite program distinguishes between the two distributions

if and only if,low-degree polynomials (degree ! "#$ %) distinguish between them.

Pseudocalibration

Planted Distribution& = ( ⋅ *⊗, +.

unknown signal * ∈ ℝ1, * = 1 and .4,5,6,ℓ ∼ 99: ;<=>>9<%

Null Distribution (no signal)& = .

.4,5,6,ℓ ∼ 99: ;<=>>9<%

-- a general candidate construction of SoS SDP solutions for null distributions

A bold conjecture:For broad families of statistical estimation problems,

low-degree (degree D) SoS semidefinite program distinguishes between the two distributions

if and only if,low-degree polynomials (degree ! "#$ %) distinguish between them.

Pseudocalibration-- a general candidate construction of SoS SDP solutions for null distributions

• Yields precise predictions on the performance of SoS SDPs for statistical estimation problems

• Predictions coincide with conjectured thresholds of computational phase transition in community detection and related models [Hopkins-Steurer]

Conclusions

• Sum of squares algorithms open an avenue to develop a general theory of efficient statistical estimation.

• Pseudocalibration potentially demonstrate computational phase transitions formally?

! has entries corresponding to elements of " #!$%&'

cadb

( = *+,- !⊗/ +,-

!$%&'

!⊗/01

cadb

cadbcadbcadbcadb

2 3!⊗/ has entries of the form

For example,!⊗4 5678, 9:;ℎ = !$%,&' ⋅ !>?,@A

Visualizing

a

jib

w

tsx

c

ℓkd

y

vuz

ab

w

tsx

cd

y

vuz

ji

ℓk

ab

w

tsx

cd

y

vuz

ji

ℓkAvg + +⋯+'(,* ≔

,⊗. → '0 1

1

2 0 2

average over permutation of

hyperedgematching from 1

→ 2

1

2

1 2

M . = 56,7 0 ∘ ,⊗. ∘ 9

' :;:<:=:>, ?;?<?=?>

=14!

<(D :;:<?;?< ⋅ D :=:>?=?> + D :;:<?;?= ⋅ D :=:>?<?> + … . . )

! " = "⊗%, !

= "⊗%', !⊗%' (/'

= "⊗*', +⊗' "⊗*',- where

+ isa6*×6* matrixflatteningof !,

i.e.,+ AB, Cℓ = !EF'ℓ ∀A, B, C, ℓ ∈ [6]

"⊗*' ∈ ℝ L M- is invariant under permutation of the NOPQR, i.e., action of the permutations on 1, … , 2C

= "⊗*', V ∘ +⊗' ∘ X "⊗*',- for every pair of permutations V, X ∈ YZ

= "⊗*', [\,] V ∘ +⊗' ∘ X "⊗*',-

≤ [_,` V ∘ +⊗' ∘ X

a/Z

! " = "⊗%, !

= ⟨"⊗(, ) "⊗(⟩ where) isa3(×3( matrixflatteningof !,i.e.,) >?, @ℓ = !BCDℓ ∀>, ?, @, ℓ ∈ [3]

≤ " ( )

≤ ) for " ≤ 1

! BKC ≤ )

A Spectral Bound

! " = "⊗%, !

= "⊗%', !⊗%' (/'

= "⊗*', +⊗' "⊗*',- where

!*,* isa6*×6* matrixflatteningof !,i.e.,!*,* AB, Cℓ = !EF'ℓ ∀A, B, C, ℓ ∈ [6]

≤ "⊗*' *+⊗'

,-

≤ +⊗' (/' for " ≤ 1= +

SoS Degree Lower BoundsApproach: Given tensor !Construct a linear functional "# ∶ ℝ &', … , &*

+, → ℝ such that,PropertyI:(largeonpolynomialT)"# ! & > B ⋅ "# 1Property II (non-negative on squares)"#[F

G] ≥ 0 ∀ F ∈ ℝ &',… , &*+,

Property III (respects constraints on x)"#[ 1 − & G ⋅ NG & ] ≥ 0 ∀ N ∈ ℝ &',… , &*

+,

B − ! & = ∑ℓ FℓG & + ∑ℓ NℓG & (1 − & G)"# ∘ "# ∘ "# ∘

< 0(Property I)

≥ 0(Property II)

≥ 0(Property III)

SoS Degree Lower BoundsApproach: Given tensor !Construct a linear functional "# ∶ ℝ &', … , &* +, → ℝ such that,PropertyI:(largeonpolynomialT)"# ! & > B ⋅ "# 1Property II (non-negative on squares)"#[FG] ≥ 0 ∀ F ∈ ℝ &',… , &* +,

Property III (respects constraints on x)"#[ 1 − & G ⋅ NG & ] ≥ 0 ∀ N ∈ ℝ &',… , &* +,

If O& is a solution to system O& G ≤ 1 and ! O& ≥ BThen the functional given by evaluation at O&,

"[F & ] = F(O&)

"# F = R OS∼UV[F( O&)]But there are no actual solutions to the system { & G ≤ 1, ! & ≥ B}, we are looking for a pseudodistribution Y

Pseudocalibration

• a candidate construction of the linear functional !" for sum-of-squares lower bounds.

• Discovered in the context of proving lower bounds for the planted clique problem.

• Still conjectural, but yields a valid linear functional for various problems like planted clique, tensor PCA

• Yields predictions on precise thresholds at which sum-of-squares SDPs fail, for various statistical estimation problems like community detection. The predicted thresholds coincide with conjectures arising from statistical physics considerations.

Pseudocalibration

Null distribution: ! random Gaussian tensor

given: tensor T

goal: construct a linear functional

"#:ℝ &', … , &* +, → ℝsuch that:

"# ! & ≥ / ⋅ "# '"# 1 − & 3 ⋅ 43 & ≥ 0 ∀ 4

"# 73(&) ≥ 0 ∀7

Planted distribution: ! = &⊗< +> random Gaussian tensor

Real

Sum of squares proofsSuppose ! " = ∑%&'ℓ !%&'ℓ "%"&"'"ℓ

) − ! " = ∑ℓ+ℓ, " + ∑ℓ-ℓ, " (1 − " ,)

123 45 1-26781 123 45 1-26781 ⋅ 1 − " ,

degree of the proof = max. deg { +ℓ, " , -ℓ,(")(1 − " ,)}

For each = ∈ ℕ,)@ ! = inf{ ) | ) − ! " ≥ 0 ℎ61 6 =8H788 = I4I +7445}

lim@→M

)@ ! = ! %N&

Finding degree =- sum of squares proof takes OP(@) computational time.

! − # $

= ! $ & − # $ + ! (1 − $ &)

= $⊗,, ! ⋅ /0 − ! $⊗, + ! (1 − $ &)

For every positive semidefinite matrix 1, and indeterminates 22,12 = 345 67 3849:;3 <6=>(2)

= 345 67 3849:;3 <6=> + 345 67 3849:;3 1 − $ , ≥ 0

SoS captures the spectral bound