Learning and Fourier Analysis Grigory Yaroslavtsev CIS 625: Computational Learning Theory.
-
Upload
rudolf-strickland -
Category
Documents
-
view
216 -
download
1
Transcript of Learning and Fourier Analysis Grigory Yaroslavtsev CIS 625: Computational Learning Theory.
Learning and Fourier Analysis
Grigory Yaroslavtsevhttp://grigory.us
CIS 625: Computational Learning Theory
Fourier Analysis and Learning
• Powerful tool for PAC-style learning under uniform distribution over
• Sometimes requires queries of the form • Works for learning many classes of functions, e.g:– Monotone, DNF, decision trees, low-degree polynomials – Small circuits, halfspaces, k-linear, juntas (depend on small
# of variables)– Submodular functions (analog of convex/concave)– …
• Can be extended to product distributions over , i.e. where means that draws are independent
Boolean Functions
• Book: “Analysis of Boolean Functions”, Ryan O’Donnellhttp://analysisofbooleanfunctions.org• Notation switch:
• Example:
–
Fourier Expansion
• Example:
• For let character
• Thm: Every function can be uniquely represented as a multilinear polynomial
• Fourier coefficient of on
Orthonormal Basis: Proof
• Thm: Characters form an orthonormal basis in
• Since we have:– = 0 if – = 1 if
• This proves that are an orthonormal basis
Functions = Vectors, Inner Product
• Functions as vectors form a vector space:
• Inner product on functions = “correlation”:
where
Fourier Coefficients
• Recall linearity of dot products:
• Lemma:
Parsevel’s Theorem
• Parseval’s Thm: For any
If then • Example:
Plancharel’s Theorem
• Plancharel’s Thm: For any
• Proof:
Basic Fourier Analysis
• Mean – For we have
• Variance
(Parseval’s Thm)
Convolution
• Def.: For their convolution :
• Properties:1. 2. 3. For all
Convolution: Proof of Property 3
• Property 3: For all • Proof:
(def. convolution)
Approximate Linearity
• Def: is linear if– for all – such that for all
• Def: is approximately linear if1. for most 2. such that for most
• Q: Does 1. imply 2.?
Property Testing [Goldreich, Goldwasser, Ron; Rubinfeld, Sudan]
NO
YES
Randomized Algorithm
Accept with probability
Reject with probability
⇒
⇒⇒
YES
NO
Property Tester
-close
Accept with probability
Reject with probability
⇒
⇒Don’t care
-close : fraction has to be changed to become YES
• = class of linear functions
• -close:
Linearity Testing
⇒
Linear
Non-linear
Linearity Tester
-close
Accept with probability
Reject with probability
⇒
⇒Don’t care
Linearity Testing [Blum, Luby, Rubinfeld]
• BLR Test (given access to ):– Choose and independently– Accept if
• Thm: If BLR Test accepts with prob. then is -close to being linear
• Proof: – Apply notation switch – BLR accepts if
• if • if
Linearity Testing: Analysis
(def of convolution) (Plancharel’s thm)
Linearity Testing: Analysis Continued
• Recall • :
Local Correction• Learning linear functions takes queries• Lem: If is -close to linear function then for every
one can compute w.p. as:– Pick – Output
• Proof: By union bound:
Then
Thanks!