Lecture'1

Maths&for&Machine&LearningDahua%Lin

The$Chinese$University$of$Hong$Kong

1

Roadmap

Mathema'cs!lies!at!the!heart!of!machine!learning!research.!

Complete(coverage(of(all(these(subjects(is(obviously(beyond(the(scope(of(this(course.(This(lecture(aims(to(give(an(overview(of(several(useful(math(concepts.

2

Concepts)to)Be)Covered• Basics'of'topology

• metrics

• open'and'closed-sets

• con.nuous-func.on

• compact-space

3

Concepts)to)Be)Covered)(cont'd)• Basics'of'func,onal'analysis

• norm,'inner'product

• Banach'space,'Hilbert'space

• func5onal,'operator

• bilinear'form

4

Concepts)to)Be)Covered)(cont'd)• Basics'of'modern'probability'theory:'

• measure'space

• Lebesgue'integra0on

• random'variables

• expecta0on

• convergence'of'laws

5

Metrics

Measurement*of*distances*or*devia*on*lies*at*the*heart*of*many*learning*problems.*

6

Metrics((Defini-on)

• "is"called"a"metric,"if"it"sa-sfies:

• (non*nega-vity)"

• (coincidence2axiom)"

• (symmetry)"

• (triangle2inequality)"

7

Metrics((Proper-es)• The%triangle)inequality%can%be%further%generalized:

!

• A#set# #together#with#a#metric# #defined#thereon#is#called#a#metric'space,#denoted#by# .

8

Examples)of)Metrics• Euclidean*metric:

• Rec$linear*metric:

!

9

Metrics(Induced(by(Transform• One%can%define%a%metric%over%an%arbitrary%set%$S$%through%an%injec&ve(map% %to%a%metric%space% :

• It$can$be$easily$verified$that$ $is$a$metric$on$ .

10

Topology'Defined'by'Metrics• Topological)space"is"a"more"general"concept"than"metric)space

• With"metrics,"topological)concepts"can"be"defined"in"a"more"intui7ve"way

11

Interior• Open%ball"of"radius" :"

• "is"an"interior'point"of" "iff" ."

• The"interior"of" ,"denoted"by" "or" ,"is"the"set"of"all"interior"points"of" .

12

Boundary

• "is"a"boundary)point"of" ,"iff"for"any" ," "overlaps"with"both" "and" ."

• The"boundary"of" ,"denoted"by" ,"is"the"set"of"all"boundary)points"of" .

13

Open%and%Closed%Sets

• Consider*a*metric*space* ,*and*a*subset*:

• *is*called*an*open%set,*if*

• *is*called*a*closed%set,*if* *is*open

• *is*open*iff* .

• *is*closed*iff* .

14

Proper&es(of(Open(and(Closed(Sets• The%union%of%arbitrary%collec-ons%of%open%sets%is%open.

• The%intersec-on%of%finitely+many%open%sets%is%open.%

• The%intersec-on%of%arbitrary%collec-ons%of%closed%sets%is%closed.

• The%union%of%finitely+many%closed%sets%is%closed.

15

Ques%ons

• Consider*an*arbitrary*metric*space* ,*are* *and**open*or*closed?

• Let* *be*a*finite*metric*space*and* ,*is* *open*or*closed?

• Consider*Euclidean*space* *and* ,*is* *open*or*closed?*what*is* ?

16

Closure

Let$ $be$a$subset$of$a$metric$space$ :

• The%closure%of% ,%denoted%by% %or% ,%is%defined%to%be%the%intersec3on%of%all%closed%sets%that%contain%.

• %is%closed%if%and%only%if% .

• .%

• .

17

Closure((Examples)

• .

• .

18

Convergence

• Let% %be%a%sequence%in%a%metric%space%,%then% %is%called%a%limit%of%this%sequence%

if

• A#sequence#is#said#to#be#convergent#if#it#has#a#limit.

• The#limit#of#a#convergent#sequence#is#unique.

19

Cauchy'Sequence

• A#sequence# #is#called#a#Cauchy'sequence#if#

!

• A#convergent)sequence#must#be#a#Cauchy)sequence

• Ques%on:#Is#a#Cauchy)sequence#always#convergent?

20

Mo#va#ng(Example• Consider*a*real*valued*sequence*

• We$have$learned$in$Calculus$that$ $as$.

• Note$that$ $is$a$ra5onal$sequence.$Is$this$sequence$convergent$in$the$ra#onal'space$ ?

21

Complete(Metric(Space• Convergence)is)not)an)intrinsic)property)of)a)sequence,)which)also)depends)on)the)space)in)which)the)sequence)lies.

• A#metric#space# #is#called#a#complete)metric)space#if#every)Cauchy)sequence)converges.

• #is#complete,#but# #is#not.

• #can#be#constructed#by#comple:ng# .

• Ques%on:#Is#the#integer#space# #complete?#22

Closedness(and(Completeness

Let$ $be$a$subset$of$a$complete$metric$space$ :

• Being'closed'means'"containing-all-the-limits":

• Let' 'be'a'sequence'in' 'and' ,'then'

• 'is'closed'if'and'only'if' 'is'complete,'meaning'all-convergent-sequences-in- -converge-within- .'

23

Bounded

• The%diameter%of% ,%denoted%by% ,%is%defined%to%be%the%largest%distance%between%points%in% ,%as

• "is"said"to"be"bounded"if" .

24

Compactness• In$elementary$Calculus,$we$learned$that$"every&bounded&sequence&in& &has&a&convergent&subsequence".$

• Does$this$hold$in$general$metric$spaces?

25

Compactness+(cont'd)

Consider)a)metric)space) :

• "is"called"compact"iff"either"of"the"following"holds:

• Any"open"cover"of" "has"a"finite"subcover.

• "is"complete"and"totally*bounded,"namely,"for"every" ,"there"is"a"finite"cover"of" "by" >balls.

• "is"sequen1ally*compact,"namely,"every"sequence"in" "has"a"convergent"subsequence.

26

Compactness+(cont'd)• Compact)sets)are)always)closed'and'bounded.

• But,)the)converse)is)generally'false.

• )is)compact)if'and'only'if) )is)bounded)and)closed.

27

Con$nuous'Func$on

• "is"called"con$nuous,"iff"either"of"the"following"holds:"

• (Topological)" "is"open"whenever" "is"open.

• (Weierstrass)"Given"any" "and" ,"

• (Sequen$al3con$nuity)" "whenever".

28

Con$nui$es)of)Various)Levels

• "is"called"uniformly*con,nuous,"if"

• "is"called"Lipschitz)con,nuous,"if"

!

• Lipschitz)con,nuous" "Uniform)con,nuous" "Con,nuous."

29

Con$nuous'Funcs'on'Compact'Set

• Let% %be%a%con+nuous%func+on,%then%%is%compact%whenever% %is%

compact.

• Let% ,%then% %assumes%maximum%and%minimum%within% %when% %is%compact.

• A%con)nuous+func)on%is%uniformly+con)nuous%over%a%compact%domain.%A%con)nuously+differen)able+func)on%is%Lipschitz+con)nuous%over%a%compact%domain.

30

Dense%Sets

• A#subset# #is#said#to#be#dense%in% ,#if# .

• A#subset# #is#dense%in% ,#iff#either#of#the#following#holds:

• Each#open#subset#of# #contains#at#least#one#element#in# .

• Each# #is#a#limit#of#some#sequence#in# .

• #is#dense#in# ,# #is#dense#in# .

31

Separable(Space

• "is"called"a"separable(space,"if" "contains"a"countable"dense"subset.

• Let" "be"a"con3nuous"func3on"on" ,"and" "is"dense,"then" "is"uniquely"determined"by" .

• Ques%on:"Let" "be"con3nuous,"and"it"has""and" ."

• Is" "determined"by"these"condi3ons?"If"so,"what"is" ?"

32

Things'become'much'more'interes0ng'when'vector'spaces'meet'

with'metrics.

33

Norms

• Consider*a*vector'space* ,*a*func0on* *is*called*a*norm*if:

•

• *iff* *

• .

• .

34

Banach&Spaces

• A#vector#space#together#with#a#norm,#as# ,#is#called#a#normed'space.#

• A#normed'space#is#always#a#metric'space,#where#the#norm#induces#a#metric#as# .

• A#complete'normed'space#is#called#a#Banach'space.

35

Proper&es(of(Norms

• The%norm%func*on% %is%Lipschitz%con*nuous:

• The%metric% %induced%by%the%norm%has:

• transla'on)invariance:%

•

36

!Norm

Consider)a)real)vector)space) .)For)each) ,)we)can)define) 6norm)as:

It#can#be#easily#verified#that# #is#a#norm.#

37

!Norm

When% ,% 'norm%is%s-ll%a%norm,%defined%as

We#will#extend#these#norms#to#the#space&of&func+ons.

38

Convergence)of)Vector)Sequences

• Let% %be%a%sequence%of%vectors%in%a%Banach%space,%we%say% %converges%to% %if% %as% .

• This%is%some9mes%called%convergence+in+norm.

39

Convergence)of)Series

• Consider*an*infinite*series* *

• Let* .*The*series* *is*said*to*be*convergent*if*the*sequence* *converges.

• *is*said*to*be*absolutely1convergent*if*the*series**converges.

• Absolute1convergence*implies*convergence1(in1norm).

40

Basis• In$elementary$linear$algebra,$we$learned$that$a$basis$of$a$vector$space$is$a$linearly*independent$set$of$vectors$that$spans$the$space.

• If$a$vector$space$has$a$finite$basis,$it$is$called$finite/dimensional.$

• The$cardinali<es$of$all$bases$of$a$finite/dimensional$space$are$the$same,$called$the$dimension$of$the$space.

41

Basis%of%Banach%Space

• Let% %be%a%sequence%in%a%Banach%space% ,%it%is%called%a%(Schauder)+basis%of% %if%for%each% ,%there%exists%a%unique%sequence%of%real%values% %such%that%

!

• A#Banach#space# #with#a#Schauder#basis#must#be#separable.

• Is#the#converse#true?#42

Basis%of%Banach%Space%(cont'd)• It$took$about$forty$years$to$get$the$answer$33$No.$

• Enflo$constructed$a$separable$Banach$space$with$no$Schauder$basis$in$1973.

43

Inner%Products

• An$inner$product$on$a$real$vector$space$ $is$a$mapping$ $that$sa5sfies:

•

• $iff$

• (symmetry)$

• (bilinearity)$

44

Hilbert(Spaces• A#vector#space#together#with#an#inner#product#is#called#an#inner%product%space.

• An#inner%product%space#is#always#a#normed%space,#where#the#norm#is#induced#by#the#inner%product,#as#

.

• A#complete%inner%product%space#is#called#a#Hilbert%space.

• A#Hilbert%space#is#always#a#Banach%space.45

Euclidean*Space

• "is"a"Hilbert(space"with"the"inner"product"defined"

by" ,"which"induces"the" 5norm"

and"Euclidean(metric.

46

Proper&es(of(Inner(Products• (Parallelogram*equality)"

"##"this"only"holds"for"norms"induced"by"inner"products.

• (Cauchy–Schwarz*inequality)" ,"or"equivalently," .

• (Con9nuity)"if" "and" ,"then".

47

Orthogonality

• "are"said"to"be"orthogonal"to"each"other,"denoted"by" ,"if" .

• A"subset" "is"called"an"orthogonal)set"if"elements"of" "are"mutually"orthogonal."Moreover,"if"each"element"has"a"unit"norm,"then" "is"called"an"orthonormal)set.

• (Pythagorean)theorem)".

48

Projec'on

• Let% %and% ,%then%the%distance%between% %and% %is%defined%to%be% .

• Let% %be%a%non2empty%convex%closed%subset%of% .%Then%there%exists%a%unique%element% %such%that%

.%This%element% %is%called%the%projec/on%of% %onto% ,%denoted%by% .

• Ques%on:%Why%does% %need%to%be%closed%and%convex?

49

Projec'on)(cont'd)

Let$ $be$a$Hilbert$space$and$ $be$a$closed$subspace:

• Given' 'and' ,'then' ,'meaning'that' .

Let$ $be$a$projec,on,$where$ $is$non3empty:

•

• "is"idempotent,"namely," ,"i.e.".

50

Orthogonal*Complement

• A#vector#space# #is#called#the#direct'sum#of#two#subspaces# #and# ,#denoted#by# ,#if#each#

#can#be#expressed#uniquely#as# #with##and# .#

• The#orthogonal'complement#of# ,#denoted#by#,#is#defined#to#be# .#

• #is#a#closed#subspace#of# .

• ,# .51

Orthonormal*Basis*of*Hilbert*Space

• Let% %be%a%Hilbert%space,%a%subset% %is%called%a%total%subset%of% %if% %is%dense.

• %is%a%total%subset%of% %if%and%only%if%.

• Ques%on:%Why%do%we%only%require% %to%be%dense%instead%of% ?

• A%total%orthonormal%set%is%called%an%orthonormal%basis.

52

Orthonormal*Basis*of*Hilbert*Space

• (Parseval)theorem)"An"orthonormal"sequence" "is"total"if"and"only"if

• In$this$case,$each$ $can$be$uniquely$expressed$as$

53

Orthonormal*Basis*of*Hilbert*Space*(cont'd)

• Every'Hilbert'space'has'a'total%orthonormal%set.'

• Every'separable'Hilbert'space'has'a'total%orthonormal%sequence.

54

Spaces

55

Linear'Operators'and'Func1onals

• Let% %and% %be%two%linear%spaces,%a%func5on:%%is%called%a%linear'operator%if%it%preserves%

linear'dependency,%that%is,%

• In$par(cular,$when$ $is$ ,$ $is$called$a$linear'func+onal.

• The$set$ $is$a$subspace$of$,$called$the$null'space$of$ .

56

Bounded'Operators

Let$ $be$a$linear$operator$on$Banach$spaces:

• "is"said"to"be"bounded,"if".

• The"(operator)-norm"of" "is"defined"as

!

• "always"holds.57

Bounded'Operators'(cont'd)

• Given'a'linear'operator' ,'the'following'statements'are'equivalent:

• 'is'bounded

• 'is'con;nuous

• 'is'finite'

• All'linear'operators'with'finite'dimensional'domain'are'bounded.

58

Space&of&Bounded&Operators

• All$bounded$operators$ $form$a$normed)space,$denoted$by$ .$

• When$ $is$a$Banach$space,$ $is$a$Banach$space.

59

Dual%Spaces

• All$bounded$func)onals$ $form$a$normed)space,$called$the$dual)space$of$ ,$denoted$by$ ,$where$the$norm$is$called$the$dual)norm,$defined$as:

• "is"always"a"Banach"space,"no"ma2er"whether" "is"or"not.

• "is"generally"not"isomorphic"to"the" .60

Func%on'Space'and'Uniform'Norm• The%set%of%all%con$nuous%real%valued%func2ons%defined%on%a%compact+space% %forms%a%normed%vector%space,%denoted%by% ,%where%the%norm%is%defined%by% ,%which%is%called%the%

uniform+norm%(or%Chebyshev+norm).

• When% %is%compact,% %is%a%Banach+space.

• There%are%other%ways%to%define%norms%of%func2ons,%which%we%will%discuss%later.

61

Ques%ons

• Let% %be%a%subspace%of% %that%comprises%all%con$nuously)differen$able)func$ons%on%

.%Let% .%We%define%a%func8onal%as%%which%takes%the%deriva8ve%at% .%

• Is% %a%linear%func8onal?

• Is% %bounded?

62

Representa)on+of+Func)onals• Every'Hilbert'space'is'isomorphic'to'its'dual%space.

• (Riesz's%theorem)'Every'bounded'linear'func8onal' 'on'a'Hilbert'space' 'can'be'represented'in'terms'of'inner'product,'namely,' ,'where' 'is'uniquely'determined'by' 'and'has' .'

• Ques%on:'How'can'you'find' 'given' ?

63

Representa)on+of+Func)onals+(cont'd)

• In$a$separable$Hilbert$space,$ $can$be$expressed$as$a$series:

64

Measure'Theory• Measure'theory"studies"assigning'values'to'subsets.

• Measure"theory"is"the"corner"stone"of"many"math"subjects

• Modern"approach"to"integra8on"is"based"on"measure"theory.

• Modern"probability"theory"is"based"en8rely"on"measure"theory.

65

Lengths(of(Real(Subsets• Intui'vely,-the-measure-of-a-set-can-be-intepreted-as-the-size,-e.g.-the-length-of-an-interval.

• How-long-are-these-subsets:- ,- ,-and- ?

• What-is-the-length-of- ?

• Now-let's-try-to-compute-the-length-of- -by-decomposing0it0into0infinitely0many0points:

66

Lengths(of(Real(Subsets((cont'd)• Sizes'(e.g.'lengths,'areas,'and'volumes)'are'countably-addi0ve.

• Let' 'be'a'collec;on'of'subsets'of' ,'then'a'func;on' 'is'said'to'be'countably-addi0ve,'if'for'any'finite'or'countable'sequence'of'disjoint'subsets' ,'it'has

67

The$Vitali$Set

• Let's'consider'a'very'interes1ng'subset'of' ,'called'Vitali&set:

• We'say' 'and' 'are'equivalent,'if' .

• 'can'then'be'par11oned'into'mul1ple'equivalent'classes.

• We'have'incountably&infinite'such'classes

• By'axiom&of&choices,'we'can'form'a'set' ,'which'contains'one'representa1ve'from'each'class.

68

The$length$of$Vitali$Set

• Enumerate*all*ra,onal*numbers*within* *as*,*and*let* .*Obviously,**are*disjoint

• Let* ,*then* ,*and*

therefore* .

• However,*if* ,*then* ,*or*if*,*then* *...

69

Ring%of%Sets

• ,#a#nonempty#collec.on#of#subsets#of# ,#is#called#a#ring,#if#it#is#closed#under#union#and#set*difference.

• A#ring#must#contain# .

• A#ring#is#closed#under#intersec.on.

• Overall,#we#have:

70

!algebra

• A#ring# #is#called#a#algebra,#if#it#is#also#closed#under#complement.

• An#algebra#must#contain# .

• An#algebra#is#called#a# 5algebra,#if#it#is#also#closed#under#countable.union.

• A# 5algebra#is#closed#under#countable#fold#of#any#elementary#set#opera9ons.

71

Measure'and'Measure'Space

• A#countably#addi/ve#func/on# #from#a# 5algebra# #into# #is#called#a#measure.#

• The#triple# #is#called#a#measure'space.#All#the#elements#of# #are#called#measurable'sets.

72

Proper&es(of(Measures

Consider)a)measure'space) :

• .

• .

• Let& &be&disjoint,&then&.

73

Con$nuity)of)Measures

.

.

74

Finite&and& )finite&Measures

• "is"called"a"finite&measure"if" .

• "is"called"a" ,finite&measure"if" "is"covered"by"countably"many"subsets"of"finite"measure.

75

Lengths(of(Open(Intervals• The%length%of%an%open%interval%is%defined%as%

.

• The%the%length%of%a%finite%union%of%disjoint%open%intervals%is%defined%to%be%the%sum%of%the%lengths%of%individual%intervals.

76

Lengths(of(Open(Intervals((cont'd)• The%collec)on%of%all%finite%unions%of%disjoint%open%intervals%cons)tutes%a%ring.

• The%length%is%well%defined%over%this%ring,%and%it%sa)sfies%finite%addi)vity.%

• Hence,%it%is%called%a%pre'measure.

• We%want%to%extend%the%length%to%as%many%subsets%as%possible.

77

Generated( )algebra

• Let% %be%a%collec+on%of%subsets.%Then%the%smallest%4algebra%that%contains% %is%called%the% !algebra(generated(by( ,%denoted%by%

• ,%

• countable%intersec+on%of%unions%of%sets%in% %are%in% .

78

Borel& 'algebra

• Let% %be%a%metric'space.%The% +algebra%generated%by%the%open%subsets%of% %is%called%the%Borel' .algebra%of% ,%denoted%by%

• Elements%of%the%Borel% +algebra%are%called%Borel'sets.

• All%the%open%sets,%closed%sets,%and%their%finite%and%countable%unions%and%intersec?ons%are%all%Borel'sets.

• Finite%or%countable%sets%are%all%Borel'sets.79

Measure'Extension

• (Hahn–Kolmogorov.theorem):"Let" "be"a"ring"that"covers" ," "be"a"pre4measure,"then" "can"be"extended"to"a"measure" ."If""is" 9finite,"then"the"extension"is"unique.

• The"length"func>on"over"the"open"interval"ring"can"be"uniquely.extended"to"a"measure"over"the"Borel" 9algebra"over" ,"called"Borel.measure.

80

Complete(Measure(Space

We#are#not#done#yet.#Given#a#measure#space#:

• A#subset# #is#called#a#null$set#if#it#is#contained#in#a#measurable#set#of#zero#measure.

• This#measure$space#is#called#a#complete$measure$space#if#every#null#set#is#measurable.

• Let# #denote#the#collec:on#of#all#the#null$sets.#Then# #is#complete.#

81

Lebesgue'Measure• The%Borel&measure&space%is%generally%not%complete.%

• The%comple.on%of%the%Borel&measure&space%is%called%the%Lebesgue&measure&space,%and%the%extended%measure%is%called%Lebesgue&measure.

82

Lebesgue'Integral'.'Indicator'Func4ons

To#derive#the#Lebesgue'integral,#we#begin#with#simple#func7ons#and#then#extend#the#defini7on#to#more#general#func7ons.#Let# #be#a#measure#space:

• Indicator+func.ons+00+the+integral+of+an+indicator+func.on+ +of+a+measurable+set+ +is+defined+to+be+the+measure+of+ ,+as

83

Lebesgue'Integral'.'Simple'Func5ons• Simple(func-ons(00(linear(combina-ons(of(indicator(func-ons:

84

Lebesgue'Integral'.'Nonnega1ve'Func1ons

• Let% %be%a%non#nega've%func,on%on% ,%then%we%define

85

Lebesgue'Integral'.'Signed'Func4ons

• Signed(func,on:(decompose( (as( ,(with( (and( ,(then

• When&both& &and& &are&infinite,&

&is&undefined.

86

Proper&es(of(Lebesgue(Integral

• "is"a"linear"func-onal.

• If" "then" "

• a"predicate"holds"almost'everywhere"means"that"it"holds"over" ,"except"for"a"null"set.

• (Monotonicity)" .

87

Monotone&Convergence&Theorem

Let$ $be$a$sequence$of$non#nega've$func.ons,$and$$pointwisely,$then$

88

Dominated*Convergence*Theorem

If# #pointwisely,# #is#dominated#by# ,#i.e.#

,#and# ,#then#

89

Integral)with)Coun0ng)Measure• The%coun%ng'measure%over%a%finite%or%countable%space% %is%defined%as%the%cardinality%of%subsets.%

• Let% %be%a%coun%ng'measure%over%a%countable%space%,%then%

!

90

Integral)with)Lebesgue)Measure

• Let% %be%a%func,on%over% ,%then%when% %is%Riemann'integrable,% %must%be%Lebesgue'integrable%(w.r.t.%the%Lebesgue'measure),%and%in%such%a%case,%the%Lebesgue'integral%is%equal%to%the%Riemann'integral.%

• Ques%on:%Consider% :

• Is%it%Riemann'integrable?

• Compute%the%Lebesgue'integral%of% %with%respect%to%the%Lebesgue'measure.

91

!space

• For%any% ,%the% %space%of% ,%denoted%by%,%is%the%vector%space%comprised%of%all%

measurable%func8ons%on% %such%that%

,%where%the% )norm%is%defined%by

• Ques%on:"Is"it"actually"a"norm?92

!space!(cont'd)

• Func&ons)that)are)equal)almost'everywhere)are)indis&nguishable)by)integra&on.)Let) )denotes)the)vector)space)with)all)essen/ally'equivalent)func&ons)merged)into)an)element.)

• Then) )is)a)Banach'space)with)the) >norm.

• The) )norm)is)defined)as)the)essen/al'supreme)of),)as

93

!Space!and!Dual

• The%dual%space%of% %is%isomorphic%to% %with%.%

• For%each%bounded%func8onal%on% ,%there%exists% :%

94

!Hilbert!Space

• "is"a"Hilbert"space,"where"the"inner"product"is"defined"as

• The%dual%space%of% %is% .

95

Absolute)Con,nuity

Let$ $and$ $be$two$measures$over$ :

• "is"said"to"be"absolutely*con-nuous"with"respect"to","denoted"by" ,"if" "whenever"

.

• "and" "are"said"to"be"singular,"denoted"by" ,"if"there"exists" "such"that" "and"

.

96

Lebesgue'Decomposi.on

Let$ $and$ $be$two$measures$over$ :

There%exists%a%unique%decomposi3on%of% %as%

!

such%that% %and% .

97

Radon&Nikodym,Theorem

Let$ $be$a$finite$measure$that$is$absolutely*con-nuous$with$respect$to$ .$Then$there$exists$an$essen-ally$unique$ :$

Here,% %is%called%the%Radon&Nikodym,density%of% %with%respect%to% .%In%this%case,%we%also%have

98

Event&Spaces

• The% &algebra% %can%be%interpreted%as%an%event%space.%Each%element% %is%an%event.

• % %both% %and% %happen

• % %either% %or% %happens

• % % %does%not%happen

• % % %and% %are%mutually%exclusive.

99

Probability*Measure

• A#probability*measure# #is#a#measure#on# #with#.

• #can#be#considered#as#the#probability#of#the#event# .

• Obviously,# #sa:sfies#all#the#requirement#of#probability#func:ons#in#classical#probability#theory.

100

Independence

• Two%events% %and% %are%said%to%be%independent%if%

• We$say$two$collec-ons$of$events$ $and$ $are$independent$when$

101

Random'Variables

• A#(real&valued)+random+variable# #is#defined#to#be#a#measurable+func4on# .

• Each#random#variable# #induces#a#sub7 7algebra,#denoted#by# ,#where# #is#the#Borel# 7algebra#of# .

• Two#random#variables# #and# #are#said#to#be#independent#if# #and# #are#independent.

102

Expecta(on

• The%expecta'on%of%a%random%variable% %is%defined%to%be

!

• The%expecta'on%is%a%bounded-linear-func'onal.

103

Probability*and*Law

• The%probability%that%the%value%of% %falling%in% %is%then%given%by%

."

• The%func*on% ,%which%maps%each%Borel%set%%to%a%probability*value,%is%itself%a%probability*

measure%over% ,%which%is%called%the%law%of% ,%denoted%by% .

104

Probability*Density

• The%Radon+Nikodym%density%of% %with%respect%to%the%Lebesgue%measure%over% ,%denoted%by% ,%is%called%the%probability*density%of% .%So%we%have%

.

105

(Seman'c)*Correspondence• "algebra) )event)space

• sub" "algebra) )a)family)of)events

• measure) ) )probability)func6on

• measurable)func6on) ) )random)variable

• ) )law)of)

• integra6on) )expecta6on

• Radon"Nikodym)density) )probability)density106

Convex'Analysis• Convex(op*miza*on(plays(a(crucial(role(in(many(machine(learning(problems

• Prof.(Stephen'Boyd(has(a(very(famous(book("Convex'Op1miza1on",(which(provides(an(excellent(treatment(of(this(subject

• The(PDF(version(is(available(for(download(in(Prof.(Boyd's(website

• You(should(at(least(read(the(first(five(chapters

107

Convex'Analysis'(cont'd)• Important*concepts

• convex'set

• convex'func,on

• conjugate'func,on

• Lagrange'dual

• strong'duality'and'weak'duality

• We*will*begin*to*use*these*concepts*in*Lecture*3.108