8/18/2019 Examples Done on Orthogonal Projection
1/79
1
Projection and Some of Its Applications
Mohammed Nasser
Professor, Dept. of Statistics, RU,BangladeshEmail: [email protected]
1
The use of matrix theory is now widespread .- - - -- areessential in ----------modern treatment of univeriate andmultivariate statistical methods. ----------C. . ao
8/18/2019 Examples Done on Orthogonal Projection
2/79
!
"#li$ue and "rtho%onal Projection in !
"rtho%onal Projection into a &ine n
Inner product SpaceProjection into a Su#space'ram-Schmidt "rtho%onali(ationProjection and )atricesProjection in Infinite-dimensional SpaceProjection in )ultivariate )ethods
Contents
!
8/18/2019 Examples Done on Orthogonal Projection
3/79
*
M a t h em
a t i c al
C on
c e p t s
8/18/2019 Examples Done on Orthogonal Projection
4/79
+
Mathematical Concepts
Covariance
VarianceProjection
8/18/2019 Examples Done on Orthogonal Projection
5/79
,
1
0 and
1
1
1."#li$ue and "rtho%onalProjection in !
are two independent vectors in!
1 2
1 0{ | } and { | }
1 1V l l R V m m R⇒ ∈ ∈
are two one-dimensional su#spaces in !
! 1 / !1 0 ! 23
! 1 !
8/18/2019 Examples Done on Orthogonal Projection
6/79
4
1."#li$ue and "rtho%onalProjection in !
51617 1
52617
!
8/18/2019 Examples Done on Orthogonal Projection
7/79
8
1
0 and
1
1
1."#li$ue and "rtho%onalProjection in !
are two independent vectors in!
=⇔=⇔+=⇒−
2
1
2
11
2
1
2
121
2
1
1 1
0 1
1 1
0 1
1
0
1
1
a
a
x
x
a
a
x
xaa
x
x
−=⇒ 121
2
1
x x x
aa
−+=⇒1
0)(
1
1121
2
1 x x x x
x
8/18/2019 Examples Done on Orthogonal Projection
8/79
9
1."#li$ue and "rtho%onalProjection in !
:e define;&; ! as → 11
t =1
11
2
1 x
x
x L
:e can easily show that it is a linear mapThis linear map is called projection 5o#li$ue
8/18/2019 Examples Done on Orthogonal Projection
9/79
=
1."#li$ue and "rtho%onalProjection in !
51617 1
!
5-1617
8/18/2019 Examples Done on Orthogonal Projection
10/79
12
1."#li$ue and "rtho%onalProjection in !
1
1- and
1
1 are two ortho%onal5>independent7 vectors in !
&et us consider −
+= 11
1
121
2
1
aa x
x
In this case we can find values of ?a@ withoutinverse
[ ] [ ]
[ ]
[ ]2
2
1
1
1
2
1
111 1
1 1
111 11 1
v
v x x
x
a
a x x
T
==
=
8/18/2019 Examples Done on Orthogonal Projection
11/79
11
1."#li$ue and "rtho%onalProjection in !
:e define;&; ! as → 11
t = 11
1
1
1
1
2
2
1
T
x
x
x L
This linear map is called ortho%onalprojection.
The vector is projected on the space %enerated #yalon% the space %enerated #y 1
1
−1
1
8/18/2019 Examples Done on Orthogonal Projection
12/79
1!
Projections
=
2
2
2
000
010
001
0
2
2
5262617
5261627
5162627
5!6!6!7
a 51627
# 5!6!7
==0
2a
aaba
c T T
8/18/2019 Examples Done on Orthogonal Projection
13/79
1*
1."rtho%onal Projection Into a &ine
Definition 1.1 : Orthogonal Projection
The orthogonal projection of v into the line spanned by a non ero s is the!ector.
( ) ( )"" proj = ×s v v s s
×= ×v s
ss s
" = =×
s ss
s s s
#$a%ple 1.1 : Orthogonal projection of the !ector ( 2 & ) T into the line y '2 x.
( )2 11 1
2 & 2
proj = + ÷ ÷
s
2 x x = ÷
s 2 2* x x x= + =s 11"2 = ÷
s
1+2 = ÷ 1*
If S has unit len%th6 then 5v.s7s and its len%th is 5v.s7
( ) ( )"" proj = ×s v v s s
8/18/2019 Examples Done on Orthogonal Projection
14/79
1+
#$a%ple 1.2 : Orthogonal projection of a general !ector in , & into the y-a$is
2
0
10
÷
= ÷ ÷ e 2
0 0
1 10 0
x x
proj y y z z
÷ ÷ ÷ ÷
= × ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ e
0
10
y
÷
= ÷ ÷
0
0 y
÷
= ÷ ÷
#$a%ple 1.& : Project ' Discard orthogonal co%ponents railroad car left on an east- est trac/ itho t its bra/e is p shed by a ind
blo ing to ard the northeast at fifteen %iles per ho r hat speed ill the car reach
1 proj= ev w 11 02 = ÷
1112
= ÷ w
12
speed = =v
1+
8/18/2019 Examples Done on Orthogonal Projection
15/79
1,
#$a%ple 1.* : 3earest Point
1,
5a6#7 A
C
5c6d7
&et A 5a6#7 and 5c6d7 #etwo vectors. :e have tofind the nearest vector to Aon
That means we have to find the value of B for which5B7 55a6#7- B5c6d77T55a6#7- B5c6d77 is minimum 6 i.e. 6 len%th
of AC is minimum.
D5c6d7
Easy application of derivative shows that
5A. 7F . 3
8/18/2019 Examples Done on Orthogonal Projection
16/79
14
Exercises 1
1. Consider the function mappin% a plane to itself thattaBes a vector to its
projection into the line y x .5a7Produce a matrix that descri#es the functionGs action.5#7Show also that this map can #e o#tained #y first
rotatin% everythin% in the plane HF+ radians clocBwise6then projectin% into the x -axis6 and then rotatin% HF+radians counterclocBwise.
!. Show that
14
0))().(( =− v projvv proj s s
8/18/2019 Examples Done on Orthogonal Projection
17/79
18
2.1Definition
n inner product on a real spaces V is a f nction that associatesa n %ber4 denoted u4v 4 ith each pair of !ectors u and v of V . This f nction has to satisfy the follo ing conditions for
!ectors u4v4 and w4 and scalar c.
1. u4v ' v4u (sy%%etry a$io%)2. u + v 4w ' u4w 5 v4w (additi!e a$io%)&. cu4v ' c u4v (ho%ogeneity a$io%)*. u4u ≥ 04 and u4u ' 0 if and only if u ' 0
(position definite a$io%)
2.Inner product pace
8/18/2019 Examples Done on Orthogonal Projection
18/79
19
vector space on which an inner product is defined iscalled an inner product space .Any function on a vectorspace that satisfies the axioms of an inner product defines
an inner product on the space. .
There can #e many inner products on a %iven vectorspace
2.Inner product pace
8/18/2019 Examples Done on Orthogonal Projection
19/79
1=
Example !.16et u ' ( x14 x2)4v ' ( y14 y2)4 and w ' ( z 14 z 2) be arbitrary !ectors in
R 2
. Pro!e that u4v 4 defined as follo s4 is an inner prod cton R 2.
u4v ' x1 y1 5 * x2 y2Deter%ine the inner prod ct of the !ectors ( −24 )4 (&4 1) nder
this inner prod ct.olution
$io% 1: u4v ' x1 y1 5 * x2 y2 ' y1 x1 5 * y2 x2 ' v4u
$io% 2: u + v 4w ' ( x14 x2) 5 ( y14 y2) 4 ( z 14 z 2)
' ( x1 5 y14 x2 5 y2)4 ( z 14 z 2)' ( x1 5 y1) z 1 5 *( x2 5 y2) z 2' x1 z 1 5 * x2 z 2 5 y1 z 1 5 * y2 z 2
' ( x14 x2)4 ( z 14 z 2) 5 ( y14 y2)4 ( z 14 z 2)'
8/18/2019 Examples Done on Orthogonal Projection
20/79
!2
$io% &: cu4v ' c( x14 x2)4 ( y14 y2)
' (cx14cx2)4 ( y14 y2) ' cx1 y1 5 * cx2 y2 ' c( x1 y1 5 * x2 y2)
' c u4v $io% *: u4u ' ( x14 x2)4 ( x14 x2) ' 0*
22
21 ≥+ x x
7 rther4 if and only if x1 ' 0 and x2 ' 0. That is u '0. Th s u4u ≥ 04 and u4u ' 0 if and only if u ' 0.The fo r inner prod ct a$io%s are satisfied4
u4v ' x1 y1 5 * x2 y2 is an inner prod ct on R 2.
0*22
21 =+ x x
The inner prod ct of the !ectors ( −24 )4 (&4 1) is(−24 )4 (&4 1) ' ( −2 × &) 5 *( × 1) ' 1*
8/18/2019 Examples Done on Orthogonal Projection
21/79
!1
Example !.!8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v defined as follo s be arbitrary 2 × 2 %atrices.
Pro!e that the follo ing f nction is an inner prod ct on M 22.
u4v ' ae 5 bf 5 cg 5 dh Deter%ine the inner prod ct of the %atrices .
==h g
f e
d c
bavu 4
olution$io% 1: u4v ' ae 5 bf 5 cg 5 dh ' ea 5 fb 5 gc 5 hd = v4
u
$io% &: 6et k be a scalar. Thenk u4v ' kae 5 kbf 5 kcg 5 kdh ' k (ae 5 bf 5 cg 5 dh) ' k u4
v *)01()90()2&()2(409
2
10
&2 =×+×+×−+×= −
−092
and10&2
8/18/2019 Examples Done on Orthogonal Projection
22/79
!!
Example !.*8onsider the !ector space P n of polyno%ials of degree ≤ n. 6et f
and g be ele%ents of P n. Pro!e that the follo ing f nctiondefines an inner prod ct of P n.
Deter%ine the inner prod ct of polyno%ials
f ( x) ' x2 5 2 x 1 and g ( x) ' * x 5 1
∫ =1
0)()(g4 dx x g x f f
olution$io% 1: f g dx x f x g dx x g x f g f 4)()()()(4
1
0
1
0=== ∫ ∫
h g h f
dx xh x g dx xh x f
dx xh x g xh x f
dx xh x g x f h g f
44
)()();()(<
);()()()(<
)();()(
8/18/2019 Examples Done on Orthogonal Projection
23/79
8/18/2019 Examples Done on Orthogonal Projection
24/79
!+
orm of a ector
Definition 2.26et V be an inner prod ct space. The norm of a !ector v isdenoted >>v>> and it defined by
vv,v =
The nor% of a !ector in R n
can be e$pressed in ter%s of the dot prod ct as follo s
)444()444()()444(
2121
22121
nn
nn
x x x x x x x x x x x⋅=
++=
?enerali e this definition:The nor%s in general !ector space do not necessary ha!e geo%etricinterpretations4 b t are often i%portant in n %erical or/.
8/18/2019 Examples Done on Orthogonal Projection
25/79
!,
Example !.+8onsider the !ector space P n of polyno%ials ith inner prod ct
The nor% of the f nction f generated by this inner prod ct is
Deter%ine the nor% of the f nction f ( x) ' x2 5 1.
∫ =1
0 )()(4 dx x g x f g f
∫ ==1
0
2);(
8/18/2019 Examples Done on Orthogonal Projection
26/79
!4
Example !., 8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v
defined as follo s be arbitrary 2 × 2 %atrices.
At is /no n that the f nction u4v ' ae 5 bf 5 cg 5 dh is aninner prod ct on M 22 by #$a%ple 2.
The nor% of the %atri$ is
==h g
f e
d c
bavu 4
22224 d cba +++== uuu
8/18/2019 Examples Done on Orthogonal Projection
27/79
!8
Definition 2.!6et V be an inner prod ct space. The angle θ bet een t onon ero !ectors u and v in V is gi!en by
vuvu,=θ cos
The dot prod ct in R n as sed to define angle bet een !ectors.The angle θ bet een !ectors u and v in R n is defined by
( )cosvuvu ⋅=θ
An%le #etween two vectors
8/18/2019 Examples Done on Orthogonal Projection
28/79
!9
An%le #etween two vectors
In " n #e first prove C$ ine%ualit&
' then define cos(
In " 2 #e first define cos(' then prove C$ine%ualit&
8/18/2019 Examples Done on Orthogonal Projection
29/79
!=
Example !.48onsider the inner prod ct space P n of polyno%ials ith inner
prod ctThe angle bet een t o non ero f nctions f and g is gi!en by
Deter%ine the cosine of the angle bet een the f nctions f ( x) ' x2 and g ( x) ' & x
∫ =1
0 )()(4 dx x g x f g f
g f
dx x g x f
g f
g f
)()(
4cos
1
0∫ ==θ
olution =e first co%p te >> f >> and >> g >>.
&;&
8/18/2019 Examples Done on Orthogonal Projection
30/79
*2
Example !.8 8onsider the !ector space M 22 of 2 × 2 %atrices. 6et u and v
defined as follo s be arbitrary 2 × 2 %atrices.
At is /no n that the f nction u4v ' ae 5 bf 5 cg 5 dh is aninner prod ct on M 22 by #$a%ple 2.
The nor% of the %atri$ is
The angle bet een u and v is
==h g
f e
d c
bavu 4
22224 d cba +++== uuu
22222222
4cos
h g f ed cba
dhcg bf ae
+++++++++==
vu
vuθ
8/18/2019 Examples Done on Orthogonal Projection
31/79
*1
"rtho%onal ectorsDef 2.4. 6et V be an inner prod ct space. T o non ero !ectors u and v in V are said to be orthogonal if
04 =vu
Example 2.8
Bho that the f nctions f ( x) ' & x 2 and g ( x) ' x are orthogonal
in P n ith inner prod ct .)()(41
0∫ = dx x g x f g f olution
0;
8/18/2019 Examples Done on Orthogonal Projection
32/79
*!
Jistance
Definition 2.)6et V be an inner prod ct space ith !ector nor% defined by
The distance bet een t o !ectors (points) u and v is definedd (u4v) and is defined by
vv,v =
)4( )4( vuvuvuvu −−=−=d
s for nor%4 the concept of distance ill not ha!e direct
geo%etrical interpretation. At is ho e!er4 sef l in n %erical%athe%atics to be able to disc ss ho far apart !ario sf nctions are .
8/18/2019 Examples Done on Orthogonal Projection
33/79
**
Example !.98onsider the inner prod ct space P n of polyno%ials disc ssed
earlier. Deter%ine hich of the f nctions g ( x) ' x2
& x 5 or h( x)' x2 5 * is closed to f ( x) ' x2.olution
1&)&(&4&4);4(<1
0
22 =−=−−=−−= ∫ dx x x x g f g f g f d 1)*(*4*4);4(<
1
0
22 =−=−−=−−= ∫ dxh f h f h f d Th sThe distance bet een f and h is *4 as e %ight s spect4 g is closer
than h to f.
.*)4(and1&)4( == h f d g f d
* ' S h idt " th % li( ti
8/18/2019 Examples Done on Orthogonal Projection
34/79
*+
*.'ram-Schmidt "rtho%onali(ation'iven a vector s 6 any vector v in an inner product space V can #e decomposed as
( ) proj proj= + −s sv v v v CC ⊥= +v v CC 0⊥× =v vwhereJefinition *.1 ; )utually "rtho%onal ectors
ectors v 16 K6 v k ∈ are mutually ortho%onal if v i Lv j 2
∀
i≠
j Theorem *.1 ;
A set of mutually ortho%onal non-(ero vectors islinearly independent.Proof ;
i ii
c =∑ v 0 0 j i i j j ji
c c=× =×∑v v v v
c j ' 0 ∀ j*+
8/18/2019 Examples Done on Orthogonal Projection
35/79
*,
8orollary .&.1 :
set of k % t ally orthogonal non ero !ectors in Vk
is a basis forthe space.
Definition &.&: Orthonor%al Easis n orthonor%al basis for a !ector space is a basis of % t ally
orthogonal !ectors of nit length.
Definition &.2 : Orthogonal Easis n orthogonal basis for a !ector space is a basis of % t ally
orthogonal !ectors.
*.'ram-Schmidt "rtho%onali(ation
Definition &.* : Orthogonal 8o%ple%ent
The orthogonal co%ple%ent of a s bspace M of * is M ⊥ ' F v ∈ * >
v is perpendic lar to all !ectors in M G ( read H M perpI ).
The orthogonal projection proj M (v ) of a !ector is its projection into
M alon M ⊥ .
8/18/2019 Examples Done on Orthogonal Projection
36/79
*4
6e%%a &.1 :
6et M be a s bspace of n. Then M ⊥ is also a s bspace and n. ' M ⊕ M ⊥ .
Jence4 ∀ v∈ n.4 v − proj M (v) is perpendic lar to all !ectors in M .
Proof : 8onstr ct bases sing ?-B orthogonali ation.
Theore% &.2 :
6et v be a !ector inn
and let M be a s bspace ofn.
ith basis β1 4 K4 βk .AfA is the %atri$ hose col %ns are the βLs then
proj M (v ) ' c1β1 5 K5 ck βk
here the coefficients ci are the entries of the !ector (A T A)-1 AT v. That is4
proj M (v ) ' A (A T A)−1 AT v.
Proof : ( ) M proj M ∈v
T T =A A c A v
here c is a col %n !ector
( )T = −0 A v Ac
( ) M proj =v A c
Ey le%%a&.M4
( ) 1T T −=c A A A v*4
8/18/2019 Examples Done on Orthogonal Projection
37/79
*8
Anterpretation of Theore% &.2:
Af ' β1 4 K4 βk is an orthonor%al basis4 then A T A ' .
An hich case4 proj M (v ) ' A (A T A)−1 AT v ' A A T v.
( ) ( )1
1
T
M k T
k
proj
÷= × ÷ ÷
β
vββv
β
L M ( )1
1
T
k T
k
× ÷= ÷ ÷×
βv
ββ
βv
L M ( )1
1 k
k
v
v
÷= ÷ ÷
ββ L M
B
1
k
j j j
v=
=∑
βith j jv = ×βv
An partic lar4 if ' k 4 then A ' A T ' .
An case is not orthonor%al4 the tas/ is to find ! s.t. " ' A! and B T B ' .
( ) ( )T = A! A! T T = ! A A! ( )1 1T T − −=A A ! ! ( )1T −= !!
Jence
( )1T T −=!! A A
( ) T M proj =v "" v ( ) T = A! A! v T T = A!! A v ( )
1T T −= A A A A v
*8
8/18/2019 Examples Done on Orthogonal Projection
38/79
*9
#$a%ple &.1 :
To orthogonally project1
1
1
÷= − ÷ ÷
v 0
x
P y x z
z
÷= + = ÷ ÷
1 0
0 1
1 0
÷= ÷ ÷−
A
into s bspace
7ro%1 0
0 1 4
1 0
P x y x y ÷ ÷= + ∈ ÷ ÷ ÷ ÷ −
R e get
( )1
1 01C 2 0 1 0 1
0 10 1 0 1 0
1 0
T T − − ÷= ÷ ÷ ÷ ÷−
A A A A
1 01 0 1
0 10 1 01 0
T
− ÷= ÷ ÷ ÷−
A A2 0
0 1
= ÷ ( )1 1C 2 0
0 1T −
= ÷ A A
1 01C 2 0 1C 2
0 10 1 0
1 0
− ÷= ÷ ÷ ÷−
1C 2 0 1C 2
0 1 0
1C 2 0 1C 2
− ÷= ÷ ÷−
( )1C 2 0 1C 2 1
0 1 0 1
1C 2 0 1C 2 1 P proj
− ÷ ÷= − ÷ ÷ ÷ ÷−
v
0
1
0
÷= − ÷ ÷
*9
8/18/2019 Examples Done on Orthogonal Projection
39/79
*=
Exercises *.
1. Perform the 'ram-Schmidt process on this #asis for , & 4
2 1 0
2 4 0 4 &
2 1 1
÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷−
2. Show that the columns of an n×n matrix form anorthonormal set if and only if the inverse of the matrix is itstranspose. Produce such a matrix.
*=
8/18/2019 Examples Done on Orthogonal Projection
40/79
+2
+ Projection Into a Su#spaceJefinition *.1 ; or any direct sum V M⊕ N and any v ∈ V
such that v m / n with m ∈ M and n ∈ N .The projection of v into M alon% N is defined as E5v7 proj M 6N 5v 7 m
eminder ; M M N need not #e ortho%onal.There need not
even #e an inner product defined.
+2
⊕
Theorem*.1; Show that 5i7 E is linear and 5ii7 E ! E.
Theorem*.!; &et E; > is linear and E! E then65i7 E5u7 u for any u N ImE. 5ii7 is the direct sum of the
ima%e and Bernel of E. i.e.6 ImE DerE. 5iii7 E is theprojection of into ImE6 its ima%e alon% DerE.
Theorem*.!; &et E; > is linear and E! E then65i7 E5u7 u for any u N ImE. 5ii7 is the direct sum of the
ima%e and Bernel of E. i.e.6 ImE DerE. 5iii7 E is theprojection of into ImE6 its ima%e alon% DerE.
8/18/2019 Examples Done on Orthogonal Projection
41/79
+1
Projection and )atrices
O &et &5 7 & & is a linear map #etween and itself3. &5 7is a vector space under function addition and scalarmultiplication.
O dim5&7 n! if d5 7 n.O If we fix a #asis in 6 there arises a one to one
correspondence #etween & and set of all matrices oforder n 5the last set is also a vector space withdimension n ! 7. The result implies that matrices identifylinear operators.
8/18/2019 Examples Done on Orthogonal Projection
42/79
+!
"rtho%onal Projection
"
Q 516!7
5!617R
two vector spaces 16 ! #y multiplyin% the vectors and#y B where B .
1 B and ! B
2
1
1
2
∈
+!
2
1
1
2
8/18/2019 Examples Done on Orthogonal Projection
43/79
+*
*rthogonal +ro,ection
ow6 we can find the vector space ! as 1 !&et #e any vector of ! and B 16 B! then we can
write B
1 /B
!
Therefore6
⊕
2
1
x
x
2
1
x
x
2
1
1
2
∈
+*
2
1
x
x
12
21
2
1
k
k
2
1
k
k 1
12
21 −
2
1
k
k
−
+−
21
21
&1
&2
&2
&1
x x
x x
2
1
x
x
8/18/2019 Examples Done on Orthogonal Projection
44/79
++
"rtho%onal Projectionow6 let P #e a projection matrix and x #e a vector in ! 6
then a projection from ! to 1 is %iven #y6
Px B 1
++
2
1-1F* x1 /!F* x ! U 2
1-
−
−
&
*
&
2&
2
&
1
2
1
x
x
Ex. 17 ChecB that P is idempotent #ut not symmetric. :hy<
!7 Prove that if the second vector is 6 P will #e then
idempotent as well as symmetric
−12
8/18/2019 Examples Done on Orthogonal Projection
45/79
+,
)eanin% of P nVn xnV1O Case 1; P nVn is sin%ular #ut not idempotent
≠21 1
, ( ) 1,2 2
P rank P P P
1 13
2 2
The whole space6 n is mapped to the column space ofPnVn 6 an improper su#space of n .
An vector of the su#space may mapped to another vector ofthe Su#space.6
1 12 21 1
1 2
2 2
1 1 1( )
2 2 2
P x x x x
x x
8/18/2019 Examples Done on Orthogonal Projection
46/79
+4
)eanin% of P nVn xnV1
O Case !; PnVn
is sin%ular and idempotent5 asymmetric7
21 0
, ( ) 11 0
P rank P and P P
¬11
2
1
1Px
x x
x 3 1
32 1
→
Px is not ortho%onal to x-Px
2 2
2 2P
The whole space6 n
is mapped to the columnspace of PnVn 6 an improper su#space of n .
An vector of the su#space is mapped to the samevector of the Su#space.6
8/18/2019 Examples Done on Orthogonal Projection
47/79
+8
)eanin% of P nVn xnV1
O Case *; P nVn is sin%ular and idempotent5 symmetric7
)eanin%; The whole space6 n is mapped to the column space ofP nVn 6 an improper su#space of n . An vector of the su#space ismapped to the same vector of the Su#space. It is ortho%onal
projection6 That is6 the su#space is to its complement. or example6
2
1
2
1
2
1
2
1
Px 5x1/x ! 7 1/ 2
1/ 2 Px is ortho%onal to x-Px
8/18/2019 Examples Done on Orthogonal Projection
48/79
+9
)eanin% of P nVn xnV1
O Case +; P nVn is non-sin%ular and non-ortho%onal )eanin%; The whole space6 n is mapped to the column space of
P nVn 6 same as n . The mappin% is one-to-one and onto.:e havenow columns of P nVn as a new 5o#li$ue7 #asis in place of standard#asis. An%les #etween vectors and len%th of vectors are notpreserved. or example6
1 22 1
1, x y Px y x P y → →
8/18/2019 Examples Done on Orthogonal Projection
49/79
+=
)eanin% of P nVn xnV1O Case ; P nVn is non-sin%ular and ortho%onal
)eanin%; The whole space6 n is mapped to the column space ofP nVn 6 same as n . The mappin% is one-to-one and onto.:ehave now columns of P nVn as a new 5ortho%onal7 #asis in place
of standard #asis. An%les #etween vectors and len%th of vectorsare preserved. :e have only a rotation of axes. or example6
2
1-
2
1
2
1
2
1 rom a symmetricmatrix we havealways such a Pof its nindependentei%en vectors
rom a symmetricmatrix we have alwaya symmetricidempotent P of itsr5Wn 7independentei%en vectors
P j i Th I Xil# S
8/18/2019 Examples Done on Orthogonal Projection
50/79
,2
Projection Theorem In a Xil#ert Space
&et #e a closed su#space of a Xil#ert space6 . Thereexists a uni$ue pair of mappin%s P;X → and Y; → suchthat x Px/Yx for all xN . P and Y have the followin%properties;
i7 x N x x6 Yx 2
ii7 x N T Px 26 Yx x
ii7 Px is closest vector in to x.
iv7 Yx is closest vector in to x
v7 Px! / Yx ! x!
vi7 P and Y are linear maps and P ! P6 Y ! Y
⇒
⇒
⇒
inite dimensional spaces are always closed
1
8/18/2019 Examples Done on Orthogonal Projection
51/79
,1
The common characteristic 5structure7 amon% thefollowin% statistical methods<
1. Principal Components Analysis!. 5 id%e 7 re%ression*. isher discriminant analysis+. Canonical correlation analysis,.Sin%ular value decomposition4. Independent component analysis
Applications
We consider linear co binations of inp!t vector" ( ) T f x # x=:e maBe use concepts of len%th and dot productavaila#le in Euclidean space.
8/18/2019 Examples Done on Orthogonal Projection
52/79
,!
:hat is feature reduction<
d $ ℜ∈ pd T % ×ℜ∈
p & ℜ∈
d T d p
& %$ & % ℜ∈=→ℜ∈ ×
:
&inear transformation
"ri%inal data reduced data
Ji i li d i
8/18/2019 Examples Done on Orthogonal Projection
53/79
,*
Jimensionality eduction
O "ne approach to deal with hi%h dimensional data is #yreducin% their dimensionality.
O Project hi%h dimensional data onto a lower dimensionalsu#-space usin% linear or non-linear transformations.
8/18/2019 Examples Done on Orthogonal Projection
54/79
,+
Principal Component Analysis 5PCA7
1 1 2 2
1 2
" ...here 4 4...4 isa basein the -di%ensionals b-space (NO3)
' '
'
x b ! b ! b !! ! ! ' = + + +
" x x=
1 1 2 2
1 2
...here 4 4...4 isa basein the original 3-di%ensionalspace
( (
n
x a v a v a vv v v
= + + +
O ind a #asis in a low dimensional su#-space;
Z Approximate vectors #y projectin% them in a low dimensional su#-space;
517 "ri%inal space representation;
5!7 &ower-dimensional su#-space representation;
O Note if D 6 then
P i i l C t A l i 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
55/79
,,
Principal Component Analysis 5PCA7O Information loss
ZJimensionality reduction implies information loss [[ Z PCA preserves as much information as possi#le;
O :hat is the ?#est@ lower dimensional su#-space< The ?#est@ low-dimensional space is centered at the sample mean and has directions determined #y the ?#est@ ei%envectors of the covariance matrix of the data x.
Z y ?#est@ ei%envectors we mean those correspondin% to the largest ei%envalues 5 i.e.6 principal components/ 7.
Z Since the covariance matrix is real and symmetric6 these ei%envectors are ortho%onal and form a set of #asis vectors.
"%in >> >> (reconstr ction error) x x−
ZSin%ular alue
8/18/2019 Examples Done on Orthogonal Projection
56/79
,4
Principal Component Analysis 5PCA7O )ethodolo%y
Z Suppose x 16 x ! 6 ...6 x M are N x 1 vectors
8/18/2019 Examples Done on Orthogonal Projection
57/79
,8
Principal Component Analysis 5PCA7O )ethodolo%y Z cont.
( )T i ib ! x x= −
8/18/2019 Examples Done on Orthogonal Projection
58/79
,9
Principal Component Analysis 5PCA7O &inear transformation implied #y PCA
Z The linear transformation R N
→ R !
that performs the dimensionality reduction is;
P i i l C t A l i
8/18/2019 Examples Done on Orthogonal Projection
59/79
,=
Principal Component Analysis5PCA7
O Ei%envalue spectrum
# i$ # %
8/18/2019 Examples Done on Orthogonal Projection
60/79
42
Principal Com(ponent Analysis 5PCA7O :hat is the error due to dimensionality reduction<
O It can #e shown that error due to dimensionality reduction is e$ual to;
∑
(
k ii
(
iie
11
2
8/18/2019 Examples Done on Orthogonal Projection
61/79
41
Principal Component Analysis 5PCA7O Standardi(ation
Z The principal components are dependent on the "nits used tomeasure the ori%inal varia#les as well as on the range of values theyassume.
Z :e should always standardi(e the data prior to usin% PCA. Z A common standardi(ation method is to transform all the data to
have (ero mean and unit standard deviation;
Principal Component Analysis 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
62/79
4!
Principal Component Analysis 5PCA7
O Case Study ; Ei%enfaces for ace JetectionF eco%nition
Z ). TurB6 A. Pentland6 \Ei%enfaces for eco%nition\6 #o"rnal of $ogniti%eNe"roscience 6 vol. *6 no. 16 pp. 81-946 1==1.
O 0ace "ecognition
Z The simplest approach is to thinB of it as a templatematchin% pro#lem
Z Pro#lems arise when performin% reco%nition in ahi%h-dimensional space.
Z Si%nificant improvements can #e achieved #y firstmappin% the data into a lo&er di'ensionalityspace.
Z Xow to find this lower-dimensional space<
8/18/2019 Examples Done on Orthogonal Projection
63/79
4*
Principal Component Analysis 5PCA7O )ain idea #ehind ei%enfaces
a!erage face
8/18/2019 Examples Done on Orthogonal Projection
64/79
4+
Principal Component Analysis 5PCA7O Computation of the ei%enfaces Z cont.
i
)
i n d t h a t t h i s i s n
or m
al i (
e d ..
8/18/2019 Examples Done on Orthogonal Projection
65/79
4,
Principal Component Analysis 5PCA7O Computation of the ei%enfaces Z cont.
P i i l C t A l i 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
66/79
44
Principal Component Analysis 5PCA7O epresentin% faces onto this #asis
Principal Component Analysis 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
67/79
48
Principal Component Analysis 5PCA7
O epresentin% faces onto this #asis Z cont.
P i i l C t A l i 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
68/79
49
Principal Component Analysis 5PCA7O ace eco%nition ]sin% Ei%enfaces
8/18/2019 Examples Done on Orthogonal Projection
69/79
4=
Principal Component Analysis 5PCA7O ace eco%nition ]sin% Ei%enfaces Z cont.
Z The distance e r is called distance &ithin the face space 5difs 7
Z Comment; we can use the common Euclidean distance to compute e r 6however6 it has #een reported that the Mahalano(is distance performs #etter;
P i i l C t A l i 5PCA7
8/18/2019 Examples Done on Orthogonal Projection
70/79
82
Principal Component Analysis 5PCA7O ace Jetection ]sin% Ei%enfaces
l l
8/18/2019 Examples Done on Orthogonal Projection
71/79
81
Principal Component Analysis 5PCA7O ace Jetection ]sin% Ei%enfaces Z cont.
Principal Component Analysis
8/18/2019 Examples Done on Orthogonal Projection
72/79
8!
Principal Component Analysis5PCA7O econstruction
of faces and non-faces
Principal Component Analysis
8/18/2019 Examples Done on Orthogonal Projection
73/79
8*
Principal Component Analysis5PCA7
O Applications
Z ace detection6 tracBin%6 and reco%nitiondffs
Principal Components
8/18/2019 Examples Done on Orthogonal Projection
74/79
8+
Principal Components Analysis
So6 principal components are %iven #y;
( 1 " 11 x 1 / " 1! x ! / ... / " 1 x
( ! " !1 x 1 / " !! x ! / ... / " ! x ...( a 1 x 1 / a ! x ! / ... / a x
x j Gs are standardi(ed if correlation 'atrix is used5mean 2.26 SJ 1.27
8/18/2019 Examples Done on Orthogonal Projection
75/79
8,
Principal Components Analysis
core of i th unit on j th principalcomponent
( i,j " j 1 x i 1 / " j ! x i ! / ... / " jN x iN
8/18/2019 Examples Done on Orthogonal Projection
76/79
84
PCA Scores
+.2 +., ,.2 ,., 4.2!
*
+
,
xi)
xi*
bi+* bi+)
8/18/2019 Examples Done on Orthogonal Projection
77/79
88
Principal Components Analysis
Amount of variance accounted for #y;1st principal component6 ^16 1st eigenvalue
!nd principal component6 ^!6 !nd eigenvalue
...
^1 _ ^ ! _ ^* _ ^ + _ ...
Avera%e ̂j 1 5correlation matrix7
Principal Components Analysis;
8/18/2019 Examples Done on Orthogonal Projection
78/79
89
Principal Components Analysis;Ei%envalues
+.2 +., ,.2 ,., 4.2!
*
+
,
1 2
1
8/18/2019 Examples Done on Orthogonal Projection
79/79
han3 &ou
Top Related