Statistical Performance of Convex Tensor Deco mpositionryotat/talks/nips11poster.pdf3)/#=",-)/!...
3
Statistical Convex !"#$% ’#()#*%+ ’%),) - ’./ 01)2/34)$" #5 ’#*"#+ - ! 6787*)+ 9#./) :%"%4.)+ Performance Tensor Deco ; <%3% =14>$7$/ #5 6?)/1?/ @ ’/?. - ; " :)4%4.) 9%4.)(% of mposition 1#A#B"+ C D!E6’F+ G6’ -+C # !"#$%& (%#)*+),-.-)/ 0!"#$%& 112 • D3#HA/(I J)2/1 % K%3>%AA" #H4/32/L %KK3#M)(%$/A" A#NO3%1* $/14#3 !+ P1L n 1 n 2 n 3 r 1 r 2 r 3 = × 1 × 2 × 3 n 1 r 1 n 2 r 2 n 3 r 3 Core Factors • QKKA)?%>#14I ?./(#ORK4"?.#O(/$3)?4+ 4)B1%A K3#?/44)1B+ ?#(K7$/3 2)4)#1+ 1/73#4?)/1?/ • E4>(%>#1I %A$/31%$/ ()1)()8%>#1 S1#1O?#12/MT $ Y ijk = r1 a=1 r2 b=1 r3 c=1 C abc U (1) ia U (2) jb U (3) kc 3)/4%5 !%/,)& 6,.-*7.-)/ U%$3)M ’/14#3 E4>(%>#1 #5 "#$% &’() (%$3)M S.%3LT V#12/M 3/A%M%>#1 E4>(%>#1 #5 "#$% &’() $/14#3 S.%3LT 6?.%W/1 -O1#3( ()1)()8%>#1 S$3%?$%HA/T XY%8/A+ :)1L)+ Z#"L [-\ F2/3A%KK/L 6?.%W/1 -O1#3( ()1)()8%>#1 XLiu+09, Signoretto+10, Tomioka+10, Gandy+11] V#12/M 3/A%M%>#1 J/1/3%A)8/ % ! !"# !"$ !"% !"& !"’ !"( !") !"* !"+ # #! !% #! ! ,-./0123 25 2678-98: 8;8<8307 =--2- >>?@!?A>> , 71B8C’!D’!D$!E -.3FC)D*D+ G2398D =H I323/2398DJ KL01<1B.0123 02;8-.3/8 8).-47.-)/9 :;7,%<.&7/,-.-)/ -/ 3)/4%5 !%/,)& 6,.-*7.-)/ J#%AI EMKA%)1 $.)4 17(H/3 #5 4%(KA/4 U 53#( $./ 4)8/ #5 $./ $/14#3 X1-+ 1;+ 1C\ %1L $./ ’7?*/3 3%1* X3-+ 3;+ 3C\ UR< S<# 1#)4/T & ’/14#3 ?#(KA/>#1 3/47A$ X’#()#*% /$ %A] ;[-[\ 8)(%=9 3)/4%5 !%/,)& 6,.-*7.-)/ Observation model Gaussian noise N(0,σ 2 ) Optimization Reg. Const. Observation model X(W)=(X 1 , W ,..., X M , W) Empirical error Regularization (N = K k=1 n k ) X : R N → R M y i = X i , W * + i (i =1,...,M ) W * true tensor rank-(r 1 ,...,r K ) ˆ W = argmin W∈R n 1 ×···×n K 1 2M y - X(W) 2 2 + λ M W S1 ’ >4%&=7++%( ?#;7..%/ @</)&* A)& !%/,)&, Schatten 1-norm for the mode-k unfolding Example of unfolding (matricization) NB: rank of mode-k unfolding = mode-k rank r k ( W S 1 := 1 K K k=1 W (k) S 1 :&%4-)", B)&$ !"#$%&’ ()’*&+,-%. 0%1*2 !’’"03-%. 4,&5*# !/?.$+ Y%8/A+ D%33)A# ;[[^ !/4$3)?$/L =4#(/$3" U%$3)M V%1L_4 @ !/?.$ ;[[‘ =1?#./3/1?/ U%$3)M </B%.H%1 @ a%)1N3)B.$ ;[-- !/4$3)$/L 6$3#1B V#12/M)$" U%$3)M ’.)4 N#3* !/4$3)$/L 6$3#1B V#12/M)$" ’/14#3 y i = X i ,W (i =1,...,M) Y ij = W ij ((i, j ) ∈ Ω) y i = X i ,W + i (i =1,...,M) y i = X i ,W + i (i =1,...,M) )
Transcript of Statistical Performance of Convex Tensor Deco mpositionryotat/talks/nips11poster.pdf3)/#=",-)/!...
Statistical
Convex!!"#$%&'#()#*%+&'%),)!
-'./&01)2/34)$"&'#*"#+!
-!
!!
6787*)+&9#./)&:%"%4.)+!
Performance
Tensor Deco!
;<%3%&=14>$7$/&6?)/1?/&@&'/?.!
-! ;!
"!
:)4%4.)&9%4.)(%!
of
mposition!
1#A#B"+&&CD!E6'F+&G6'!
-+C!
#!
!"#$%&'(%#)*+),-.-)/'0!"#$%&'112!•! D3#HA/(I&J)2/1&%&K%3>%AA"&#H4/32/L&%KK3#M)(%$/A"&
A#NO3%1*&$/14#3&!+&P1L&
n1 n2
n3
r1 r2
r3 = !1 !2 !3
n1 r1
n2 r2
n3 r3
Core Factors
•! QKKA)?%>#14I&?./(#ORK4"?.#O(/$3)?4+&4)B1%A&
K3#?/44)1B+&?#(K7$/3&2)4)#1+&1/73#4?)/1?/&
•! E4>(%>#1I&%A$/31%$/&()1)()8%>#1&S1#1O?#12/MT& $!
!Yijk =
r1"
a=1
r2"
b=1
r3"
c=1
CabcU(1)ia U (2)
jb U (3)kc
#
3)/4%5'!%/,)&'6,.-*7.-)/!
U%$3)M!
'/14#3!
E4>(%>#1&"#$%
&'()&(%$3)M&
S.%3LT! V#12/M&
3/A%M%>#1!
E4>(%>#1&"#$%
&'()&$/14#3&
S.%3LT!
6?.%W/1&-O1#3(&
()1)()8%>#1&
S$3%?$%HA/T&XY%8/A+&:)1L)+&Z#"L&[-\&
F2/3A%KK/L&
6?.%W/1&-O1#3(&
()1)()8%>#1&XLiu+09, Signoretto+10, Tomioka+10, Gandy+11]&
V#12/M&
3/A%M%>#1!
J/1/3%A)8/!
%!
! !"# !"$ !"% !"& !"' !"( !") !"* !"+ #
#!!%
#!!
,-./012342542678-98:48;8<8307
=--2-4>>?@!?A>> ,
71B8C'!D'!D$!E4-.3FC)D*D+
4
4
G2398D
=H4I323/2398DJ
KL01<1B.0123402;8-.3/8
8).-47.-)/9':;7,%<.&7/,-.-)/'-/'
3)/4%5'!%/,)&'6,.-*7.-)/!
J#%AI&EMKA%)1&$.)4&17(H/3&4%(KA/4&U&53#(&$./&4)8/&
$./&$/14#3&X1-+&1;+&1C\&%1L&$./&'7?*/3&3%1*&X3-+&3;+&3C\!
UR<!
S<#&1#)4/T!
&!
'/14#3&?#(KA/>#1&3/47A$&X'#()#*%&/$&%A]&;[-[\!
8)(%=9'3)/4%5'!%/,)&'6,.-*7.-)/!
Observation model
Gaussian noise N(0,σ2) Optimization
Reg. Const. Observation model X(W) = (!X 1, W" , . . . , !X M , W")!
Empirical error Regularization
(N =!K
k=1 nk) X : RN ! RM
yi = !X i, W!" + !i (i = 1, . . . , M)
W! true tensor rank-(r1,...,rK)
W = argminW!Rn1!···!nK
! 12M
!y " X(W)!22 + !M
""""""W""""""
S1
#
'!
>4%&=7++%('?#;7..%/'@</)&*'A)&'!%/,)&,!
Schatten 1-norm for the mode-k unfolding
Example of unfolding (matricization)
NB: rank of mode-k unfolding = mode-k rank rk (!
!!!!!!W!!!!!!
S1:=
1K
K"
k=1
!W (k)!S1
:&%4-)",'B)&$!!"#$%&'! ()'*&+,-%./0%1*2! !''"03-%.! 4,&5*#!
!/?.$+&Y%8/A+&
D%33)A#&;[[^!
!/4$3)?$/L&
=4#(/$3"!
U%$3)M!
V%1L_4&@&!/?.$&
;[[`!
=1?#./3/1?/! U%$3)M!
</B%.H%1&@&
a%)1N3)B.$&
;[--!
!/4$3)$/L&
6$3#1B&V#12/M)$"!
U%$3)M!
'.)4&N#3*! !/4$3)$/L&
6$3#1B&V#12/M)$"!
'/14#3!
yi = !Xi, W "(i = 1, . . . , M)
Yij = Wij
((i, j) ! !)
yi = !Xi, W " + !i
(i = 1, . . . , M)
yi = !Xi, W " + !i
(i = 1, . . . , M))!
C%,.&-#.%(',.&)/D'#)/4%5-.E'FC?3G!
•! Q447(/&$.%$&$./3/&)4&%&K#4)>2/&?#14$%1$bScT&
47?.&$.%$&5#3&%AA&$/14#34&d*V&
S'./&4/$&V&1//L4&$#&H/&L/P1/L&?%3/57AA"T&
<#$/I&
•! =5&Ve!*+&bScTe()1&/)BSc'cT&&&&&Sc*!+M*T&
•! a./1&Uf<+&3/4$3)?>#1&)4&1/?/44%3"]&
•! './&4(%AA/3&V+&$./&N/%*/3&$./&%447(K>#1]&
S?5]&</B%.H%1&@&a%)1N3)B.$&--T
1M
!X(!)!22 " !(X)
!!!!!!!!!!!!!2
F
!+!
H%**7'@9'I'$%E'-/%J"7=-.E!
!!!!!!X!!!!!!
mean:=
1K
K"
k=1
##X(k)
##S!
!W ,X " #!!!!!!W
!!!!!!S1
!!!!!!X!!!!!!
mean
!X!S!:= max
j!{1,...,m}!j(X)
W ,X ! Rn1!···!nK
!X!S1 :=m!
j=1
!j(X)
where
K=2: norm duality (tight) K>2: not tight
!!!!!!W!!!!!!
S1:=
1K
K"
k=1
!W (k)!S1
!!!
!;%)&%*'@'F(%.%&*-/-,.-#G!
•! 6#A7>#1&$./&#K$]&K3#HA/(&&
•! !/B&?#14$&gU&4%>4P/4&
•! 01L/3&$./&!6V&%447(K>#1&
W
!M ! 2!!!!!!X!(!)
!!!!!!mean
/M
!!!!!!W !W!!!!!!!F" 32!M
"(X)1K
K"
k=1
#rk
!!!!!!X!!!!!!
mean:=
1K
K"
k=1
##X(k)
##S!
N./3/ S1#)4/&L/4)B1&?#33/A%>#1T X!(!) =!M
i=1 !iX i
S?5]&</B%.H%1&@&a%)1N3)B.$&--T !"!
!B)',+%#-7='#7,%,!•! <#)4"&$/14#3&L/?#(K#4)>#1&SUe<T&
–!!6VI&$3)2)%A]&
–!H#71L&$./&1#)4/OL/4)B1&?#33/A%>#1&$/3(&
!(X) = 1/M
E!!!!!!X!(!)
!!!!!!mean
! !
K
K"
k=1
#"nk +
$N/nk
%
E!!!!!!X!(!)
!!!!!!mean
! !"
M
K
K"
k=1
#"nk +
$N/nk
%
•! !%1L#(&J%744&L/4)B1&
–!!6VI&(#3/&L)h?7A$&Si/((%&jT&
–!H#71L&$./&1#)4/OL/4)B1&?#33/A%>#1&$/3(&
!#!
Si/((%&CT!
Si/((%&kT!
!;%)&%*'K'F/)-,E'.%/,)&'(%#)*+LG!
a./1&%AA&$./&/A/(/1$4&%3/&#H4/32/L&SUe<T&%1L&
$./&3/B7A%3)8%>#1&?#14$]&4%>4P/4&
where
!n!1!1/2 :=!
1K
"Kk=1
#1/nk
$2, !r!1/2 :=
!1K
"Kk=1
"rk
$2
!!!!!!W !W!!!!!!!2F
N" Op
"!2#n"1#1/2#r#1/2
#
Normalized rank
=5&1*e1&%1L&3*e3+&1#3(%A)8/L&3%1*&e&3R1!!$!
!M ! c0"!K
k=1
""nk +
#N/nk
$/(KN)
Then
0 0.2 0.4 0.6 0.8 10
0.005
0.01
0.015
0.02
0.025
0.03
Normalized rank
Mean
sq
uare
d e
rror
size=[50 50 20] !M
=0.33/N
size=[50 50 20] !M
=2.34/N
size=[50 50 20] !M
=6/N
size=[100 100 50] !M
=0.66/N
size=[100 100 50] !M
=4.5/N
size=[100 100 50] !M
=12/N
! !"# !"$ !"% !"& '!
'
#
()*'!
!$
+,-./01234*-/56
73/5*89:/-34*3--,-
*
*
8123;<=!*=!*#!>*!7;!"!(?+
8123;<=!*=!*#!>*!7;!"((?+
8123;<=!*=!*#!>*!7;!"=$?+
8123;<'!!*'!!*=!>*!7;!"!%?+
8123;<'!!*'!!*=!>*!7;!"%@?+
8123;<'!!*'!!*=!>*!7;'"''?+
?-*"=7.-)/9'M)-,E'.%/,)&'(%#)*+),-.-)/!
6(%AA&1#)4/&Sle[][-T! i%3B/&1#)4/&Sle[]-T!
!!!!!!W !W!!!!!!!2F
NMean squared error
A)1/%3&3/A%>#1&H/$N//1&U6E&%1L&1#3(%A)8/L&3%1*m! !%!
!;%)&%*'N9'&7/()*'O7",,'(%,-D/!
!n!1!1/2 :=!
1K
"Kk=1
#1/nk
$2, !r!1/2 :=
!1K
"Kk=1
"rk
$2
Q447(/&/A/(/1$4&c)&%3/&L3#N1&))L&53#(&4$%1L%3L&
1#3(%A&L)4$3)H7>#1．U#3/#2/3&
<#3(%A)8/L&3%1*
!!!!!!W !W!!!!!!!2F
N" Op
"!2#n"1#1/2#r#1/2
M
#V#12/3B/1?/m
#samples (M)
#variables (N)! c1"n!1"1/2"r"1/2
!&!
!M ! c0"!K
k=1
""nk +
#N/nk
$/(K
"M) %1L!
!r
n
! !"# !"$ !"% !"&!
!"#
!"$
!"%
!"&
'
()*+,-./012*,342553!'55'6#55*55
'6#
7*,89.)32,920**:;!"!'
2
2
<./0;=>!2>!2#!?
<./0;='!!2'!!2>!?
<./0;=>!2>!2#!2'!?
?-*"=7.-)/9'!%/,)&'3)*+=%.-)/!
! !"# !"$ !"% !"& '
'!!(
'!!
)*+,-./01/21/345*6571585950-4
:4-.9+-./015**/*
1
1
;/065<1=>1&1?@
;/65<1=$!1?1>@
AB-.9.C+-./01-/85*+0,5
S<#&1#)4/T!<#3(%A)8/L&3%1*!
Y3%?>#1&U
R<&%$&
/33fe[][-!
!'!
! !"# !"$ !"% !"& !"' !"(!
!"$
!"&
!"(
!")
#
*+,-./01234,.564775!#77#8$77,77
#8$
9,.:;0+54.;42,,<=!"!#
4
4
>012=?'!4$!@
>012=?#!!4&!@
>012=?$'!4$!!@
87.&-5'P'.%/,)&'#)*+=%.-)/!Matrix completion Tensor completion
Frac
tion M/N
at e
rror<=
0.01
Tensor completion easier than matrix completion!?
! !"# !"$ !"% !"&!
!"#
!"$
!"%
!"&
'
()*+,-./012*,342553!'55'6#55*55
'6#
7*,89.)32,920**:;!"!'
2
2
<./0;=>!2>!2#!?
<./0;='!!2'!!2>!?
<./0;=>!2>!2#!2'!?
!(!
3)/#=",-)/!•! V#12/M&$/14#3&L/?#(K#4)>#1&OOO&1#N&N)$.&K/35#3(%1?/&B7%3%1$//&
•! <#3(%A)8/L&3%1*&K3/L)?$4&/(K)3)?%A&4?%A)1B&H/.%2)#3&N/AA&&
=447/4&
•! a."&(%$3)M&?#(KA/>#1&(#3/&L)h?7A$&$.%1&$/14#3&?#(KA/>#1n&
•! a#34$&?%4/&%1%A"4)4Oo&%2/3%B/&?%4/&%1%A"4)4&
•! Q1%A"8/&$/14#3&?#(KA/>#1&(#3/&?%3/57AA"&
–! =1?#./3/1?/&XV%1L/4&@&!/?.$&[`\&
–!6K)*)1/44&X</B%.H%1&/$&%A]&-[\&!)!
C%A%&%/#%,!•! V%1L/4&@&!/?.$&S;[[`T&EM%?$&(%$3)M&?#(KA/>#1&2)%&?#12/M&#K>()8%>#1]&Y#71L]&V#(K7$]&
U%$.]+&`SpTI^-^q^^;]&
•! J%1L"+&!/?.$+&@&r%(%L%&S;[--T&'/14#3&?#(KA/>#1&%1L&A#NO1O3%1*&$/14#3&3/?#2/3"&2)%&?#12/M&
#K>()8%>#1]&=12/34/&D3#HA/(4+&;^I[;j[-[]&
•! 9#AL%&@&Z%L/3&S;[[`T&'/14#3&L/?#(K#4)>#14&%1L&%KKA)?%>#14]&6=QU&!/2)/N+&j-SCTIkjjqj[[]&
•! i)7+&U74)%A4*)+&a#1*%+&@&r/]&S;[[`T&'/14#3&?#(KA/>#1&5#3&/4>(%>1B&()44)1B&2%A7/4&)1&2)47%A&
L%$%]&=1&D3#5]&=VVs]&
•! </B%.H%1&@&a%)1N3)B.$&S;[--T&E4>(%>#1&S1/%3T&A#NO3%1*&(%$3)?/4&N)$.&1#)4/&%1L&.)B.O
L)(/14)#1%A&4?%A)1B]&Q11]&6$%>4$]+&C`S;T]&
•! !/?.$+&Y%8/A+&@&D%33)A#&S;[-[T&J7%3%1$//L&()1)(7(O3%1*&4#A7>#14&A)1/%3&(%$3)M&/t7%>#14&
2)%&17?A/%3&1#3(&()1)()8%>#1]&6=QU&!/2)/N+&j;SCTIk^-qj[-]&
•! 6)B1#3/W#+&L/&i%$.%7N/3+&@&67"*/14&S;[-[T&<7?A/%3&1#3(4&5#3&$/14#34&%1L&$./)3&74/&5#3&?#12/M&
(7A>A)1/%3&/4>(%>#1]&'/?.&!/K#3$&-[O-up+&9]0]i/72/1]&
•! '#()#*%+&:%"%4.)+&@&9%4.)(%&S;[-[T&F1&$./&/M$/14)#1&$3%?/&1#3(&$#&$/14#34]&=1&<=D6;[-[&
a#3*4.#KI&'/14#34+&9/31/A4&%1L&U%?.)1/&i/%31)1B]&
•! '#()#*%+&:%"%4.)+&@&9%4.)(%&S;[--T&E4>(%>#1&A#NO3%1*&$/14#34&2)%&?#12/M&#K>()8%>#1]&
'/?.1)?%A&3/K#3$+&%3c)2I-[-[][^u`+&;[--]&
•! '7?*/3&S-`ppT&6#(/&(%$./(%>?%A&1#$/4&$.3//O(#L/&5%?$#3&%1%A"4)4]&D4"?.#(/$3)*%+&C-SCTI
;^`qC--]& "+! "!!
3;)),-/D'.;%',%.'3!•! a/A"&1//L&$./&3/4)L7%A&d&$#&H/&)1&V!
mode-k unfolding of the residual
Component spanned by the truth
!(k) = !!k + !!!
k
Orthogonal to the truth
H%**7'Q'FC?3'A)&'&7/()*'O7",,-7/G!
X : Rn1!···!nK " RMi/$!
H/&%&3%1L#(&J%744)4%1&L/4)B1]&'./1!
N)$.&K3#H%H)A)$"&%$&A/%4$!1! 2exp(!N/32)
D3##5I&%1%A#B#74&$#&$.%$&D3#K&-&)1&</B%.H%1&@&
a%)1N3)B.$&;[--&S74/&i/((%&-T!"#!
![AJP-1700VGQ manual 6987799-00...y × e L [ m k ÷ N Z z z Ñ 3 q % k q ÷ [ | Í z 3 3 ´ s s l L i < ] H % ù p Z ' p Z . 6 ÿ L i L N L q % k < ] H % ù y k s ( à < A Ì O ´ [](https://static.fdocuments.in/doc/165x107/610d78eb763be737dd7e673c/ajp-1700vgq-manual-6987799-00-y-e-l-m-k-n-z-z-z-3-q-k-q-.jpg)
![Class 3 Ab + L AbL Review d/ dt [ AbL ] = k on [ Ab ][L] – k off [ AbL ]](https://static.fdocuments.in/doc/165x107/56815b21550346895dc8dddd/class-3-ab-l-abl-review-d-dt-abl-k-on-ab-l-k-off-abl-.jpg)
![h ^ b l e y · 2016. 3. 4. · 3 G Z k l h y s Z b g k l j m k h ^ _ j ` b l k \ _ ^ _ g b y h [ m k l j h c k l \ _ i j b g p b i _ ^ _ c k l l Z o h ] j Z n Z D : DT-20 F ( Z e](https://static.fdocuments.in/doc/165x107/60c6687d4e08d061ac5b25ca/h-b-l-e-y-2016-3-4-3-g-z-k-l-h-y-s-z-b-g-k-l-j-m-k-h-j-b-l-k-.jpg)
![AJP-4170VGQ manual 6987801-00 · y × e L [ m k ÷ N Z z z Ñ 3 q % k q ÷ [ | Í z 3 3 ´ s s l L i < ] H % ù ...](https://static.fdocuments.in/doc/165x107/60af670b560890721044ed25/ajp-4170vgq-manual-6987801-00-y-e-l-m-k-n-z-z-z-3-q-k-q-z.jpg)
