Towards Independent Subspace Analysis in Controlled ...
Transcript of Towards Independent Subspace Analysis in Controlled ...
![Page 1: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/1.jpg)
Towards Independent Subspace Analysis inControlled Dynamical Systems
Zoltán Szabó, András Lorincz
Neural Information Processing Group,Department of Information Systems,
Eötvös Loránd University,Budapest, Hungary
ICA Research Network 2008
Acknowledgements:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 2: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/2.jpg)
Tools to Integrate-1 (Independent Subspace Analysis)
Cocktail party problemGeneralization of ICA:
multidimensional components,groups of ’people/music bands’
Hidden, independent, multidimensional processes – NOCONTROL.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 3: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/3.jpg)
Tools to Integrate-2 (D-optimal Identification ofDynamical Systems)
Problem: estimate the parameters of a fully observablecontrolled dynamical system by the ‘optimal’ choice of thecontrol.
’Parameters’: dynamics, noise.’Optimal’: in information theoretical sense → D-optimality.
Synonyms: active learning, optimal experimental design.
For ARX models: QP in Bayesian framework.
Controlled dynamical system – FULLY OBSERVABLE.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 4: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/4.jpg)
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 5: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/5.jpg)
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
EEG data analysis: recognition + prediction, e.g., epileptic
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 6: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/6.jpg)
Motivation
Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.
EEG data analysis: recognition + prediction, e.g., epileptic
EU-FP7: interested in partnership
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 7: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/7.jpg)
Contents
1 Independent Subspace Analysis2 D-optimal ARX Identification
︸ ︷︷ ︸
↓
3 D-optimal Hidden ARX Identification4 Illustrations
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 8: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/8.jpg)
Independent Subspace Analysis (ISA/MICA)
ISA equations: Observation x is linear mixture ofindependent multidimensional components:
x(t) = As(t),
s(t) = [s1(t); . . . ; sM(t)],
wheresm(t) ∈ R
dm are i.i.d. sampled random variables in time,I(s1, . . . , sM) = 0,mixing matrix A ∈ R
D×D is invertible, with D := dim(s).
Goal: s. Specially for ∀dm = 1: ICA.
Ambiguities: permutation, linear (/orth.) transformation.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 9: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/9.jpg)
D-optimal ARX Identification
Observation equation (ARX model; u: control, e: noise):
s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1).
Task: ’efficient’ estimation ofsystem parameters: Θ = [F, B, parameters(e)], ornoise: e
by the ’optimal’ choice of control u.
Optimality (D-optimal/’InfoMax’):
Jpar (ut+1) := I(Θ, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U
, or
Jnoise(ut+1) := I(et+1, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U
.
Result (Póczos & Lorincz, 2008): In the Bayesian setting,optimization of J can be reduced to QP.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 10: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/10.jpg)
D-optimal Hidden ARX Identification
State (s) + observation (x) equation:
s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1), (1)
x(t) = As(t). (2)
Assumptions: I(e1, . . . , eM) = 0 – hidden non-Gaussianindependent multiD componentsTrick: reduce the problem to the fully observable case(d-dep. CLT) + ISA:
x(t + 1) = [AFA−1]x(t) + [AB]u(t + 1) + [Ae(t + 1)],x – ’fully observable tool’ → [AFA−1], [AB], Ae,Ae – ISA→ A} ⇒ F, B, e.
Note: for higher order ARX systems the same idea holds.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 11: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/11.jpg)
Databases, Performance Measure, Questions
Databases (3D-geom, ABC, celebrities):
Performance measure: Amari-index (r ) ∈ [0, 1], 0–perfect.Questions:
1 Dependence on δu = |Ucontrol|,2 Dependence on J = deg(Bcontrol[z]),3 Dependencies on I = deg(FAR[z]) and λ:
FAR[z] =
I−1∏
i=0
(I − λOiz) (|λ| < 1, λ ∈ R, Oi : RND orth.).
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 12: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/12.jpg)
Illustrations: Dependence on δu = |Ucontrol|
Decline of the estimation error: power-law [r(T ) ∝ T−c (c > 0)]
1 2 5 10 20 50
10−2
10−1
100
3D−geom (I=3, J=3, λ=0.95)
Number of samples (T)
Am
ari−
inde
x (r
)
δu:0.01
δu:0.1
δu:0.2
δu:0.5
δu:1
x1031 2 5 10 20 50
10−2
10−1
100
I=3, J=3, δu=0.2, λ=0.95
Number of samples (T)A
mar
i−in
dex
(r)
3D−geomABCcelebrities
x103
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 13: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/13.jpg)
Illustration: 3D-geom
observation:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 14: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/14.jpg)
Illustration: 3D-geom
observation:
estimated innovation (input of ISA):
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 15: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/15.jpg)
Illustration: 3D-geom
observation:
estimated innovation (input of ISA):
Hinton-diagram, estimated components:
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 16: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/16.jpg)
Illustration: Dependencies on J = deg(Bcontrol[z]),I = deg(FAR[z])
Precise even for J = 50; I = 50 (ր, λ ց)
1 2 5 10 20 50
10−2
10−1
100
3D−geom (I=3, δu=0.2, λ=0.95)
Number of samples (T)
Am
ari−
inde
x (r
)
J:3J:5J:10J:20J:50
x1031 2 5 10 20 50
10−2
10−1
100
3D−geom (J=1, δu=0.2)
Number of samples (T)
Am
ari−
inde
x (r
)
I:5, λ:0.8I:5, λ:0.85I:5, λ:0.9I:10, λ:0.6I:10, λ:0.65I:10, λ:0.7I:50, λ:0.25I:50, λ:0.3
x103
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 17: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/17.jpg)
Summary
Integration of two methodologies:hidden independent multidimensional sources,optimal design in controlled dynamical systems.
Numerical experiences:Decline of the estimation error follows power-law.Robust against:
the order of the AR process,temporal memory of the control.
Possibility to apply ICA, ISA, . . . in controlled systems.
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 18: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/18.jpg)
TYFYA!
Zoltán Szabó, András Lorincz nipg.inf.elte.hu
![Page 19: Towards Independent Subspace Analysis in Controlled ...](https://reader031.fdocuments.in/reader031/viewer/2022012414/616e1dbedf947a5e151ce3b0/html5/thumbnails/19.jpg)
References
B. Póczos and A. Lorincz.
D-optimal Bayesian interrogation for parameter and noise identificationof recurrent neural networks.
2008.
(submitted; available at http://arxiv.org/abs/0801.1883).
V. V. Petrov.
Central limit theorem for m-dependent variables.
In Proceedings of the All-Union Conference on Probability Theory andMathematical Statistics, pages 38–44, 1958.
Z. Szabó.
Separation Principles in Independent Process Analysis.PhD thesis, Eötvös Loránd University, Budapest, 2008.
(submitted; available at nipg.inf.elte.hu: Download).
Zoltán Szabó, András Lorincz nipg.inf.elte.hu