Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples...
Transcript of Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples...
![Page 1: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/1.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Local Multidimensional Scaling
For Nonlinear Dimension Reduction, GraphLayout and Proximity Analysis
Lisha Chen (Statistics, Yale)Joint work with Andreas Buja (Upenn)
Feb 6th, 2007
![Page 2: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/2.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Machine Learning Facial Images
I Facial images vary from moment to moment.
I The mystery of perception:How does brain perceive constancy in flux?
I Goal: Construct machines to capture the variability of facialimages.
![Page 3: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/3.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Machine Learning Facial Images
I Facial images vary from moment to moment.
I The mystery of perception:How does brain perceive constancy in flux?
I Goal: Construct machines to capture the variability of facialimages.
![Page 4: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/4.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Characterizing Image Variability
High-dimensional data . . .
I An image is regarded as a collection of numbers.
I Each number is the light intensity at an image pixel.
I An Example: 64 by 64 images ⇒ 4096-D data.
Low-dimensional manifold . . .
Degree of freedom
I Left-right pose
I Up-down pose
I Lighting-direction
I Expression change(not shown)
![Page 5: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/5.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Characterizing Image Variability
High-dimensional data . . .
I An image is regarded as a collection of numbers.
I Each number is the light intensity at an image pixel.
I An Example: 64 by 64 images ⇒ 4096-D data.
Low-dimensional manifold . . .
Degree of freedom
I Left-right pose
I Up-down pose
I Lighting-direction
I Expression change(not shown)
![Page 6: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/6.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Classical Method 1: PCA
I Principal Components: Fit the best ellipsoid to multivariatedata x1, ..., xN ∈ IRp.
I Linear method: Provide a sequence of best linearapproximation to the data.
I Uses: Plot the wide dimensions; interpret their directions.
![Page 7: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/7.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Classical Method 2: Multidimensional Scaling
Multidimensional Scaling(MDS) for proximity data . . .
I Make a map from given pairwise dissimilarities.
I A Toy Example:
D =
0 3 43 0 54 5 0
MDS for dimension reduction . . .
I Calculate dissimilarities(Dij) in high-dimensional space.
I Match Dij in low-dimensional space.
![Page 8: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/8.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Nonlinear Structure in High Dimensional Data
I Classical methods fail to find nonlinear structure.
−200 0 200 400
−200
0
200
400
Swiss Roll
−300 −200 −100 0 100 200 300
−300
−200
−100
0
100
200
300
2D PCA
0 500 1000 1500
0
500
1000
1500
True 2D Manifold
I Nonlinear dimension reduction methods:I Isomap(Tenenbaum et. al., 2000)I LLE(Roweis & Saul, 2000)I Hessian Eigenmaps(Donoho & Carrie, 2003)I Laplacian Eigenmaps (Belkin & Niyogi, 2003)I Diffusion Maps(Coifman et. al., 2005)
![Page 9: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/9.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Isometric Feature Mapping (Isomap)
The idea . . .
I Characterize the intrinsic geometry by geodesicdistances(shortest path).
I Match geodesic distances by Euclidean distances inlow-dimensional space.
The algorithm . . .
1 : Construct neighborhoodgraph.
2 : Compute shortest path.
3 : Construct embedding by
MDS.
![Page 10: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/10.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Local Linear Embedding(LLE)
The idea . . .
I Nearby points lie on a locally linear patch.
I Characterize the local geometry by linear coefficientsthat reconstruct each data point from its neighbors.
The algorithm . . .
1 : Select neighbors.
2 : Construct weights.
ε(W ) =∑
i
|Xi−∑
j∈N(i)
WijXj |2
3 : Embedding.
Φ(Y ) =∑
i
|Yi−∑
j∈N(i)
WijYj |2
![Page 11: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/11.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Commonality of The Proposals: Localization
I Discard the information of large distances.
I Use local information:I Fixed-radius metric neighborhoods.I K nearest neighborhoods(KNN).
I Find a way to reassemble local structure and flatten themanifold.
I Our goal: Apply localization to MDS.
![Page 12: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/12.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Revisit MDS
I Data:Objects i = 1, 2, ...,N (subjects, stimuli, graph nodes,...)Dissimilarities Dij for all pairs of objects i and j
I Goal: Visualization of objectsFind configuration {xi ∈ IRk |i = 1..N} such that:
‖xi − xj‖ ≈ Dij
k: embedding dimension
I Stress function: Goodness of fit criterionThe simplest version:
MDSD(x1, ..., xN) =∑
i,j=1..N
(Dij − ||xi − xj ||)2
![Page 13: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/13.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Localization Problem in MDS
A simple proposal . . .
I Localization: Drop the large dissimilarities.
I Stress function becomes:
MDSD(x1, ..., xN) =∑
(i,j)∈E
(Dij − ||xi − xj ||)2
E = edge set of local links
Unstable system: Graef and Spence (1979)
![Page 14: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/14.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Localization Problem in MDS
A simple proposal . . .
I Localization: Drop the large dissimilarities.
I Stress function becomes:
MDSD(x1, ..., xN) =∑
(i,j)∈E
(Dij − ||xi − xj ||)2
E = edge set of local links
Unstable system: Graef and Spence (1979)
![Page 15: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/15.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
New Proposal: Local MDS(LMDS)
The idea . . .
I Approximate the small dissimilarities.
I Give a proper amount of repulsive force amongthe points to avoid instability.
![Page 16: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/16.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
New Proposal: Local MDS(LMDS)
I Step 1: Localization: E = edge set of local links; EC =edge set of non-local links.
I Step 2: Replace large distances (EC ) with ∞, but withinfinitesimal weight.
SD(x1, ..., xN) =∑
(ij)∈E
(Dij − ‖xi − xj‖)2
+ w ·∑
(ij)∈EC
(D∞ − ‖xi − xj‖)2
I Step 3: Take limit w → 0 and D∞ →∞ subject to2 w D∞ = t.
LMDSD(x1, ..., xN) =∑
(ij)∈E
(Dij − ‖xi − xj‖)2
− t ·∑
(ij)∈EC
‖xi − xj‖
![Page 17: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/17.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
New Proposal: Local MDS(LMDS)
LMDSD(x1, ..., xN) =∑
(ij)∈E
(Dij − ‖xi − xj‖)2
− t ·∑
(ij)∈EC
‖xi − xj‖
I The first term: Local stress to match small dissimilarities.
I The second term: Repulsive force for spreading out.
I The choice of t: t = τ · (K/N) ·medianE (Dij)
I τ repulsion tuning parameter, usually 0.001 ∼ 1.I K : the number of neighbors for each point .
![Page 18: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/18.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
The LC Meta-criterion: Motivation andDefinition
I Motivation
I Notation:I ND
K (i): Data point i ’s K nearest neighbors with regard tothe target dissimilarities Dij .
I N XK (i): Data point i ’s K nearest neighbors with regard to
the configuration distances ‖xi − xj‖.I | S |: Cardinality of a set S.
I The local version of the LC meta-criterion:
NK (i) = | NDK (i) ∩ NX
K (i) |
I The global version of the LC meta-criterion:
NK =1
N
N∑i=1
NK (i)
![Page 19: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/19.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
The LC meta-criterion: Normalization andAdjustment for Baselines
I Compare LC meta-criteria for different values of K.
I Normalization:
MK =1
KNK
I The problem of meta-criterion for large K : MN−1 = 1.
I Adjusted LC Meta-criterion:
MadjK = MK −
K
N − 1
![Page 20: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/20.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture Face
−50 0 50
−40
−30
−20
−10
0
10
20
30
3D LMDS
Left−right Pose
Up−
dow
n P
ose
−50 0 50
−40
−30
−20
−10
0
10
20
30
Left−right Pose
Ligh
ting
Dire
ctio
n
![Page 21: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/21.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture Face
Adjust the repulsion tuning parameter τ in LMDS:
5e−04 5e−03 5e−02 5e−01
2.5
3.0
3.5
4.0
4.5
5.0
K=6
τ
KM
K
![Page 22: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/22.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture FaceMethods PCA MDS Isomap LLE LMDS
N6 2.6 3.1 4.5 2.8 5.2
−15 −5 0 5 10
−15
−5
515
−10 0 5 10
−10
05
15
−10 0 5 10
−10
05
15
−2 −1 0 1 2
−2
−1
01
2
−2 −1 0 1 2
−2
−1
01
2
−2 −1 0 1 2
−2
−1
01
2
−2 −1 0 1 2−
20
12
−1.0 0.0 1.0
−1.
00.
01.
0
−1.0 0.0 1.0
−1.
00.
01.
0−0.25 −0.15 −0.05 0.05−
0.25
−0.
100.
00−0.25 −0.15 −0.05 0.05
−0.
25−
0.10
0.00
−0.25 −0.15 −0.05 0.05
−0.
25−
0.10
0.00
−50 0 50
−50
050
−40 −20 0 20
−40
−20
020
−40 −20 0 20−
40−
200
20
Left−
right
Pos
eU
p−do
wn
Pos
eLi
ghtin
g D
irect
ion
PCA MDS ISOMAP LLE LMDS
![Page 23: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/23.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture Face
−15 −10 −5 0 5 10 15
−15
−10
−5
05
1015
3D PCA (M6=2.6)
−2 −1 0 1 2
−2
−1
01
2
3D ISOMAP (M6=4.5)
−0.05 0.00 0.05
−0.
050.
000.
05
3D LLE (M6=2.8)
−50 0 50
−50
050
3D LMDS (M6=5.2)
![Page 24: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/24.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture Face
5 10 20 50 100 200 500
0.2
0.4
0.6
0.8
1.0
K
MK
MK
AdjustedMK
![Page 25: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/25.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Sculpture Face
5 10 20 50 100 200 500
0.0
0.2
0.4
0.6
0.8
1.0
K
MK
Ko=4Ko=6Ko=8Ko=10Ko=25Ko=125Ko=650
5 10 20 50 100 200 500
0.0
0.2
0.4
0.6
0.8
1.0
K
Adj
uste
d M
K
Ko=4Ko=6Ko=8Ko=10Ko=25Ko=125Ko=650
![Page 26: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/26.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 2: Frey Face
−600 −400 −200 0 200 400 600
−60
0−
400
−20
00
200
400
600 3D PCA
Left−right Pose
Ser
ious
−S
mili
ng
−2 −1 0 1 2
−2
−1
01
2
3D MDS
Left−right Pose
Ser
ious
−S
mili
ng
−2 −1 0 1 2
−2
−1
01
2
3D ISOMAP
Left−right Pose
Ser
ious
−S
mili
ng
−0.10 −0.05 0.00 0.05
−0.
10−
0.05
0.00
0.05
3D LLE
Left−right Pose
Ser
ious
−S
mili
ng
![Page 27: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/27.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 2: Frey Face
−10000 −5000 0 5000 10000
−10
000
−50
000
5000
1000
0
3D LMDS K0=12
Left−right Pose
Ser
ious
−S
mili
ng
−3e+06 −1e+06 1e+06 2e+06 3e+06
−3e
+06
−1e
+06
1e+
063e
+06
3D LMDS K0=4
Left−right Pose
Ser
ious
−S
mili
ng
![Page 28: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/28.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 2: Frey Face3D PCA
Left−right Pose
Ser
ious
−S
mili
ng
3D MDS
Left−right Pose
Ser
ious
−S
mili
ng
3D ISOMAP
Left−right Pose
Ser
ious
−S
mili
ng
3D LLE
Left−right Pose
Ser
ious
−S
mili
ng
3D LMDS K0=12
Left−right Pose
Ser
ious
−S
mili
ng
3D LMDS K0=4
Left−right Pose
Ser
ious
−S
mili
ng
![Page 29: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/29.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 2: Frey Face3D PCA (M12=3.6)
Left−right Pose
Ser
ious
−S
mili
ng
3D MDS (M12=4.8)
Left−right Pose
Ser
ious
−S
mili
ng
3D ISOMAP (M12=4.2)
Left−right Pose
Ser
ious
−S
mili
ng
3D LLE (M12=3.2)
Left−right Pose
Ser
ious
−S
mili
ng
3D LMDS K0=12 (M12=4.6)
Left−right Pose
Ser
ious
−S
mili
ng
3D LMDS K0=4 (M12=5.1)
Left−right Pose
Ser
ious
−S
mili
ng
![Page 30: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/30.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 1: Frey Face
5 10 20 50 100
0.3
0.4
0.5
0.6
K
MK
Ko=4Ko=8Ko=12Ko=16Ko=20Ko=50Ko=100Ko=150Ko=1964
5 10 20 50 100
−0.
10−
0.05
0.00
0.05
K
Adj
uste
d M
K w
ith M
DS
![Page 31: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/31.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Outline for the Rest of the Talk
I Graphs and energy-based models.
I Stress in LMDS and energy in graph layout.
I Clustering postulate.
I Generalized LMDS: A Box-Cox family of energy functions.
![Page 32: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/32.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Graph-layout
I A graph G = (V ,E ):I V ↔ Entities: Telephone numbers, carbon atoms . . .I E ↔ Relations: Phone calls, chemical bonds . . .
I Goal: Two-dimensional layout ↔ Visualize the relationalinformation.
I Algorithm: Force-directedPlacement(Eades, 1987)
I A physical system:Vertices → Steel rings;Edges → Springs.
I Best layout is achieved atthe minimal energy state.
![Page 33: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/33.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Energy-based Models
I Notation:I E = edge set; EC = non-edge set; V 2 = E ∪ EC .I Denote dij = ‖xi − xj‖ = drawn distances.I U: Energy of the system.I f (·): energy corresponding to attraction
g(·):energy corresponding to repulsion
I Common in graph layout: attraction-repulsion forces
U =∑E
f (dij) −∑V 2
g(dij)
I LMDS and energy-based models.
LMDSD =∑E
(Dij − dij)2 − t ·
∑EC
dij
![Page 34: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/34.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Energy-based Models
Some energy functions . . .
I Eades (1984)
U =∑
E
dij ·(log(dij)− 1− dij
−2)
+∑V 2
dij−1
I Fruchterman & Reingold (1991)
U =∑
E
dij3/3 −
∑V 2
log(dij)
I Davidson & Harel (1996)
U =∑
E
dij2 +
∑V 2
dij−2
I Noack (2003), LinLog model
U =∑
E
dij −∑V 2
log(dij)
![Page 35: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/35.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Clustering Postulate
I A graph with two clusters:
n1ne
n2
I Problem: What should be the inter-cluster distance d?
I Energy:U(d) = ne f (d)− n1n2g(d)
I Clustering Postulate(Noack 2004):I U(d) should be minimized at d = n1n2/ne = 1/C .I C = ne/n1n2 is called coupling strength, C < 1.
I Stationary condition yields:
f ′(d) = d · g ′(d)
I Weak Clustering Postulate: More strongly coupled clustersshould be placed more closely to each other.
![Page 36: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/36.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Distance-weighted Graphs
I Assume target distances Dij for (i , j) ∈ E .
I Extended clustering condition: In the 2-cluster situation,require the minimum at
d = D · ( 1C)λ
I D: the target distanceI ( 1
C)λ: a generalized clustering factor.
I λ > 0: clustering power; C < 1: coupling strength.
I New stationarity condition:
f ′(d) =
(d
D
) 1λ
g ′(d)
I f (d) and g(d) satisfy the extended coupling condition:
f (d) =dµ+ 1
λ − 1
µ + 1λ
, g(d) =dµ − 1
µD
1λ
with logarithmic fill-in for zero exponents.
![Page 37: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/37.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
A Box-Cox Family of Energy Functions
I Energy:
U ∼∑
E
(dij
µ+ 1λ −1
µ+ 1λ
− Dij1λ
dijµ−1µ
)− t
∑E c
dijµ−1µ
I Special cases:I LMDS: λ = µ = 1I Energy for unweighted graph(Dij = 1):
I Noack(LinLog): λ = 1, µ = 0I Fruchterman&Reingold: λ = 1
3, µ = 0
I Davison&Harel: λ = 14, µ = −2
Advantages . . .
I Flexibility for achieving desirable configurations.
I Clustering power λ: visualize the clusters.
![Page 38: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/38.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 3: Olivetti Faces
![Page 39: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/39.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Example 3: Olivetti FacesMethods λ = 0.25 λ = 0.5 λ = 1.0 λ = 1.5
N4 0.64 1.01 1.44 1.57
−6000 −2000 0 2000 6000
−60
00−
2000
020
0060
00
λ=0.25
−15000 −5000 0 5000 15000−15
000
−50
000
5000
1500
0 λ=0.5
−20000 0 10000
−20
000
010
000
λ=1
−40000 −20000 0 20000 40000
−40
000
020
000
4000
0 λ=1.5
![Page 40: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/40.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Conclusion and Discussion
I Nonlinear dimension reduction, proximity analysis andgraph-layout
I ContiguityI DistancesI Localization
I Graph Layout: Two Fundamental ApproachesI Complete the graph (Isomap, Krustal and Seery(1980))I Apply repulsive force
I B-C Energy Functions
I The Selection Problem: LC Meta-Criteria
![Page 41: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/41.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Diagnostic Plot: Frey Face
500 1000 1500
05000
10000
15000
20000
Diagnostic Plot
D_ij
d_ij
![Page 42: Local Multidimensional Scaling [1em] For Nonlinear ......for Measuring Local Continuity Examples Graph-layout and Energy functions Generalized LMDS Conclusion and Discussion Example](https://reader030.fdocuments.in/reader030/viewer/2022041100/5ed68eec5cd0d56eef02e827/html5/thumbnails/42.jpg)
Local MultidimensionalScaling
For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis
L. Chen
Motivation
Some Existing Methods
New proposal: LMDS
The LC Meta-criterionfor Measuring LocalContinuity
Examples
Graph-layout and Energyfunctions
Generalized LMDS
Conclusion andDiscussion
Diagnostic Plot: Frey Face