Lecture II: Data Modeling and Applications
Transcript of Lecture II: Data Modeling and Applications
IPAM Computer Vision Summer School, Los Angeles, 2013
Lecture II:
Data Modeling and Applications
Yi Ma John Wright
Visual Computing Group Electrical Engineering
Microsoft Research Asia, Beijing Columbia University
??Images
VideosCompression
De-noising
Super-resolution
Recognition…Streaming
Tracking
Stabilization…
User data
Clustering
Classification
Collaborative filtering…
How to extract low-dim structures from such high-dim data?
1M pixels
1B voxels
U.S. COMMERCE'S ORTNER SAYS YEN UNDERVALUED
Commerce Dept. undersecretary of economic a®airs Robert Ortner said that
he believed the dollar at current levels was fairly priced against most European
currencies.
In a wide ranging address sponsored by the Export-Import Bank, Ortner,
the bank's senior economist also said he believed that the yen was undervalued
and could go up by 10 or 15 pct.
"I do not regard the dollar as undervalued at this point against the yen,"
he said.
On the other hand, Ortner said that he thought that "the yen is still a
little bit undervalued," and "could go up another 10 or 15 pct."
In addition, Ortner, who said he was speaking personally, said he thought
that the dollar against most European currencies was "fairly priced."
Ortner said his analysis of the various exchange rate values was based on
such economic particulars as wage rate di®erentiations.
Ortner said there had been little impact on U.S. trade de¯cit by the decline
of the dollar because at the time of the Plaza Accord, the dollar was extremely
overvalued and that the ¯rst 15 pct decline had little impact.
He said there were indications now that the trade de¯cit was beginning to
level o®.
Turning to Brazil and Mexico, Ortner made it clear that it would be
almost impossible for those countries to earn enough foreign exchange to pay
the service on their debts. He said the best way to deal with this was to use
the policies outlined in Treasury Secretary James Baker's debt initiative.
Web data
Indexing
Ranking
Search…
100B webpages
1B users
CONTEXT – Data increasingly massive, high-dimensional…
CONTEXT – Low dimensional structures in visual data
Visual data exhibit low-dimensional structures
due to rich local regularities, global symmetries,
repetitive patterns, or redundant sampling.
CONTEXT – Recent related progress
Sparse recovery:
Impossible in general ( )
Well-posed if is structured (sparse), but still NP-hard
Tractable via convex optimization:
… if is “nice” (random, incoherent, RIP)
Hugely active area: Donoho+Huo ’01, Elad+Bruckstein ‘03, Candès+Tao ’04,’05,
Tropp ’04, ‘06, Donoho ‘04, Fuchs ‘05, Zhao+Yu ‘06, Meinshausen+Buhlmann
‘06, Wainwright ‘09, Donoho+Tanner ‘09 … and many others
=
CONTEXT – Recent related progress
Low-rank sensing:
Impossible in general ( )
Well-posed if is structured (low-rank), but still NP-hard
Tractable via convex optimization:
… if is “nice” (random, rank-RIP)
Hugely active area: Recht+Fazel+Parillo ‘07, Candès+Plan ’10, Mohan+Fazel
‘10, Recht+Xu+Hassibi ’11, Chandrasekaran+Recht+Parillo+Willsky ‘11,
Negahban+Wainwright ’11 …
,=
APPLICATIONS – How to apply such models and tools?
1. Justify models of low-dimensional structures in practical data, such
as face images, textures, objects etc…
2. Customize models to incorporate additional variations in a problem,
such as incomplete or corrupted measurements, and/or deformation
or misalignment.
Ex: Face recognition with -regression:
If test image is also of subject , then
Linear subspace model for images of same face under varying illumination:
Subject i training
for some .
[Wright, Yang, Ganesh, Sastry, Ma, TPAMI’09]
Underdetermined system of linear equations in unknowns :
Seek the sparsest solution:
Solution is not unique … but
should be sparse: ideally, only supported on images of the same subject
expected to be sparse: occlusion only affects a subset of the pixels
convex relaxation
Ex: Face recognition with -regression:
[Wright, Yang, Ganesh, Sastry, Ma, TPAMI’09]
Extended Yale B Database
(38 subjects) Testing: subset 3 (453 images)
Training: subsets 1 and 2 (717 images)
30% corruption
50%
70%
99.3% 90.7%
37.5%
Ex: Face recognition with -regression:
[Wright, Yang, Ganesh, Sastry, Ma, TPAMI’09]
Ex: Face recognition with -regression:
If is “nice” and the model fits, can make strong statements about the performance of regression.
[Wright, Ma, Trans. Information Theory’09]
The space/model of face images of a person?
Conceptual Problem:
Given (information about) an object , can we provably detect or recognize the object from a new image ?
This talk: Attempt this for fixed pose, variations in illumination, (and possibly occlusion).
Geometry of illumination variations
Distant illumination identified with a Riemann integrable, nonnegative function .
Assume a linear sensor response. The image can often be written as
What is the set of possible images of ?
Let , and
The set is a convex cone. [Belhumeur + Kriegman ’98]
Geometry of illumination variations
Distant illumination identified with a Riemann integrable, nonnegative function .
Assume a linear sensor response. The image can often be written as
What is the set of possible images of ?
.
Geometry of illumination variations
Ambient cone model. Illumination is the sum of ambient and directional (arbitrary) components:
and corresponding cone of possible images:
Approximating a convex cone
Angular detector: is in or close to ?
Accept any input that is an image of under some valid illumination.
Reject any input that is not.
Work with a “Hausdorff distance” between cones:
If approximates in Hausdorff sense, we lose little in working with .
.
Cone Approximation: Existing Results
Approximation of general convex bodies in high dimensions is a disaster:
Theorem 2. [Bronstein, Ivanov ‘76] Let be a convex body. There exists an -approximation to in Hausdorff distance, with
vertices. For the unit sphere, this is optimal up to a constant.
Informal claim [Basri and Jacobs ‘03, Basri and Frolova ’04]:
For an convex, Lambertian object , there exists a subspace , with , such that if is a uniformly oriented random
point source,
In vision literature, results for average case, for convex objects:
Is there any special structure we can use here?
Extreme rays
Image formation: .
are images under directional illumination:
Extreme rays lie on a low dimensional submanifold of :
Extreme rays
Image formation: .
are images under directional illumination:
Lemma. [Just approximate the extreme rays]. Let . Then
where
Extreme rays lie on a low dimensional submanifold of :
Cone Approximation: New Results
If we define an illumination covering number
,
This implies in general, ,
and for convex objects .
Proof Sketch
Control the complexity of cone approximation in terms of object convexity defect.
The conceptual idea:Separate the direct illumination operator into
a low-rank part (due to smooth variations) and a sparse part (due to cast shadows):
The low-rank term can be bounded by directcalculation; the sparse term needs a more involved accounting.
LR+S captures physical intuition
Calculations with the Lambertian sphere suggest that sets of images of smooth, near-convex objects should be approximately low-rank: [Basri+Jacobs ’03,Ramamoorthi ‘04].
Cast shadows are often sparse[W., Yang, … Ma, ‘09] , [Candes, Li, Ma, W. ‘11]:
Observations used in photometric stereo:[Wu et. Al. ‘11] :
How can we exploit these structures to obtain compact representation/approximation of the cone?
Recover low-rank and sparse components
Convex relaxation:
with
Provably effective for recovery (and statistical estimation):
[Chadrasekaran,Sanghavi,Willsky,Parillo’11], [Candes, Li, Ma, W. ‘11], [Hsu,Kakade,Zhang ’12], [Agarwal, Wainwright ‘12], …
Cone-preserving dimensionality reduction
A low-rank and sparse decomposition that provably preserves verification performance?
Would need to solve
… but constraint is very complicated. Relax to:
Guaranteed Illumination Models for Nonconvex Objects, Zhang, Mu, Kuo, W., Arxiv ’13
Compressive Principal Component Pursuit, W., Ganesh, Min, Ma, I&I ’13
Towards a Practical Automatic Face Recognition, Wagner, W., Ganesh, Zhou, Mobahi, Ma,
PAMI ’12
Robust Principal Component Analysis? Candes, Li, Ma, W. JACM ’11
Dense Error Correction via L1 Minimization. Wright and Ma, Information Theory ’10
Robust Face Recognition via Sparse Representation, W., Yang, Ganesh,, Sastry, Ma, PAMI ’09
Provable Low-Dim Structures of Visual Data
Robust Alignment of Multiple (Face) Images
Problem: Given recover , and .
Low-rank component Sparse component
……
– corrupted & misaligned
observation
– aligned low-rank
signals
– sparse errors
…
Parametric deformations
(rigid, affine, projective…)
o
Solution: Robust Alignment via Low-rank and Sparse (RASL) Decomposition
Iteratively solving the linearized convex program::
RASL: Faces Detected
Input: faces detected by a face detector ( )
Average
Peng, Ganesh, Wright, Ma, CVPR’10, TPAMI’11
RASL: Faces Cleaned as the Low-Rank Component
Output: clean low-rank faces ( )
Average
Peng, Ganesh, Wright, Ma, CVPR’10, TPAMI’11
RASL: Sparse Errors of the Face Images
Output: sparse error images ( )
Peng, Ganesh, Wright, Ma, CVPR’10, TPAMI’11
RASL: Video Stabilization and Enhancement
Original video ( ) Low-rank part ( ) Sparse part ( )
Peng, Ganesh, Wright, Ma, CVPR’10, TPAMI’11
Aligned video ( )
RASL: Aligning Handwritten Digits
Learned-Miller PAMI’06 Vedaldi CVPR’08
Peng, Ganesh, Wright, Ma, CVPR’10, TPAMI’11
CONTEXT – Low dimensional structures in visual data
Individual images exhibit low-dimensional structures
due to rich local regularities, global symmetries,
repetitive patterns, or redundant sampling.
Robust Recovery of Low-Dimensional Models
Recover low-dimensional structures from a fraction of missing
measurements with structured support and/or a fraction of
random corruptions.
Low-rank Texture Sparse Errors
MAIN THEORY – Corrupted, Incomplete Matrix
Candes, Li, Ma, and Wright, Journal of the ACM, May 2011.
“Convex optimization recovers almost any matrix of rank from
any small fraction of entries, even with entries corrupted!”
Repairing Images: Highly Robust Repairing of Low-rank Textures!
Low-rank Texture Corruptions
Liang, Ren, Zhang, and Ma, in ECCV 2012.
Sensing or Imaging of Low-rank and Sparse Structures
corrupted data Low-rank Structures Sparse Structures
Fundamental Problem: How to recover low-rank and sparse structures from
subject to either nonlinear deformation or linear compressive sampling ?
Reconstructing 3D Geometry and Structures
Problem: Given recover , and simultaneously.
Low-rank component
(regular patterns…)Sparse component
(occlusion, corruption, foreground…)
– deformed observation – low-rank structures – sparse errors
Parametric deformations
(affine, projective, radial distortion, 3D shape…)
o
Solution: Iteratively solving the linearized convex program::
Objective: Transformed Principal Component Pursuit::
Or reduced version:
– deformed observation – low-rank structures – sparse errors
o
Transform Invariant Low-rank Textures (TILT)
Zhang, Liang, Ganesh, Ma, ACCV’10, IJCV’12
TILT: Shape from texture
Input (red window )
Output (rectified green window )
Zhang, Liang, Ganesh, Ma, ACCV’10, IJCV’12
TILT versus Hough Transform
Recognition: Character/Text Rectification
Xin Zhang, Zhouchen Lin, and Ma, ICDAR 2013
Object Recognition: Regularity of Texts at All Scales!
Input (red window )
Output (rectified green window )
Zhang, Liang, Ganesh, Ma, ACCV’10 and IJCV’12
From one input image From four input images
Mobahi, Zhou, and Ma, in ICCV 2011
TILT: Holistic 3D Reconstruction of Urban Scenes
From eight input images
Mobahi, Zhou, and Ma, in ICCV 2011
TILT: Holistic 3D Reconstruction of Urban Scenes
TILT: Camera Calibration with Radial Distortion
Zhang, Matsushita, and Ma, in CVPR 2011
Previous approach Low-rank method
Take-home Messages for Visual Data Processing:
1. (Transformed) low-rank and sparse structures are central to visual data
modeling, processing, and analyzing;
2. Such structures can now be extracted correctly, robustly, and efficiently,
from raw image pixels (or high-dim features);
3. These new algorithms unleash tremendous local or global information from
multiple or single images, emulating or surpassing human perception;
4. These algorithms start to exert significant impact on image/video processing,
3D reconstruction, and object recognition.
… …
But try not to abuse or misuse them…
Other Data/Applications: Rectifying 3D Object Orientation
Jin, Wu, and Liu, Graphical Models, 2012.
TILT for 3D: Unsupervised upright orientation of man-made 3D objects
Other Data/Applications: Web Document Corpus Analysis
Documents
Words
word frequency (or TF/IDF)
a better model/solution?
Informative,
discriminative
“keywords”
Low-rank
“background”
topic model
Latent Semantic Indexing: the classical solution (PCA)
Dense, difficult to interpret
CHRYSLER SETS STOCK SPLIT, HIGHER DIVIDEND
Chrysler Corp said its board declared a three-for-two stock split in the
form of a 50 pct stock dividend and raised the quarterly dividend by
seven pct.
The company said the dividend was raised to 37.5 cts a share from
35 cts on a pre-split basis, equal to a 25 ct dividend on a post-split
basis.
Chrysler said the stock dividend is payable April 13 to holders of
record March 23 while the cash dividend is payable April 15 to holders
of record March 23. It said cash will be paid in lieu of fractional shares.
With the split, Chrysler said 13.2 mln shares remain to be purchased
in its stock repurchase program that began in late 1984. That program
now has a target of 56.3 mln shares with the latest stock split.
Chrysler said in a statement the actions "re°ect not only our out-
standing performance over the past few years but also our optimism
about the company's future."
Other Data/Applications: Sparse Keywords Extracted
CHRYSLER SETS STOCK SPLIT, HIGHER DIVIDEND
Chrysler Corp said its board declared a three-for-two stock split in the
form of a 50 pct stock dividend and raised the quarterly dividend by
seven pct.
The company said the dividend was raised to 37.5 cts a share from
35 cts on a pre-split basis, equal to a 25 ct dividend on a post-split
basis.
Chrysler said the stock dividend is payable April 13 to holders of
record March 23 while the cash dividend is payable April 15 to holders
of record March 23. It said cash will be paid in lieu of fractional shares.
With the split, Chrysler said 13.2 mln shares remain to be purchased
in its stock repurchase program that began in late 1984. That program
now has a target of 56.3 mln shares with the latest stock split.
Chrysler said in a statement the actions "re°ect not only our out-
standing performance over the past few years but also our optimism
about the company's future."
Reuters-21578 dataset: 1,000 longest documents; 3,000 most frequent words
Min, Zhang, Wright, Ma, CIKM 2010.
Other Data/Applications: Protein-Gene Correlation
Microarray data
Wang, Machiraju, and Huang, submitted to Bioinformatics 2012.
Other Data/Applications: Lyrics and Music Separation
Songs (STFT)
Po-Sen Huang, Scott Chen, Paris Smaragdis, Mark Hasegawa-Johnson, ICASSP 2012.
Low-rank (music) Sparse (voices)
Other Data/Applications: Internet Traffic Anomalies
Mardani, Mateos, and Giannadis, submitted to Trans. Information Theory, 2012.
Network Traffic = Normal Traffic + Sparse Anomalies + Noise
Other Data/Applications: View-Invariant Gait Recognition
Same gait from different views Perspective distortion rectified
Other Data/Applications: Robust Filtering and System ID
½_x = Ax+Bu; A 2 <r£r
y = Cx+ z + e
x̂t+1 =Axt +K(yt ¡Cx̂t)
266666664
yn yn¡1 yn¡2 ¢ ¢ ¢ y0
yn¡1 yn¡2 ¢ ¢ ¢ . . . y¡1
yn¡2 ¢ ¢ ¢ . . .. . .
......
. . .. . .
. . . y¡n+2y0 y¡1 ¢ ¢ ¢ y¡n+2 y¡n+1
377777775= On£rXr£n + S
Robust Kalman Filter:
Robust System ID:
gross sparse errors
(due to buildings, trees…)
GPS on a Car:
Hankel matrix
Dynamical System Identification, Maryan Fazel, Stephen Boyd, 2000
Other Data/Applications: Learning Graphical Models
cond. indep. given other variables
Separation Principle:
• sparse pattern conditional (in)dependence
• rank of second component number of hidden variables
Chandrasekharan, Parrilo, and Wilsky, Annual of Statistics, 2012
A Perfect Storm in the Cloud…
Cloud Computing
(parallel, distributed,
networked)
Mathematical Theory
(high-dimensional statistics, convex geometry
measure concentration, combinatorics…)
Computational Methods
(convex optimization, first-order algorithms,
random sampling, approximate solutions…)
Massive Data
(images, videos,
texts, audios,
speeches, stocks,
user rankings…)
Applications
& Services
(data processing,
analysis, compression,
knowledge discovery,
search, recognition…)U.S. COMMERCE'S ORTNER SAYS YEN UNDERVALUED
Commerce Dept. undersecretary of economic a®airs Robert Ortner said that
he believed the dollar at current levels was fairly priced against most European
currencies.
In a wide ranging address sponsored by the Export-Import Bank, Ortner,
the bank's senior economist also said he believed that the yen was undervalued
and could go up by 10 or 15 pct.
"I do not regard the dollar as undervalued at this point against the yen,"
he said.
On the other hand, Ortner said that he thought that "the yen is still a
little bit undervalued," and "could go up another 10 or 15 pct."
In addition, Ortner, who said he was speaking personally, said he thought
that the dollar against most European currencies was "fairly priced."
Ortner said his analysis of the various exchange rate values was based on
such economic particulars as wage rate di®erentiations.
Ortner said there had been little impact on U.S. trade de¯cit by the decline
of the dollar because at the time of the Plaza Accord, the dollar was extremely
overvalued and that the ¯rst 15 pct decline had little impact.
He said there were indications now that the trade de¯cit was beginning to
level o®.
Turning to Brazil and Mexico, Ortner made it clear that it would be
almost impossible for those countries to earn enough foreign exchange to pay
the service on their debts. He said the best way to deal with this was to use
the policies outlined in Treasury Secretary James Baker's debt initiative.
Core References:
• RASL: Robust Alignment by Sparse and Low-rank Decomposition? Peng, Ganesh, Wright,
Xu, and Ma, Trans. PAMI, 2012.
• TILT: Transform Invariant Low-rank Textures, Zhang, Liang, Ganesh, and Ma, IJCV 2012.
• Compressive Principal Component Pursuit, Wright, Ganesh, Min, and Ma, ISIT 2012.
More references, codes, and applications on the website:
Colleagues: Students:
• Prof. Emmanuel Candes (Stanford)
• Prof. John Wright (Columbia)
• Prof. Zhouchen Lin (Peking University)
• Dr. Yasuyuki Matsushita (MSRA)
• Dr. Allen Yang (Berkeley)
• Dr. Arvind Ganesh (IBM Research, India)
• Prof. Shuicheng Yan (Na. Univ. Singapore)
• Prof. Jian Zhang (Sydney Tech. Univ.)
• Prof. Lei Zhang (HK Polytech Univ.)
• Prof. Liangshen Zhuang (USTC)
REFERENCES + ACKNOWLEDGEMENT
http://perception.csl.illinois.edu/matrix-rank/home.html
• Zhengdong Zhang (MSRA, Tsinghua University)
• Xiao Liang (MSRA, Tsinghua University)
• Xin Zhang (MSRA, Tsinghua University)
• Kerui Min (UIUC)
• Dr. Zhihan Zhou (UIUC)
• Dr. Hossein Mobahi (UIUC)
• Dr. Guangcan Liu (UIUC)
• Dr. Xiaodong Li (Stanford)