Post on 11-Jan-2016
Online Learning for Matrix Factorization and Sparse Coding
Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro
Journal of Machine Learning Research 2010
Introduction
• This paper focuses on the large scale matrix factorization problem, including– Dictionary learning for sparse coding– Non-negative matrix factorization (NMF)– Sparse principal component analysis (SPCA)
• Contributions of this paper:– An iterative online algorithm is proposed for large scale matrix
factorization– This algorithm is proved to converge almost surely to a stationary
point of the objective function– This algorithm is shown to be much faster than previous methods
in the experiment.
Problem Statement
• Classical dictionary learning problem Given a finite training set , the objective is
to optimize the following function
where
• Online Learning
This algorithm process one sample (or a mini-batch) at a time and sequentially minimize the following function:
Basic Algorithm
Dictionary Update
Optimizing the Algorithm
• Handling fixed-sized data sets
• Scaling the “past” data
• Mini-batch extension
Proof of Convergence
• Assumptions:
• Main results
Extensions to Matrix Factorization
• Non-negative matrix factorization (NMF)
• Non-negative sparse coding (NNSC)
• Sparse principal component analysis (SPCA)
Data for Experiment
• 1.25 million patches from Pascal VOC’06 image database
Online VS. Batch
• Training data size: 1 million• OL1:• OL2:• OL3:
Comparison with NMF and NNSC• NMF
• NNSC
Face Results
Image Patches Results
Inpainting Results
• Image size: 12-Megapixel• Dictionary with 256 elements• Training data: 7 million 12 by 12 color patches
Conclusion
• A new online algorithm for learning dictionaries adapted to sparse coding tasks, and proven its convergence.
• Experiments demonstrate that this algorithm is significantly faster than existing batch methods.
• This algorithm can be extended to other matrix factorization problems such as non-negative matrix factorization and sparse principal component analysis.