Schedule for near future….wcohen/10-605/sgd-mf.pdf · • Rich Dad’s Guide to Investing: What...
91
Schedule for near future…. Previous SGD 1
Transcript of Schedule for near future….wcohen/10-605/sgd-mf.pdf · • Rich Dad’s Guide to Investing: What...
Schedule for near future….
Previous SGD
1
Midterm • Will cover all the lectures scheduled through today • There are some sample questions up already from
previous years – syllabus is not very different for first half of course.
• Problems are mostly going to be harder than the quiz questions
• Questions often include material from a homework – so make sure you understand a HW if you decided
to drop it • Closed book and closed internet • You can bring in one sheet
– 8.5x11 or A4 paper front and back
2
Wrap-up on iterative parameter mixing
3
NAACL 2010
4
Recap
Parallelizing perceptrons – take 2
Instances/labels
Instances/labels – 1
Instances/labels – 2
Instances/labels – 3
w -1 w- 2 w-3
w
Split into example subsets
Combine by some sort of
weighted averaging
Compute local vk’s
w (previous)
5
Recap: Iterative Parameter Mixing
Parallelizing perceptrons – take 2
Instances/labels
Instances/labels – 1
Instances/labels – 2
Instances/labels – 3
w -1 w- 2 w-3
w
Split into example subsets
Combine by some sort of
weighted averaging
Compute local vk’s
w (previous)
6
Recap: Iterative Parameter Mixing
Parallel Perceptrons – take 2
Idea: do the simplest possible thing iteratively. • Split the data into shards • Let w = 0 • For n=1,…
• Train a perceptron on each shard with one pass starting with w • Average the weight vectors (somehow) and let w be that average
Extra communication cost: • redistributing the weight vectors • done less frequently than if fully synchronized, more frequently than if fully parallelized
All-Reduce
7
Recap: Iterative Parameter Mixing
ALL-REDUCE
8
Introduction • Common pattern:
– do some learning in parallel – aggregate local changes from each processor
• to shared parameters – distribute the new shared parameters
• back to each processor
– and repeat….
• AllReduce implemented in MPI, also in VW code (John Langford) in a Hadoop/compatible scheme
MAP
ALLREDUCE
9
10
11
12
13
14
15
Gory details of VW Hadoop-AllReduce • Spanning-tree server:
– Separate process constructs a spanning tree of the compute nodes in the cluster and then acts as a server
• Worker nodes (“fake” mappers): – Input for worker is locally cached – Workers all connect to spanning-tree server – Workers all execute the same code, which
might contain AllReduce calls: • Workers synchronize whenever they reach an all-
reduce 16
Hadoop AllReduce
don’t wait for duplicate jobs
17
Second-order method - like Newton’s method
18
2 24 features ~=100 non-zeros/example 2.3B examples example is user/page/ad and conjunctions of these, positive if there was a click-thru on the ad
19
50M examples explicitly constructed kernel è 11.7M features 3,300 nonzeros/example old method: SVM, 3 days: reporting time to get to fixed test error
20
21
Matrix Factorization
22
Recovering latent factors in a matrix
m columns
v11 …
… …
vij
…
vnm
n ro
ws
23
Recovering latent factors in a matrix
K * m
n *
K
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
24
What is this for?
K * m
n *
K
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
25
MF for collaborative filtering
26
What is collaborative filtering?
27
What is collaborative filtering?
28
What is collaborative filtering?
29
What is collaborative filtering?
30
31
Recovering latent factors in a matrix
m movies
v11 …
… …
vij
…
vnm
V[i,j] = user i’s rating of movie j
n us
ers
32
Recovering latent factors in a matrix
m movies
n us
ers
m movies
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
V[i,j] = user i’s rating of movie j
33
34
MF for image modeling
35
36
Data: many copies of an image, rotated and shifted (matrix with one image/row)
Image “prototypes:” a smaller number of row vectors (green=negative)
Reconstructed images : linear combinations of prototypes
MF for images
10,000 pixels
1000
imag
es
1000 * 10,000,00
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 … … …
… …
vij
…
vnm
~
V[i,j] = pixel j in image i
2 prototypes
PC1
PC2
37
MF for modeling text
38
• The Neatest Little Guide to Stock Market Investing • Investing For Dummies, 4th Edition • The Little Book of Common Sense Investing: The Only
Way to Guarantee Your Fair Share of Stock Market Returns
• The Little Book of Value Investing • Value Investing: From Graham to Buffett and Beyond • Rich Dad’s Guide to Investing: What the Rich Invest in,
That the Poor and the Middle Class Do Not! • Investing in Real Estate, 5th Edition • Stock Investing For Dummies • Rich Dad’s Advisors: The ABC’s of Real Estate
Investing: The Secrets of Finding Hidden Profits Most Investors Miss
https://technowiki.wordpress.com/2011/08/27/latent-semantic-analysis-lsa-tutorial/
39
https://technowiki.wordpress.com/2011/08/27/latent-semantic-analysis-lsa-tutorial/
TFIDF counts would be better
40
Recovering latent factors in a matrix
m terms
n do
cum
ents
doc term matrix
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
V[i,j] = TFIDF score of term j in doc i
41
=
42
Investing for real estate
Rich Dad’s Advisor’s:
The ABCs of Real Estate
Investment …
43
The little book of common
sense investing: …
Neatest Little Guide to Stock
Market Investing
44
MF is like clustering
45
k-means as MF
cluster means
n ex
ampl
es
0 1
1 0
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
original data set indicators for r
clusters
Z
M
X
46
How do you do it?
K * m
n *
K
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
47
talk pilfered from à …..
KDD 2011
48
49
Recovering latent factors in a matrix
m movies
n us
ers
m movies
x1 y1
x2 y2
.. ..
… …
xn yn
a1 a2 .. … am
b1 b2 … … bm v11 …
… …
vij
…
vnm
~
V[i,j] = user i’s rating of movie j
r
W
H
V
50
51
52
53
for image denoising
54
Matrix factorization as SGD
step size why does this work? 55
Matrix factorization as SGD - why does this work? Here’s the key
claim:
56
Checking the claim
Think for SGD for logistic regression • LR loss = compare y and ŷ = dot(w,x) • similar but now update w (user weights) and x (movie weight)
57
What loss functions are possible?
N1, N2 - diagonal matrixes, sort of like IDF
factors for the users/movies
“generalized” KL-divergence
58
What loss functions are possible?
59
What loss functions are possible?
60
ALS = alternating least squares
61
talk pilfered from à …..
KDD 2011
62
63
64
65
Like McDonnell et al with perceptron learning
66
Slow convergence…..
67
68
69
70
71
72
73
More detail….
• Randomly permute rows/cols of matrix • Chop V,W,H into blocks of size d x d
– m/d blocks in W, n/d blocks in H • Group the data:
– Pick a set of blocks with no overlapping rows or columns (a stratum)
– Repeat until all blocks in V are covered • Train the SGD
– Process strata in series – Process blocks within a stratum in parallel
74
More detail….
Z was V
75
More detail…. • Initialize W,H randomly
– not at zero J • Choose a random ordering (random sort) of the
points in a stratum in each “sub-epoch” • Pick strata sequence by permuting rows and columns
of M, and using M’[k,i] as column index of row i in subepoch k
• Use “bold driver” to set step size: – increase step size when loss decreases (in an
epoch) – decrease step size when loss increases
• Implemented in Hadoop and R/Snowfall
M=
76
77
Wall Clock Time 8 nodes, 64 cores, R/snow
78
79
80
81
82
Number of Epochs
83
84
85
86
87
Varying rank 100 epochs for all
88
Hadoop scalability
Hadoop process setup time starts
to dominate
89
Hadoop scalability
90
91
