IML Tutorial Review HW5 - ETH Z
Transcript of IML Tutorial Review HW5 - ETH Z
![Page 1: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/1.jpg)
IML TutorialReview HW5
Jakob Jakob 1
1ETH Zurich
1
![Page 2: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/2.jpg)
K-means
Given: n points xi ∈ Rd, i ∈ 1, ..., nGoal: Find the k clusters µ = (µ1, ..., µk)Minimize
L(µ) =n∑
i=1
minj∈1,...,k
‖xi − µj‖22
Algorithm : Assignment and refitting step
zi ∈ arg minj∈{1,...,k}
‖xi − µj‖22 µj =1
|{i : zi = j}|∑i:zi=j
xi
2
![Page 3: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/3.jpg)
K-means — visualizationGiven data
Example k = 2, d = 2
3
![Page 4: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/4.jpg)
K-means — visualizationInitialization of µ1 and µ2
Example k = 2, d = 2
4
![Page 5: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/5.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
5
![Page 6: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/6.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
6
![Page 7: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/7.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
7
![Page 8: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/8.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
8
![Page 9: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/9.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
9
![Page 10: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/10.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
10
![Page 11: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/11.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
11
![Page 12: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/12.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
12
![Page 13: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/13.jpg)
K-means — visualizationAssignment step
Example k = 2, d = 2
13
![Page 14: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/14.jpg)
K-means — visualizationRefitting step
Example k = 2, d = 2
14
![Page 15: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/15.jpg)
K-means — visualizationRefitting step
Example k = 2, d = 2
15
![Page 16: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/16.jpg)
K-means — visualizationRefitting step
Example k = 2, d = 2
16
![Page 17: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/17.jpg)
K-means — visualizationRefitting step
Example k = 2, d = 2
17
![Page 18: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/18.jpg)
ReviewQ6
Explanation of Q6 (switch to solutions)
18
![Page 19: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/19.jpg)
L2 Loss
L(µ) =n∑
i=1
minj∈1,...,k
‖xi − µj‖22
L(µ) =n∑
i=1
‖xi − µzi‖22 zi = arg minj∈1,...,k
‖xi − µj‖22
µ = arg minµ
n∑i=1
‖xi − µzi‖22
µj = arg minµj
∑i:zi=j
‖xi − µj‖22
19
![Page 20: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/20.jpg)
L2 Loss
µj = arg minµj
∑i:zi=j
‖xi − µj‖22
∂L
∂µj=
∑i:zi=j
−2(xi − µj) = −2∑i:zi=j
(xi − µj) = 0
=∑i:zi=j
(xi − µj) =∑i:zi=j
xi − |{i : zi = j}|µj = 0
=⇒ µj =1
|{i : zi = j}|∑i:zi=j
xi
20
![Page 21: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/21.jpg)
L1 Loss
L(µ) =n∑
i=1
minj∈1,...,k
‖xi − µj‖1
L(µ) =n∑
i=1
‖xi − µzi‖1 zi = arg minj∈1,...,k
‖xi − µj‖1
µ = arg minµ
n∑i=1
‖xi − µzi‖1
µj = arg minµj
∑i:zi=j
‖xi − µj‖1 = arg minµj
∑i:zi=j
d∑q=1
|xi,q − µj,q|
21
![Page 22: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/22.jpg)
L1 Loss
µj = arg minµj
∑i:zi=j
d∑q=1
|xi,q − µj,q|
µj,q = arg minµj,q
∑i:zi=j
|xi,q − µj,q|
L(µj,q) =∑i:zi=j
|xi,q − µj,q|
=∑
i:zi=j, xi,q≤µj,q
|xi,q − µj,q|+∑
i:zi=j, xi,q>µj,q
|xi,q − µj,q|
22
![Page 23: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/23.jpg)
L1 Loss
L(µj,q) =∑
i:zi=j, xi,q≤µj,q
|xi,q − µj,q|+∑
i:zi=j, xi,q>µj,q
|xi,q − µj,q|
=∑
i:zi=j, xi,q≤µj,q
(µj,q − xi,q) +∑
i:zi=j, xi,q>µj,q
(xi,q − µj,q)
∂L
∂µj= |{i : zi = j, xi,q ≤ µj,q}| − |{i : zi = j, xi,q > µj,q}| = 0
=⇒ µj,q = median (xi,q, i : zi = j)
23
![Page 24: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/24.jpg)
Mean vs Median — visualization
Example with an outlier k = 1, d = 1
24
![Page 25: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/25.jpg)
Mean vs Median — visualization
Example with an outlier k = 1, d = 1, median in blue and mean in red
25
![Page 26: IML Tutorial Review HW5 - ETH Z](https://reader034.fdocuments.in/reader034/viewer/2022042617/62641fef030fee5162287cc3/html5/thumbnails/26.jpg)
END OF REVIEW OF HW5
26