Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6....
Transcript of Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6....
![Page 1: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/1.jpg)
Assignment 6
![Page 2: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/2.jpg)
Question 1.1Odds ratio
Linear model
Objective function
![Page 3: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/3.jpg)
Question 1.1Odds ratio
Linear model
Objective function
First step
replace withusing the fact that,
Second step
Apply “exp” on both sides to get rid of the log
Third stepSolve for and plug it into the objective function
Objective function
+ = 1
Starting from equation 1
![Page 4: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/4.jpg)
Question 1.2 One-vs-all Logistic Regression
![Page 5: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/5.jpg)
Question 1.2 One-vs-all Logistic Regression
use findMin with LogisticLoss instead(see assignment 4 for the LogisticLoss function)
Matrix WMatrix X Matrix y
=*
n samples, k classes, p features
dimensions = ? dimensions = ? dimensions = ?
![Page 6: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/6.jpg)
Question 1.2 One-vs-all Logistic Regression
use findMin with LogisticLoss instead(see assignment 4 for the LogisticLoss function)
Matrix WMatrix X Matrix y
=*
n samples, k classes, p features
dimensions = n x p dimensions = p x k dimensions = n x k
![Page 7: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/7.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is given as,
● Assume
● Convert y to binary form; i.e.
![Page 8: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/8.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is given as,
● Assume
● Convert y to binary form; i.e.
● Therefore,
● C is the number of classes ● W j refers to column j of W● is the binary predicted target value for class ‘c’ of
sample ‘i’
![Page 9: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/9.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,
![Page 10: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/10.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,
Only one of these terms is non-zero for any training example ‘i’
![Page 11: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/11.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,
(apply log)
(multiply by -1)
● The negative logarithm of the probability:
= ?
= ?
● The derivative of the negative log probability with respect to can be broken into two cases :
![Page 12: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/12.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,
(apply log)
(multiply by -1)
● The negative logarithm of the probability:
= ?
= ?
● The derivative of the negative log probability with respect to can be broken into two cases :
is column c of row j of Wwhich corresponds to the coefficient offeature j of class c
![Page 13: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/13.jpg)
Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,
(apply log)
(multiply by -1)
● The negative logarithm of the probability:
= ?
= ?
● The derivative of the negative log probability with respect to :
Hint: Use the indicator function to distinguishbetween the two cases
![Page 14: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/14.jpg)
Question 1.4 - Softmax Classifier
![Page 15: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/15.jpg)
Question 1.4 - Softmax Classifier
Use findMin instead with the softmax loss grad function
![Page 16: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/16.jpg)
Question 1.4 - Softmax Classifier
Change the contents of the green box on the left using that of the green boxes on the right
![Page 17: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/17.jpg)
Question 1.4 - Softmax Classifier
Change the contents of the green box on the left using that of the green boxes on the right
![Page 18: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/18.jpg)
Question 1.5 - Cost of Multinomial Logistic Regression
Time complexity for processing one example = ?
Time complexity for predicting one example = ?
![Page 19: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/19.jpg)
Question 1.5 - Cost of Multinomial Logistic Regression
![Page 20: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/20.jpg)
Question 2 random walk
un
un
-1
un
1
1
1
2 3
4
56
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
3 1
4 1
5 -1
matrix labelList
![Page 21: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/21.jpg)
Question 2 random walk
un
un
-1
un
1
1
1
2 3
4
56
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
v=
3 1
4 1
5 -1
matrix labelList
![Page 22: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/22.jpg)
Question 2 random walk
un
un
-1
un
1
1
1
2 3
4
56
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
prob = 1/2
prob = 1/2v=
3 1
4 1
5 -1
matrix labelList
![Page 23: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/23.jpg)
Question 2 random walk
un
un
-1
un
1
1
1
2 3
4
56
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
v=
3 1
4 1
5 -1
matrix labelList
![Page 24: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/24.jpg)
Question 2 random walk
un
un -1
un
1
1
1
2 3
4
56
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
v=prob = 1/2
prob = 1/2
3 1
4 1
5 -1
matrix labelList
![Page 25: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/25.jpg)
Question 2 random walk
un
un
-1
un
1
1
1
2 3
456
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
v=
yhat = -1 withprob = 1/(1+2) = 1/3
prob = 1/3
prob = 1/3
3 1
4 1
5 -1
matrix labelList
![Page 26: Assignment 6 - University of British Columbiaschmidtm/Courses/340-F15/t6.pdf · Assignment 6. Question 1.1 Odds ratio Linear model Objective function. Question 1.1 Odds ratio Linear](https://reader034.fdocuments.in/reader034/viewer/2022042417/5f3353dc6867cb14a816239a/html5/thumbnails/26.jpg)
Question 2 random walk
un
un
-1
-1
1
1
i=1
2 3
456
1
2
3
4
5
6
1 2 3 4 5 6
0 1 0 0 0 1
1 0 1 0 0 0
0 1 0 0 0 0
0 0 0 0 1 0
0 0 0 1 0 1
1 0 0 0 1 0
matrix AAdjacency matrix
v=
yhat = -1 withprob = 1/(1+2) = 1/3
prob = 1/3
prob = 1/3
3 1
4 1
5 -1
matrix labelList